Crawl Insights / Reading the dashboard

Crawl Insights

Reading the dashboard

Date ranges, metrics and domains on the Crawl Insights page.

Overview shows the whole window at a glance; Details lists every path with its own drill-down. Both share the filters (domain, platform, bot type, bot and range), kept in the page address so a link shows the same view.

Date ranges

RangeBucketsNotes
Last 3 daysPer hour or per dayWhole days in UTC, today included
Last 7 daysPer day
Last 4 weeks / 3 monthsPer day or per weekWeeks start on Monday
Last 12 monthsPer week or per month

Hour buckets are shown in your browser’s time zone; day, week and month buckets are UTC calendar dates.

Metrics

MetricWhat it shows
Bot visitsRequests from AI bots in the period (after de-duplication), against the period of equal length just before.
Active botsDistinct bots seen.
Failure rateShare of visits your server answered with a 4xx or 5xx status. Pages bots cannot read cannot be cited.
Responses by status codeEvery status code bots got back, with its count and share. Click a code to list the paths that returned it.
Bot typeTraining (model training data), Search (search and answer indexes), User query (fetched for one person’s question) or Other.
PlatformThe company running the bot: GPTBot, OAI-SearchBot and ChatGPT-User are all OpenAI.
DetailsEvery path with visits, folder, platform share, status codes, errors and last visit; search, sort, filter by folder or status code, and open a path for its visits over time and per-bot status codes.

Domains

Each request is counted for the host it was made to, and every host found appears in the domain picker on the Crawl Insights page (default: all domains). Nothing has to be registered first.

SourceWhere the host comes from
CloudFrontThe Host header the visitor sent (x-host-header), e.g. www.example.com; the *.cloudfront.net name only for requests made to it directly.
ALB (connector or upload)The full URL in each log line.
Webhookrequest_url, else request_host, else the webhook’s domain.
JSON lines / custom formatThe host field or variable, else the domain entered for the upload.
nginx / Apache combined, CSV with pathsThe domain entered for the upload.

Host names are lower-cased and ports removed. www.example.com and example.com are separate domains.

Last updated 20 November 2026

Questions about this page? Write to connect@branly.ai