Crawl Insights / Reading the dashboard
Crawl Insights
Reading the dashboard
Date ranges, metrics and domains on the Crawl Insights page.
Overview shows the whole window at a glance; Details lists every path with its own drill-down. Both share the filters (domain, platform, bot type, bot and range), kept in the page address so a link shows the same view.
Date ranges
| Range | Buckets | Notes |
|---|---|---|
| Last 3 days | Per hour or per day | Whole days in UTC, today included |
| Last 7 days | Per day | |
| Last 4 weeks / 3 months | Per day or per week | Weeks start on Monday |
| Last 12 months | Per week or per month |
Hour buckets are shown in your browser’s time zone; day, week and month buckets are UTC calendar dates.
Metrics
| Metric | What it shows |
|---|---|
| Bot visits | Requests from AI bots in the period (after de-duplication), against the period of equal length just before. |
| Active bots | Distinct bots seen. |
| Failure rate | Share of visits your server answered with a 4xx or 5xx status. Pages bots cannot read cannot be cited. |
| Responses by status code | Every status code bots got back, with its count and share. Click a code to list the paths that returned it. |
| Bot type | Training (model training data), Search (search and answer indexes), User query (fetched for one person’s question) or Other. |
| Platform | The company running the bot: GPTBot, OAI-SearchBot and ChatGPT-User are all OpenAI. |
| Details | Every path with visits, folder, platform share, status codes, errors and last visit; search, sort, filter by folder or status code, and open a path for its visits over time and per-bot status codes. |
Domains
Each request is counted for the host it was made to, and every host found appears in the domain picker on the Crawl Insights page (default: all domains). Nothing has to be registered first.
| Source | Where the host comes from |
|---|---|
| CloudFront | The Host header the visitor sent (x-host-header), e.g. www.example.com; the *.cloudfront.net name only for requests made to it directly. |
| ALB (connector or upload) | The full URL in each log line. |
| Webhook | request_url, else request_host, else the webhook’s domain. |
| JSON lines / custom format | The host field or variable, else the domain entered for the upload. |
| nginx / Apache combined, CSV with paths | The domain entered for the upload. |
Host names are lower-cased and ports removed. www.example.com and example.com are separate domains.
Last updated 20 November 2026
Questions about this page? Write to connect@branly.ai