Crawl Insights / Overview
Crawl Insights
Overview
See which AI crawlers and assistants read your pages, measured from your own server or CDN logs, and choose how to send them.
Crawl Insights shows which AI crawlers and AI assistants request your pages (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and about 40 others), measured from your own server or CDN logs rather than estimated. You see which pages they read, how often, which ones fail, and how that changes over time.
How it works
Every source works the same way:
Your logs reach Branly: streamed by the Cloudflare Worker, the CloudFront or ALB connector or a webhook, or uploaded as files.
Each request is classified by its user agent. Only requests from known AI bots are kept; everything else is discarded on arrival (with the Cloudflare Worker and the AWS connectors it never even leaves your systems).
Requests are de-duplicated and counted per day (and per hour for recent days), which is what the dashboard reads. Counts appear within about 30 seconds.
Which bots are counted
| Group | Examples | What a visit means |
|---|---|---|
| Training crawlers | GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider | Content collected to train models. |
| Search / retrieval | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Applebot | Indexing for AI search and answers. |
| User-triggered | ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User | A person asked an assistant something and it fetched your page live. |
| Coding agents | Claude Code, Cursor, Codeium, Devin | Developer tools reading docs and code. |
Generic crawlers (SEO tools, uptime monitors, “bot/spider” user agents) are deliberately not counted, so the numbers are about AI only.
Choosing a data source
| Source | Best for | Effort | Freshness | History | What leaves your systems |
|---|---|---|---|---|---|
| Cloudflare | Sites on Cloudflare (any plan, including Free) | Paste an API token (or a Worker script), ~2 minutes | Seconds | From when it is connected | AI-bot requests only |
| AWS CloudFront | Sites served through CloudFront | One CloudFormation stack, ~5 minutes | Minutes (as CloudFront writes log files) | From when it is connected | AI-bot requests only |
| AWS ALB | Sites behind an Application Load Balancer | One stack per region, ~5 minutes | ~5 minutes | From when it is connected | AI-bot requests only |
| Webhook | Any server or CDN with a log shipper | Configure your shipper to POST JSON | Seconds | Up to 90 days back (send old entries) | What you send (non-AI is discarded) |
| File upload | Backfilling history, one-off checks | Drag and drop a file | When you upload | Up to 90 days back | The file (deleted after analysis) |
You can combine them: for example, upload last month’s logs once, then connect CloudFront for everything after. The same request sent twice is only counted once.
Set up every source in the Branly app under Crawl Insights → Data sources.
Last updated 20 November 2026
Questions about this page? Write to connect@branly.ai