Crawl Insights / Overview

Crawl Insights

Overview

See which AI crawlers and assistants read your pages, measured from your own server or CDN logs, and choose how to send them.

Crawl Insights shows which AI crawlers and AI assistants request your pages (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and about 40 others), measured from your own server or CDN logs rather than estimated. You see which pages they read, how often, which ones fail, and how that changes over time.

How it works

Every source works the same way:

Your logs reach Branly: streamed by the Cloudflare Worker, the CloudFront or ALB connector or a webhook, or uploaded as files.

Each request is classified by its user agent. Only requests from known AI bots are kept; everything else is discarded on arrival (with the Cloudflare Worker and the AWS connectors it never even leaves your systems).

Requests are de-duplicated and counted per day (and per hour for recent days), which is what the dashboard reads. Counts appear within about 30 seconds.

Which bots are counted

GroupExamplesWhat a visit means
Training crawlersGPTBot, ClaudeBot, Google-Extended, CCBot, BytespiderContent collected to train models.
Search / retrievalOAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, ApplebotIndexing for AI search and answers.
User-triggeredChatGPT-User, Claude-User, Perplexity-User, MistralAI-UserA person asked an assistant something and it fetched your page live.
Coding agentsClaude Code, Cursor, Codeium, DevinDeveloper tools reading docs and code.

Generic crawlers (SEO tools, uptime monitors, “bot/spider” user agents) are deliberately not counted, so the numbers are about AI only.

Choosing a data source

SourceBest forEffortFreshnessHistoryWhat leaves your systems
CloudflareSites on Cloudflare (any plan, including Free)Paste an API token (or a Worker script), ~2 minutesSecondsFrom when it is connectedAI-bot requests only
AWS CloudFrontSites served through CloudFrontOne CloudFormation stack, ~5 minutesMinutes (as CloudFront writes log files)From when it is connectedAI-bot requests only
AWS ALBSites behind an Application Load BalancerOne stack per region, ~5 minutes~5 minutesFrom when it is connectedAI-bot requests only
WebhookAny server or CDN with a log shipperConfigure your shipper to POST JSONSecondsUp to 90 days back (send old entries)What you send (non-AI is discarded)
File uploadBackfilling history, one-off checksDrag and drop a fileWhen you uploadUp to 90 days backThe file (deleted after analysis)

You can combine them: for example, upload last month’s logs once, then connect CloudFront for everything after. The same request sent twice is only counted once.

Set up every source in the Branly app under Crawl Insights → Data sources.

Last updated 20 November 2026

Questions about this page? Write to connect@branly.ai