Crawl Insights / Troubleshooting
Crawl Insights
Troubleshooting
Fixes for common problems with AWS stacks, the Cloudflare Worker, uploads and webhooks.
Common problems with each source, and how to fix them.
AWS connectors
The stack failed: “Distribution … already sends access logs to …”
That distribution already logs to another bucket. If the bucket belongs to another Branly stack, delete that stack first. Otherwise set it up again with “My distributions already send standard logs to an S3 bucket” and that bucket’s name, so its logs are read instead of redirected. Delete the failed stack before relaunching (it keeps its name until deleted).
The stack failed with HTTP 401 / 403
The key was rotated, the source disconnected, or the IngestKey parameter was edited. Get a new launch link from the setup page.
“Waiting for stack” never changes
Check the stack in CloudFormation: still creating, rolled back (see Events), or created in a different region than expected. The setup page’s View stack in AWS link opens it.
Connected, but no visits after 30 minutes
Were there AI-bot requests? Test with the command below; it shows up with the next log file. With an existing bucket, check the prefix and that logging v2 output is W3C or JSON, not Parquet.
curl -A "Mozilla/5.0 (compatible; GPTBot/1.2)" https://your-site/ALB stack failed: “… is a network load balancer” or “already sends access logs to …”
Only Application Load Balancers are supported. For one that already logs to a bucket, connect again with “My load balancers already send access logs to an S3 bucket” and that bucket and prefix. Delete the failed stack first.
ALB stack failed creating the log bucket or enabling logging
The stack must run in the load balancers’ region (the launch link opens it there). Access logs cannot be delivered to a bucket encrypted with a KMS key.
“No heartbeat”
The stack has not checked in for over 2 hours: it was deleted, or its Lambda cannot reach Branly. The processor Lambda’s CloudWatch logs (stack output ProcessorFunction) show the error. For the Cloudflare Worker, check its cron trigger (*/30 * * * *).
Uploads and webhooks
Upload: “Needs log format”
The layout was not recognised. Paste the log_format / LogFormat line from your server config; see Custom log formats.
Upload: lines “could not be read”
The analysis lists example lines and reasons (missing user agent, unparseable time…). A handful of bad lines is normal; if most fail, the upload asks for the log format.
Webhook: entries rejected
The 202 response and the webhook in Data sources show the reason for each rejected entry. Use ?dry_run=1 while setting up.
Numbers
Numbers differ from my analytics tool
Analytics tags only see browsers that run JavaScript; bots rarely do. Crawl Insights counts server-side requests, AI bots only.
Last updated 20 November 2026
Questions about this page? Write to connect@branly.ai