Crawl Insights / Troubleshooting

Crawl Insights

Troubleshooting

Fixes for common problems with AWS stacks, the Cloudflare Worker, uploads and webhooks.

Common problems with each source, and how to fix them.

AWS connectors

The stack failed: “Distribution … already sends access logs to …”

That distribution already logs to another bucket. If the bucket belongs to another Branly stack, delete that stack first. Otherwise set it up again with “My distributions already send standard logs to an S3 bucket” and that bucket’s name, so its logs are read instead of redirected. Delete the failed stack before relaunching (it keeps its name until deleted).

The stack failed with HTTP 401 / 403

The key was rotated, the source disconnected, or the IngestKey parameter was edited. Get a new launch link from the setup page.

“Waiting for stack” never changes

Check the stack in CloudFormation: still creating, rolled back (see Events), or created in a different region than expected. The setup page’s View stack in AWS link opens it.

Connected, but no visits after 30 minutes

Were there AI-bot requests? Test with the command below; it shows up with the next log file. With an existing bucket, check the prefix and that logging v2 output is W3C or JSON, not Parquet.

Test
curl -A "Mozilla/5.0 (compatible; GPTBot/1.2)" https://your-site/

ALB stack failed: “… is a network load balancer” or “already sends access logs to …”

Only Application Load Balancers are supported. For one that already logs to a bucket, connect again with “My load balancers already send access logs to an S3 bucket” and that bucket and prefix. Delete the failed stack first.

ALB stack failed creating the log bucket or enabling logging

The stack must run in the load balancers’ region (the launch link opens it there). Access logs cannot be delivered to a bucket encrypted with a KMS key.

“No heartbeat”

The stack has not checked in for over 2 hours: it was deleted, or its Lambda cannot reach Branly. The processor Lambda’s CloudWatch logs (stack output ProcessorFunction) show the error. For the Cloudflare Worker, check its cron trigger (*/30 * * * *).

Uploads and webhooks

Upload: “Needs log format”

The layout was not recognised. Paste the log_format / LogFormat line from your server config; see Custom log formats.

Upload: lines “could not be read”

The analysis lists example lines and reasons (missing user agent, unparseable time…). A handful of bad lines is normal; if most fail, the upload asks for the log format.

Webhook: entries rejected

The 202 response and the webhook in Data sources show the reason for each rejected entry. Use ?dry_run=1 while setting up.

Numbers

Numbers differ from my analytics tool

Analytics tags only see browsers that run JavaScript; bots rarely do. Crawl Insights counts server-side requests, AI bots only.

Last updated 20 November 2026

Questions about this page? Write to connect@branly.ai