Crawl Insights / AWS ALB

Crawl Insights

AWS ALB

Forward AI-bot requests from Application Load Balancer access logs with one CloudFormation stack per region.

For sites behind an AWS Application Load Balancer (ALB). Like the CloudFront connector, a CloudFormation stack in your account turns on the load balancers’ access logging and runs a small Lambda that forwards only AI-bot requests; everything else stays in your account. Each log line carries the full request URL, so the host is always known.

Setting it up

You need the ARNs of your load balancers (EC2 console → Load Balancers → select one → ARN), and an AWS user that can create IAM roles, S3 buckets and Lambda functions.

In Branly, open Crawl Insights → Data sources → Connect AWS ALB and paste the ARNs.

They must all be Application Load Balancers in one region. The stack is created in that region, because AWS only delivers ALB logs to a bucket in the same region. Load balancers in other regions need their own connection.

Click Launch stack in AWS: CloudFormation opens in the load balancers’ region with everything pre-filled. Tick the IAM acknowledgement and click Create stack (2–5 minutes).

The setup page checks off stack connected → heartbeat → first log file → first AI visit. Load balancers deliver a log file every 5 minutes, so the first visits appear within about 5–10 minutes of traffic.

What the stack creates

The same pieces as the CloudFront stack, with these differences:

ItemALB stack
RegionMust be the load balancers’ region; one stack per region.
Log bucketEncrypted with S3-managed keys (ALB access logs do not support KMS keys); a bucket policy lets Elastic Load Balancing write to AWSLogs/<account>/ only. Files are deleted after 3 days.
Logging settingThe load balancers’ access-log attributes (access_logs.s3.*). A load balancer that already logs to another bucket is never redirected: the stack stops and explains.
Supported typesApplication Load Balancers. Network and Gateway Load Balancers are refused (their logs have no HTTP requests).

Pausing, disconnecting, new keys, updates and deleting the stack work exactly as for CloudFront.

Existing log bucket

If access logging is already on, tick “My load balancers already send access logs to an S3 bucket” and enter that bucket (it is in the same region) and the prefix set in the load balancers’ access-log settings. The stack only reads new files from it and leaves the logging configuration alone.

Last updated 20 November 2026

Questions about this page? Write to connect@branly.ai