Crawl Insights / AWS CloudFront

Crawl Insights

AWS CloudFront

Launch one CloudFormation stack that turns on CloudFront logging and forwards only AI-bot requests from your AWS account.

A CloudFormation stack in your AWS account turns on CloudFront standard logging for your distributions and runs a small Lambda function that reads each new log file, keeps only the lines from AI bots and sends those to Branly. There is nothing to install on your servers, and the stack can be deleted at any time.

Setting it up

You need the IDs of your distributions, and an AWS user that can create IAM roles, S3 buckets and Lambda functions.

In Branly, open Crawl Insights → Data sources → Connect AWS CloudFront and enter your distribution IDs (CloudFront console → Distributions → ID, e.g. E1ABCDEF123456) and the AWS region for the stack. Any region works.

Click Launch stack in AWS. CloudFormation opens in your account with the template and every parameter pre-filled, including this source’s API key.

Tick “I acknowledge that AWS CloudFormation might create IAM resources” and click Create stack. Creation takes 2–5 minutes.

Back in Branly, the setup page checks off stack connected → heartbeat → first log file → first AI visit by itself. CloudFront writes log files every few minutes, so the first visits usually appear within 5–30 minutes of traffic.

The stack calls Branly when it is created. If the key or address is wrong, creation fails straight away with the reason in the CloudFormation Events tab, and nothing is left half set up.

What the stack creates

ResourcePurpose
S3 bucket (unless you use your own)Receives CloudFront logs. Encrypted, private, HTTPS-only; files are deleted after 3 days.
CloudFront logging settingStandard logging switched on for each distribution, into that bucket. A distribution that already logs elsewhere is never redirected: the stack stops with an explanation instead.
Processor Lambda (Node.js 22)Streams each new log file, keeps the lines whose user agent is a known AI bot and sends them gzipped. Other traffic never leaves your account, even when its URL mentions a bot.
EventBridge rulesOne triggers the Lambda for each new log file; one runs the hourly heartbeat.
SQS dead-letter queueLog files that failed to send after 2 retries; the heartbeat retries them.
Configurator Lambda + IAM rolesTurns logging on and off, empties the bucket on deletion. Roles are limited to this stack’s bucket, queue and distributions.

Cost is typically a few cents a month: one short Lambda run per log file plus S3 storage for three days of logs. The (legacy) standard logging the stack turns on has no charge of its own beyond that S3 storage.

Existing log bucket and standard logging v2

If your distributions already send logs to a bucket, tick “My distributions already send standard logs to an S3 bucket” and enter its name (same account and region as the stack) and, optionally, the key prefix. The stack then only reads new files from it and leaves your logging configuration alone.

Log deliverySupported
Legacy standard logging (W3C, gzipped)Yes: this is what the stack sets up
Standard logging v2, W3C outputYes, gzipped or not
Standard logging v2, JSON outputYes, gzipped or not; fields are matched by name (see JSON field names)
Standard logging v2, Parquet outputNo: files are skipped; choose JSON or W3C
Real-time logs (Kinesis)No

Without a prefix, the stack picks up files ending in .gz, .json or .log; with a prefix, every file under it.

Keys, updates and deleting the stack

Action or statusWhat it does
New key & templateReplaces the source’s key; the old one stops working at once. Update the stack in CloudFormation (Update → Use existing template) and set IngestKey to the new key.
Update availableShown when a newer connector version exists. Update the stack with the new template (Update → Replace current template); data keeps flowing during the update.
Last error in your AWS accountThe stack’s most recent failure, reported with its hourly heartbeat (connector 1.6.0 and later), for example S3 access denied on a bucket encrypted with your own KMS key. It clears once failed files go through.
Last rejectionLog files from a distribution that is not part of this source (a shared bucket, or one removed from the stack) are accepted but not counted, and named here.
Deleting the stackIn CloudFormation, select the stack → Delete. Logging is switched off again, the log bucket is emptied and removed, and the source shows as disconnected. Data already received stays in your dashboard.

Last updated 20 November 2026

Questions about this page? Write to connect@branly.ai