Crawl Insights / Cloudflare

Crawl Insights

Cloudflare

Install a small Cloudflare Worker, automatically with an API token or by hand, that reports AI-bot requests on any Cloudflare plan.

A small Cloudflare Worker runs on your zone in front of your site. It passes every request to your origin and returns the response unchanged; after the response is sent, it reports requests whose User-Agent is a known AI crawler to Branly, in small batches. Human traffic is never sent. If Branly cannot be reached, your site is not affected.

Automatic setup (API token)

The quickest way: give Branly a Cloudflare API token once, and it creates the Worker, its secret, its cron trigger and its routes for you. The token is used for that install only (it is never stored or logged), and you can delete it in Cloudflare right after.

In Branly, open Crawl Insights → Data sources → Connect Cloudflare, enter your domain and click Continue.

Click Create a token in Cloudflare. Cloudflare opens with the three permissions the install needs: Workers Scripts: Edit, Workers Routes: Edit and Zone: Read. Optionally limit it to your account and zone, then Create Token and copy it.

Paste it and click Check token. Branly finds your zone and suggests routes (your domain, and www for the apex). Routes already used by another Worker are shown but never changed: Cloudflare runs one Worker per route.

Click Install Worker. The routes are set to fail open, so the Free plan’s daily limit never blocks your site.

Update Worker under Data sources repeats the install with a new token (new script, new key, same routes). Remove from Cloudflare deletes the Worker and its routes; other Workers and routes are not touched.

Manual setup

You need a site whose DNS is proxied by Cloudflare (orange cloud), and access to its Cloudflare account.

In Branly, open Connect Cloudflare, enter your domain and click Continue. Copy the Worker script and the API key (the key is shown once).

In Cloudflare: Workers & Pages → Create → Start with Hello World!, name it branly-crawl-insights, deploy, then Edit code, replace the code with the script and deploy again.

Don’t upload the file: Cloudflare’s upload option is for static sites and refuses JavaScript.

Settings → Variables and Secrets: add a Secret named BRANLY_INGEST_KEY with the API key.

Open your domain in Cloudflare → Workers Routes → Add route: a route such as example.com/*, the Worker branly-crawl-insights, and Request limit failure mode: Fail open. Add one per hostname you serve (or a wildcard like *.example.com/*).

Settings → Trigger Events: add the cron trigger */30 * * * *, the Worker’s heartbeat.

Back in Branly, the setup page checks off the first delivery and the first AI visit. Test it right away:

Test
curl -A "GPTBot/1.2" https://example.com/

Prefer the command line? The setup page has a wrangler.toml to deploy the same Worker with npx wrangler deploy.

Plans and limits

Workers run on every Cloudflare plan. The Workers Free plan allows 100,000 requests a day across your account. With the route’s failure mode set to Fail open, requests over the limit skip the Worker: your site keeps working and only those visits go uncounted. For busier sites, use Workers Paid or narrow the route (for example to your content paths).

A Worker only sees requests that reach Cloudflare’s proxy, including cached ones. Requests to hostnames that are not proxied (grey cloud) or not covered by a route are not counted.

Keys, updates and removing it

New key & script under Data sources issues a new key (the old one stops at once) and the current script: put the key in the BRANLY_INGEST_KEY secret. When the panel shows Update available (a new Worker version or a new list of AI crawlers), replace the Worker’s code with the new script.

The panel shows No heartbeat when the cron trigger has not checked in for over 2 hours. To remove the integration, click Disconnect, then delete the Worker (or its route) in Cloudflare.

Last updated 20 November 2026

Questions about this page? Write to connect@branly.ai