Crawl Insights / Webhook

Crawl Insights

Webhook

Send access-log entries as JSON from any server, CDN or log shipper: request format, fields, responses and limits.

Send access-log entries as JSON over HTTPS from any server, CDN or log shipper (Vector, Fluent Bit, Logstash, Cloudflare Workers, a cron script…). Create a webhook in Branly under Crawl Insights → Data sources → Webhook; you get its URL and an API key (bsk_live_…, shown once). Optionally give it a domain: entries that send only a path are counted for that site.

Request format

POST to the webhook URL with Authorization: Bearer bsk_live_… and Content-Type: application/json. The body is an array of entries, or {"logs": [...]}. Add Content-Encoding: gzip to send it compressed.

Send one entry
curl -X POST 'https://<your-webhook-url>/v1/logs' \
  -H 'Authorization: Bearer bsk_live_…' \
  -H 'Content-Type: application/json' \
  -d '[{
    "timestamp": "2026-10-13T08:00:00Z",
    "request_method": "GET",
    "request_url": "https://www.example.com/blog/post?x=1",
    "response_status": 200,
    "user_agent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)",
    "client_ip": "20.171.207.1"
  }]'

Fields

FieldRequiredNotes
timestampyesISO 8601 (2026-10-13T08:00:00Z) or epoch seconds / ms / µs / ns. Up to 90 days old, at most a day ahead.
request_methodyesGET, HEAD, POST…
request_urlyes*Full URL with scheme and host. *Or request_host + request_path, or just a path when the webhook has a domain.
request_hostHost the request was for, e.g. www.example.com.
request_pathPath and query, e.g. /blog/post?x=1.
response_statusyesHTTP status 100–599 (number or string).
user_agentyesThe raw User-Agent header.
client_ipVisitor IP. Stored only as a keyed hash.
country_codeTwo-letter country code.
refererReferer header.
bytes_sentResponse size in bytes.
response_time_msTime to serve the request.
idYour unique ID for the entry. When given, retries with the same id are ignored; otherwise duplicates are detected from the entry’s content.

Send everything: there is no need to filter for bots yourself. Non-AI entries are counted in the response and then discarded; only AI-bot requests are stored.

Responses and limits

StatusMeaning
202Accepted. Body: received, ai_bot_entries, non_ai_entries, rejected_count and the first 50 rejected entries with reasons. A paused source answers 202 with "paused": true and discards the entries.
400 / 422Malformed JSON, empty body or more than 500 entries. Fix and resend.
401 / 403Missing, wrong or revoked key, or a disconnected source. Do not retry.
413Body over 5 MB (50 MB after gzip decoding). Send smaller batches.
429Rate limit: 20 requests a second per source (bursts up to 100). Retry after a second.

Add ?dry_run=1 to validate entries without storing anything: each one comes back with whether it is valid, the reason if not, and whether it is an AI bot. Each webhook in Data sources shows its last request, entries received and the last rejection reason.

Last updated 20 November 2026

Questions about this page? Write to connect@branly.ai