Crawl Insights / Webhook
Crawl Insights
Webhook
Send access-log entries as JSON from any server, CDN or log shipper: request format, fields, responses and limits.
Send access-log entries as JSON over HTTPS from any server, CDN or log shipper (Vector, Fluent Bit, Logstash, Cloudflare Workers, a cron script…). Create a webhook in Branly under Crawl Insights → Data sources → Webhook; you get its URL and an API key (bsk_live_…, shown once). Optionally give it a domain: entries that send only a path are counted for that site.
Request format
POST to the webhook URL with Authorization: Bearer bsk_live_… and Content-Type: application/json. The body is an array of entries, or {"logs": [...]}. Add Content-Encoding: gzip to send it compressed.
curl -X POST 'https://<your-webhook-url>/v1/logs' \
-H 'Authorization: Bearer bsk_live_…' \
-H 'Content-Type: application/json' \
-d '[{
"timestamp": "2026-10-13T08:00:00Z",
"request_method": "GET",
"request_url": "https://www.example.com/blog/post?x=1",
"response_status": 200,
"user_agent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)",
"client_ip": "20.171.207.1"
}]'Fields
| Field | Required | Notes |
|---|---|---|
timestamp | yes | ISO 8601 (2026-10-13T08:00:00Z) or epoch seconds / ms / µs / ns. Up to 90 days old, at most a day ahead. |
request_method | yes | GET, HEAD, POST… |
request_url | yes* | Full URL with scheme and host. *Or request_host + request_path, or just a path when the webhook has a domain. |
request_host | Host the request was for, e.g. www.example.com. | |
request_path | Path and query, e.g. /blog/post?x=1. | |
response_status | yes | HTTP status 100–599 (number or string). |
user_agent | yes | The raw User-Agent header. |
client_ip | Visitor IP. Stored only as a keyed hash. | |
country_code | Two-letter country code. | |
referer | Referer header. | |
bytes_sent | Response size in bytes. | |
response_time_ms | Time to serve the request. | |
id | Your unique ID for the entry. When given, retries with the same id are ignored; otherwise duplicates are detected from the entry’s content. |
Send everything: there is no need to filter for bots yourself. Non-AI entries are counted in the response and then discarded; only AI-bot requests are stored.
Responses and limits
| Status | Meaning |
|---|---|
| 202 | Accepted. Body: received, ai_bot_entries, non_ai_entries, rejected_count and the first 50 rejected entries with reasons. A paused source answers 202 with "paused": true and discards the entries. |
| 400 / 422 | Malformed JSON, empty body or more than 500 entries. Fix and resend. |
| 401 / 403 | Missing, wrong or revoked key, or a disconnected source. Do not retry. |
| 413 | Body over 5 MB (50 MB after gzip decoding). Send smaller batches. |
| 429 | Rate limit: 20 requests a second per source (bursts up to 100). Retry after a second. |
Add ?dry_run=1 to validate entries without storing anything: each one comes back with whether it is valid, the reason if not, and whether it is an AI bot. Each webhook in Data sources shows its last request, entries received and the last rejection reason.
Last updated 20 November 2026
Questions about this page? Write to connect@branly.ai