Crawl Insights / File upload
Crawl Insights
File upload
Upload nginx, Apache, CloudFront, ALB, JSON or CSV logs to backfill up to 90 days, including custom log formats.
Drop a log file in Branly under Crawl Insights → Data sources → Upload access logs. Plain text, .gz and .zip (with many files inside) are accepted, up to 200 MB per upload and 2 GB once decompressed.
How an upload works
The file is analysed first: format, number of lines, AI-bot requests by bot, date range, and any lines that could not be read (with examples).
Nothing counts towards your monthly limit until you click Import. You can discard the analysis instead.
Import adds the requests to your dashboard. The file itself is deleted as soon as analysis finishes.
Uploading the same file again, or overlapping files, imports nothing twice. Requests older than 90 days are skipped (shown as out of range).
Supported formats
The format is detected from the first lines of each file:
| Format | Notes |
|---|---|
| nginx / Apache combined | The default access log layout. Extra fields at the end of the line are fine. Has no host: enter the site’s domain. |
| Common Log Format | Recognised but has no user agent, so it cannot be used. Switch to combined. |
| AWS CloudFront standard logs | Any field selection; the field order is read from the #Fields header. |
| AWS ALB access logs | The fixed AWS layout, including fields AWS adds later. |
| JSON lines | One JSON object per line: Caddy, Traefik, Cloudflare Logpush, nginx escape=json, CloudFront v2 JSON, Google Cloud load balancers, most log shippers. |
| CSV with a header row | Columns by name: timestamp, request_method, response_status, user_agent and request_url (or request_host + request_path). |
| Anything else | Paste your server’s log format: see Custom log formats. |
JSON field names
JSON logs are read by field name, not position. Names are compared ignoring case and punctuation (so User-Agent, user_agent, userAgent and cs(User-Agent) are the same), nested objects are looked into (request.headers.User-Agent) and for lists the first value is used. These names are recognised, first match wins:
| Field | Accepted names (examples) |
|---|---|
| Time | timestamp, ts, time, @timestamp, time_iso8601, time_local, msec, StartUTC, EdgeStartTimestamp, date + time (CloudFront) |
| Method | request_method, method, http_method, cs-method, ClientRequestMethod, RequestMethod |
| URL / path | request_url, url, request_uri, uri, path, RequestPath, ClientRequestURI, cs-uri-stem (+ cs-uri-query), httpRequest.requestUrl, or request ("GET /x HTTP/1.1") |
| Host | request_host, host, http_host, x-host-header, ClientRequestHost, RequestHost, server_name, authority |
| Status | status, status_code, response_status, sc-status, EdgeResponseStatus, DownstreamStatus, httpRequest.status |
| User agent | user_agent, http_user_agent, ua, cs(User-Agent), ClientRequestUserAgent, request_User-Agent, request.headers.User-Agent, httpRequest.userAgent |
| Client IP (optional) | client_ip, remote_addr, remote_ip, ClientIP, ClientHost, c-ip, x_forwarded_for (first address) |
| Referer, country, bytes (optional) | referer / http_referer, country / ClientCountry / c-country, bytes_sent / body_bytes_sent / size / sc-bytes |
Traefik drops request headers from its access log by default, so there is no user agent to classify. Keep it with accesslog.fields.headers.names.User-Agent=keep.
Custom log formats
If your nginx or Apache log layout was changed (the host first, extra fields in the middle, ISO timestamps), paste the format and the file is read with it. When a file is not recognised (or most of its lines don’t match), the upload stops at Needs log format and shows the first lines; paste the format, click Try on the first lines to see how they parse, then Use this format. You can also give it before uploading, under Custom nginx / Apache log format?.
Paste the whole directive or just the format string:
# nginx (nginx.conf)
log_format vhosts '$host $remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent" rt=$request_time';
# Apache (httpd.conf)
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined| Needed for | nginx | Apache |
|---|---|---|
| Time (required) | $time_local, $time_iso8601 or $msec | %t |
| Request (required) | $request, or $request_method + $request_uri / $uri (+ $args) | %r, or %m + %U (+ %q) |
| Status (required) | $status | %>s or %s |
| User agent (required) | $http_user_agent | %{User-Agent}i |
| Host (optional) | $host, $http_host, $server_name | %v, %V, %{Host}i |
| Client IP (optional) | $remote_addr, $http_x_forwarded_for | %h, %a, %{X-Forwarded-For}i |
| Also used if present | $http_referer, $body_bytes_sent, $scheme, $geoip_country_code, $http_cf_ipcountry | %{Referer}i, %b / %O, %{CF-IPCountry}i |
Any other variable is allowed and simply skipped. Without a host variable you are asked for the site’s domain. A format with escape=json writes JSON lines, which are recognised automatically, so no format is needed.
Last updated 20 November 2026
Questions about this page? Write to connect@branly.ai