Crawl Insights / File upload

Crawl Insights

File upload

Upload nginx, Apache, CloudFront, ALB, JSON or CSV logs to backfill up to 90 days, including custom log formats.

Drop a log file in Branly under Crawl Insights → Data sources → Upload access logs. Plain text, .gz and .zip (with many files inside) are accepted, up to 200 MB per upload and 2 GB once decompressed.

How an upload works

The file is analysed first: format, number of lines, AI-bot requests by bot, date range, and any lines that could not be read (with examples).

Nothing counts towards your monthly limit until you click Import. You can discard the analysis instead.

Import adds the requests to your dashboard. The file itself is deleted as soon as analysis finishes.

Uploading the same file again, or overlapping files, imports nothing twice. Requests older than 90 days are skipped (shown as out of range).

Supported formats

The format is detected from the first lines of each file:

FormatNotes
nginx / Apache combinedThe default access log layout. Extra fields at the end of the line are fine. Has no host: enter the site’s domain.
Common Log FormatRecognised but has no user agent, so it cannot be used. Switch to combined.
AWS CloudFront standard logsAny field selection; the field order is read from the #Fields header.
AWS ALB access logsThe fixed AWS layout, including fields AWS adds later.
JSON linesOne JSON object per line: Caddy, Traefik, Cloudflare Logpush, nginx escape=json, CloudFront v2 JSON, Google Cloud load balancers, most log shippers.
CSV with a header rowColumns by name: timestamp, request_method, response_status, user_agent and request_url (or request_host + request_path).
Anything elsePaste your server’s log format: see Custom log formats.

JSON field names

JSON logs are read by field name, not position. Names are compared ignoring case and punctuation (so User-Agent, user_agent, userAgent and cs(User-Agent) are the same), nested objects are looked into (request.headers.User-Agent) and for lists the first value is used. These names are recognised, first match wins:

FieldAccepted names (examples)
Timetimestamp, ts, time, @timestamp, time_iso8601, time_local, msec, StartUTC, EdgeStartTimestamp, date + time (CloudFront)
Methodrequest_method, method, http_method, cs-method, ClientRequestMethod, RequestMethod
URL / pathrequest_url, url, request_uri, uri, path, RequestPath, ClientRequestURI, cs-uri-stem (+ cs-uri-query), httpRequest.requestUrl, or request ("GET /x HTTP/1.1")
Hostrequest_host, host, http_host, x-host-header, ClientRequestHost, RequestHost, server_name, authority
Statusstatus, status_code, response_status, sc-status, EdgeResponseStatus, DownstreamStatus, httpRequest.status
User agentuser_agent, http_user_agent, ua, cs(User-Agent), ClientRequestUserAgent, request_User-Agent, request.headers.User-Agent, httpRequest.userAgent
Client IP (optional)client_ip, remote_addr, remote_ip, ClientIP, ClientHost, c-ip, x_forwarded_for (first address)
Referer, country, bytes (optional)referer / http_referer, country / ClientCountry / c-country, bytes_sent / body_bytes_sent / size / sc-bytes

Traefik drops request headers from its access log by default, so there is no user agent to classify. Keep it with accesslog.fields.headers.names.User-Agent=keep.

Custom log formats

If your nginx or Apache log layout was changed (the host first, extra fields in the middle, ISO timestamps), paste the format and the file is read with it. When a file is not recognised (or most of its lines don’t match), the upload stops at Needs log format and shows the first lines; paste the format, click Try on the first lines to see how they parse, then Use this format. You can also give it before uploading, under Custom nginx / Apache log format?.

Paste the whole directive or just the format string:

Log format examples
# nginx (nginx.conf)
log_format vhosts '$host $remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" "$http_user_agent" rt=$request_time';

# Apache (httpd.conf)
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
Needed fornginxApache
Time (required)$time_local, $time_iso8601 or $msec%t
Request (required)$request, or $request_method + $request_uri / $uri (+ $args)%r, or %m + %U (+ %q)
Status (required)$status%>s or %s
User agent (required)$http_user_agent%{User-Agent}i
Host (optional)$host, $http_host, $server_name%v, %V, %{Host}i
Client IP (optional)$remote_addr, $http_x_forwarded_for%h, %a, %{X-Forwarded-For}i
Also used if present$http_referer, $body_bytes_sent, $scheme, $geoip_country_code, $http_cf_ipcountry%{Referer}i, %b / %O, %{CF-IPCountry}i

Any other variable is allowed and simply skipped. Without a host variable you are asked for the site’s domain. A format with escape=json writes JSON lines, which are recognised automatically, so no format is needed.

Last updated 20 November 2026

Questions about this page? Write to connect@branly.ai