AI crawler log analysis answers questions no rank tracker can: Is OAI-SearchBot fetching our new buying guide? Did a WAF rule start serving 403s to PerplexityBot last Tuesday? Which product pages do people ask ChatGPT about? Your access logs already contain the evidence. This guide shows how to extract it, verify it and turn it into GEO decisions — honestly.
Why AI crawler log analysis matters for GEO
Answer engines can only cite what their crawlers can reach. Logs show three things polling cannot:
- Coverage. Which URLs each AI crawler has fetched, and how recently.
- Blocking. Status codes and challenge pages served to legitimate bots.
- Demand. User-initiated fetches that reveal which pages people are asking assistants about.
Know the three bot purposes
Group user agents by purpose, because each needs a different policy:
| Purpose | Example user-agent tokens | What a hit means |
|---|---|---|
| Training | GPTBot, ClaudeBot, CCBot |
Content may inform future models |
| Search index | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
Eligible to appear in AI answers |
| User fetch | ChatGPT-User, Claude-User, Perplexity-User |
A person asked about this page now |
Google-Extended is a robots.txt control token, not a crawler you will see in logs; Google fetches with its normal crawlers and applies the token to AI use.
Step 1: Extract AI bot hits from access logs
Start with the standard combined log format. A quick shell pass finds the volume per bot:
grep -Eo 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User' \
/var/log/nginx/access.log | sort | uniq -c | sort -rn
For anything beyond a glance, parse properly:
import re
from collections import Counter
LINE = re.compile(r'(?P<ip>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) [^"]*" '
r'(?P<status>\d{3}) \S+ "[^"]*" "(?P<ua>[^"]*)"')
BOTS = ("OAI-SearchBot", "ChatGPT-User", "GPTBot", "Claude-SearchBot", "Claude-User",
"ClaudeBot", "Perplexity-User", "PerplexityBot")
def bot_token(user_agent: str) -> str | None:
return next((b for b in BOTS if b in user_agent), None)
hits = Counter()
with open("/var/log/nginx/access.log", encoding="utf-8", errors="replace") as fh:
for raw in fh:
m = LINE.match(raw)
if m and (bot := bot_token(m["ua"])):
hits[(bot, m["status"], m["path"])] += 1
Order matters in BOTS: match the specific token (ChatGPT-User) before broader ones so each hit is classified once.
Step 2: Verify the bot is real
A user-agent string is a claim, not an identity. Anyone can send GPTBot in a header, so unverified hits must never drive decisions such as "allow this IP through the WAF".
Published IP ranges
Several operators publish machine-readable IP ranges for their crawlers (OpenAI and Perplexity do, for example). Load them and check membership:
import ipaddress
def in_ranges(ip: str, cidrs: list[str]) -> bool:
addr = ipaddress.ip_address(ip)
return any(addr in ipaddress.ip_network(c) for c in cidrs)
Refresh the range files on a schedule; they change.
Forward-confirmed reverse DNS
Where an operator documents a hostname pattern instead, do a reverse lookup on the IP, check the hostname suffix, then resolve that hostname forward and confirm it returns the same IP. Both directions must match; a reverse record alone is trivially forged.
Label every hit verified, unverified or failed verification, and report them separately.
Step 3: Turn hits into GEO signals
Useful derived metrics, computed on verified hits only:
- Crawl recency per money page. Days since each search-index bot last fetched your top product and guide URLs. A guide nobody has fetched cannot be cited.
- Block rate. Share of bot hits answered with
403,429or a challenge page. A jump usually means a new WAF rule, not a bot policy change. - User-fetch demand. Pages with rising
ChatGPT-UserorPerplexity-Userhits are being discussed in assistants right now — prioritise them for answer-first rewrites. - Coverage ratio. Sitemap URLs fetched by at least one search-index bot in the last 30 days, divided by sitemap URLs.
Write a robots.txt policy on purpose
Decide per purpose, then write it down:
# Allow AI search and user-initiated fetches; opt out of model training.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Then confirm in logs that the bots you allowed actually receive 200 responses. robots.txt permission is worthless if a CDN rule blocks the request first.
Why we never spoof user agents
It is tempting to "test as GPTBot" by sending its user-agent from your own laptop. Don't build a workflow on it: you are not on the operator's network, so your request proves nothing about what the real crawler receives, and impersonating another company's bot against third-party sites is deceptive.
BuyerCite's own crawler identifies itself as BuyerCiteBot/1.0, honours robots.txt, never solves CAPTCHAs or rotates residential proxies, and pauses a host after repeated blocks. For your own site, the honest test is the real thing: verified hits from the real crawlers, in your own logs.
A weekly AI crawler report
Five numbers per week are enough for most teams: verified hits per search-index bot, block rate, median crawl recency for the top 50 money pages, user-fetch hits by page, and the coverage ratio against your sitemap. Plot each against the previous four weeks; a sudden change is almost always caused by a deploy, a CDN rule or a robots.txt edit, so annotate the chart with releases.