AI Bot Log Analysis: Find AI Crawlers in Your Logs
Your analytics dashboard is blind to the most interesting visitors you have. GPTBot doesn't run JavaScript, so it never fires a tracking tag. ChatGPT fetching your pricing page on behalf of a user who asked "is this tool worth it" leaves no trace in GA4. The only place this traffic exists is your server access logs, and most sites never look.
That's a shame, because the logs answer questions nothing else can: Is OpenAI training on my content? Do AI assistants actually read my pages when users ask about my category? Is that aggressive crawler really ClaudeBot, or a scraper wearing its name? Here's how to find out with nothing fancier than grep and awk.
The AI bot user agents worth grepping
These are the user agent substrings that matter in 2026, grouped by operator:
- GPTBot — OpenAI's training crawler. Collects content for model training.
- OAI-SearchBot — OpenAI's search index crawler. Feeds ChatGPT search results.
- ChatGPT-User — fires when ChatGPT visits your page live because a user's question needed it.
- PerplexityBot — Perplexity's index crawler. Its sibling Perplexity-User handles live, user-triggered fetches.
- ClaudeBot — Anthropic's training crawler, joined by Claude-User (live fetches) and Claude-SearchBot.
- Bytespider — ByteDance's crawler. Aggressive, historically careless about robots.txt.
- CCBot — Common Crawl. Not an AI company itself, but its corpus feeds many training runs, so blocking or allowing it is an AI decision.
One trap worth calling out: Google-Extended is not a crawler. It's a robots.txt token that tells Google not to use content for Gemini training. The actual fetching is done by regular Googlebot, so grepping your logs for "Google-Extended" returns nothing, ever. People burn real hours on this. Applebot-Extended works the same way. If you're deciding which of these bots to allow or block in the first place, our robots.txt guide covers that side.
A real GPTBot line in an nginx combined log looks like this:
20.171.207.14 - - [07/Aug/2026:09:14:22 +0000] "GET /pricing HTTP/1.1" 200 18204 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot"
Training crawls vs user-triggered fetches
This distinction is the whole reason the analysis is worth doing, so don't collapse everything into one "AI bots" bucket.
Training and index crawlers (GPTBot, ClaudeBot, CCBot, PerplexityBot, OAI-SearchBot) behave like classic search spiders: broad sweeps, many URLs, steady patterns, no connection to any live user. High GPTBot volume means OpenAI finds your content worth ingesting. It says nothing about whether anyone sees it.
User-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) are different animals entirely. Each hit means a human, right now, asked an AI assistant something that made it open your page. This is the closest thing AI search has to real-time demand data. A spike in ChatGPT-User hits on a specific article means real users are asking questions your page answers — and the assistant is reading your version of the answer before it replies.
Treat the first group as a supply signal (are the models fed?) and the second as a demand signal (are the assistants using it?). A site with heavy GPTBot traffic and zero ChatGPT-User traffic is being archived, not consulted.
A practical grep and awk workflow for AI bot log analysis
Everything below assumes an nginx or Apache combined log at access.log. Adjust the path; the field positions hold.
First, the headcount — how many hits per AI bot:
grep -iE 'gptbot|oai-searchbot|chatgpt-user|perplexitybot|perplexity-user|claudebot|claude-user|bytespider|ccbot' access.log \
| awk -F'"' '{print $6}' \
| grep -ioE 'gptbot|oai-searchbot|chatgpt-user|perplexitybot|perplexity-user|claudebot|claude-user|bytespider|ccbot' \
| sort | uniq -c | sort -rn
The -F'"' tells awk to split on double quotes, which makes $6 the user agent field in combined format. Output looks like:
1842 gptbot
311 claudebot
204 bytespider
97 chatgpt-user
23 perplexitybot
Next, what a specific bot is actually reading. Top 20 URLs fetched by ChatGPT-User:
grep -i 'chatgpt-user' access.log \
| awk '{print $7}' \
| sort | uniq -c | sort -rn | head -20
Daily trend for one bot, using the timestamp field:
grep -i 'gptbot' access.log \
| awk '{print substr($4, 2, 11)}' \
| sort | uniq -c
And the one people forget — status codes. If a bot is drowning in 404s or 403s, it's reading nothing useful:
grep -i 'gptbot' access.log | awk '{print $9}' | sort | uniq -c | sort -rn
If your logs rotate, run everything against access.log* with zgrep for the gzipped ones. Ten minutes of this per month is more AI visibility insight than most dashboards will give you.
Two caveats before you trust the numbers. First, if you sit behind a CDN or proxy, the $1 field is probably the proxy's IP, not the bot's — look for an X-Forwarded-For value in your log format, or pull the report from the CDN's own logs instead. Second, check your retention. Plenty of default setups keep 14 days of logs and silently delete the rest, which makes month-over-month comparison impossible. If AI crawl trends matter to you, archive the rotated files somewhere cheap before logrotate eats them.
Verifying real AI bots against spoofed user agents
A user agent string is a text field anyone can type. Scrapers routinely claim to be GPTBot or ClaudeBot because sites are likelier to let those names through. Before you make decisions on the numbers — or firewall someone off — verify.
The major operators publish their egress IP ranges as JSON:
- OpenAI:
https://openai.com/gptbot.json,https://openai.com/searchbot.json, andhttps://openai.com/chatgpt-user.json - Perplexity:
https://www.perplexity.ai/perplexitybot.jsonandhttps://www.perplexity.ai/perplexity-user.json
Pull the claimed GPTBot IPs from your logs and check them against the published list:
# IPs claiming to be GPTBot
grep -i 'gptbot' access.log | awk '{print $1}' | sort -u > claimed.txt
# Official ranges
curl -s https://openai.com/gptbot.json \
| jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix' > ranges.txt
# Anything printed here is a spoofer
grepcidr -vf ranges.txt claimed.txt
grepcidr is in most package managers (apt install grepcidr); the -v flag prints IPs that fall outside every published range. For Googlebot, Bingbot, and Applebot, use reverse DNS instead: host <ip> should resolve to the operator's domain, and the forward lookup of that hostname should return the same IP. Not every operator makes this easy — Bytespider publishes no ranges at all, which tells you something about how much weight to give a Bytespider UA string. When there's no published list, the honest options are treating the traffic as unverified or leaning on your CDN's verified-bots feature (Cloudflare maintains one) to do the vouching.
The payoff is real: it's common to find a third or more of "GPTBot" traffic failing verification. Those are scrapers to block, not AI visibility to celebrate.
Reading the results — a hypothetical month
Say you run an online store for espresso gear and pull one month of logs. The breakdown: 2,100 verified GPTBot hits sweeping your entire catalog, 340 ClaudeBot hits doing the same, 95 ChatGPT-User hits — and 80 of those 95 land on a single page, your grinder comparison guide. Meanwhile 600 "GPTBot" hits fail IP verification and trace back to a hosting provider in odd places.
That one afternoon of grep just told you four things. Models are ingesting your catalog (supply: fine). Real ChatGPT users are asking grinder questions and the assistant reads your guide to answer them (demand: strong, and concentrated). Your product pages get crawled but never live-fetched, so nobody's asking assistants about your house-brand machines yet — a content gap, not a crawling problem. And a scraper is stealing your catalog under OpenAI's name — a firewall rule, not a bragging right. No analytics suite sees any of this.
Log analysis shows AI systems reading your site; it can't show whether they cite you or send people your way afterward. For that half of the picture, check your AI-driven referral traffic with our AI referral check — the two views together tell you whether getting read is turning into getting recommended.
This is one of eight metrics in the complete AI visibility measurement playbook.
