Your server logs already contain a record of every AI system that has read your site, and most teams have never looked at it. I understand why, because log files are ugly and reading them is nobody's job, but the volume has grown to the point where Cloudflare now publishes a dedicated bot report about it, and a log line is evidence you can verify yourself, since a model either fetched your pricing page or it did not, and the line says which.

Reading those logs well comes down to one idea, which is that every major AI vendor now runs up to three different crawlers, and each one means a different thing for your brand. Once you can tell them apart, a morning with grep tells you a lot about your AI presence, and this article will show you how to find each one, what the patterns mean, and which pages to fix first.

The three kinds of AI crawler

Training crawlers collect pages to build future models. OpenAI's GPTBot, Anthropic's ClaudeBot, Meta-ExternalAgent, ByteDance's Bytespider, which is reported to be used for ByteDance model training, and Common Crawl's CCBot, the open-archive crawler whose dataset is used for training, are in this group, and a visit from any of them today influences what a model knows next year, which is why their traffic looks slow and broad.

Search index crawlers build the retrieval layer that assistants search when they answer. OAI-SearchBot for ChatGPT search and Claude-SearchBot for Claude are the main ones, and a visit here means your page can be found and cited in answers, long before any future model generation ships.

User triggered fetchers are the ones I would read first. ChatGPT-User, Claude-User, and Perplexity-User fire when a real person's question sends the assistant to your page right now, so behind every one of those lines there is a human, mid conversation, whose answer is about to be assembled partly from whatever your page says in that moment. These fetchers carry real volume now, because in Cloudflare Radar's July 2026 crawler share, as reported by SEOmator, Claude-User accounted for 10.1 percent of crawler traffic, which put a live answer fetcher ahead of Bingbot and Applebot.

For scale on the crawler side, Cloudflare wrote on July 1, 2026 that 52 percent of crawler requests on its network were for AI training as of June 2026, up from 22 percent in spring 2025, so the training group above has gone from a minority of crawl traffic to the majority in a little over a year.

Google is the odd one out. There is no separate Google AI crawler to find in your logs, because ordinary Googlebot feeds AI Overviews and AI Mode, while Google-Extended is only a robots.txt token that opts your content out of Gemini training and never visits a page itself.

Finding them in your logs

Pull a week of access logs and search the user agent field for the bot names. The strings identify themselves, in the style of, for example, compatible; GPTBot/1.4; +https://openai.com/gptbot or compatible; ClaudeBot/1.0; +claudebot@anthropic.com, and the version numbers change over time, so match on the name and leave the number out of the pattern. Searching for these covers the traffic that matters in 2026: 

# Every AI crawler hit in the file
grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Agent|Meta-ExternalAgent|Amazonbot|Bytespider|CCBot' access.log

# Just the live-answer fetchers, and which pages they read most
grep -Ei 'ChatGPT-User|Claude-User|Perplexity-User|Google-Agent' access.log |
  awk '{print $7}' | sort | uniq -c | sort -rn | head -20

That second command assumes the combined log format, where the requested path is the seventh field, so adjust the column to match whatever your server writes. Two newer strings sit in that first pattern on purpose: OAI-AdsBot, which OpenAI has run since May 2026 to visit the landing pages submitted for ads and which is not governed by robots.txt, and Google-Agent, Google's user triggered fetcher.

User agents are just text, and plenty of scrapers send the GPTBot string without being GPTBot. The documented check for the AI vendors is their published IP range file, OpenAI's gptbot.json, searchbot.json and chatgpt-user.json, Anthropic's bots.json, and Perplexity's perplexitybot.json and perplexity-user.json, so my advice is to match the visiting IP against the right file before you trust a line. Reverse DNS is the documented method for Googlebot and Bingbot, where you resolve the IP to a hostname, check the hostname belongs to the vendor, and confirm it resolves forward to the same IP. If a "GPTBot" comes from a residential ISP range, it is somebody's scraper, and blocking it costs you nothing.

What the patterns mean

Cloudflare now measures how many pages each AI operator crawls for every referral visit it sends back, and it puts that range across operators at 118:1 up to nearly 50,000:1. I believe you should calibrate your expectations against that exchange rate before you read anything else in the log, because in the 28 day window ending July 21, 2026, as SEOmator reports the Cloudflare Radar figures, OpenAI crawled about 217 pages for every visitor it referred, Perplexity about 225, and Anthropic about 2,237, against roughly 4.6 for Google. Cloudflare adds its own caveat, which is that referrals from the Claude app send no Referer header, so Anthropic's ratio is overstated.

That caveat applies more widely than Anthropic. User fetchers arrive with no Referer header, so a fetch never shows up as a referral in your analytics, and one April 2026 nginx study saw ChatGPT-User, Claude-User and Perplexity-User all arrive referrer-less while Gemini produced no identifiable request at all, which is why these ratios overstate the gap between crawling and value.

A ChatGPT-User visit puts your page in an answer a buyer is reading right now.

There is money at the end of that chain even without the click, since Adobe measured AI referred traffic to US retail sites up 393 percent year over year in Q1 2026, with AI sourced visitors converting about 42 percent better than non-AI visitors in March. For the human side of that scale, TollBit measured one AI bot visit for every 31 human visits by Q4 2025, up from 1 in 200 at the start of that year. That is the same shift we mapped out in GEO is the new SEO, seen from the log file end.

These ratios also change fast, which is a reason to watch trends over snapshots. In SEOmator's own 500 site panel, Anthropic's ratio fell from roughly 57,000:1 in January 2026 to about 2,300:1 by July, an order of magnitude in six months, so a ratio you read last quarter says little about what this week's logs will show.

Past the exchange rate, look at which crawler types show up together. A site with heavy GPTBot traffic and no OAI-SearchBot visits is being read for training while staying thin in the retrieval layer, which leaves you trainable but unciteable in ChatGPT search, and the fixes for that live in why ChatGPT does not mention your company.

The reverse pattern, search crawlers present but user fetchers absent, means you are indexed without being pulled into live answers, which most often means the models can find you and do not need you for that question.

User triggered fetches are the signal I would watch weekly. Every ChatGPT-User hit on your pricing page is a person who asked an assistant something that made it check your pricing, at that moment, in that conversation, and these hits are the closest thing AI search has to intent data, because they tell you which pages are being read to buyers. Those pages deserve your accuracy budget first, and if the pattern surprises you, say a demo request page fetched constantly while your comparison page never is, you have found the page to go and fix.

Check what the crawlers received as well. A page that renders everything client side can return a near empty document to a bot, so the assistant reads a header and a cookie notice where you see a product page, and the same rule we described in what is cloaking in SEO applies in reverse here, since the version of your page a machine receives has to say the same things as the version a person sees.

Should you block any of them?

Some of them, maybe. Plenty of sites have made that choice, and Cloudflare reported that more than one million of its customers had turned on AI crawler blocking by July 2025, with GPTBot the single most blocked agent in robots.txt files by Cloudflare's own count. My issue is with treating the three crawler types as one decision, because blocking training bots in robots.txt is a defensible call for content businesses that do not want to feed future models, while blocking search index crawlers and user fetchers takes you out of the answers your buyers are already reading, which is a large price for one robots.txt line.

The robots.txt part of that decision only reaches some of the crawlers. The training and search crawlers from OpenAI, Anthropic, Google, Meta and Amazon honor it, while the user triggered fetchers do not have to, since OpenAI's bots page says of ChatGPT-User that "because those actions are initiated by a user, robots.txt rules may not apply", Perplexity's docs say Perplexity-User "generally ignores robots.txt rules", and Google says the same for Google-Agent, with Anthropic's Claude-User the one user fetcher documented as honoring it. Compliance is also uneven in practice, since Cloudflare caught Perplexity running undeclared crawlers around no-crawl rules in August 2025 and removed its verified bot status, and TollBit's April 2026 data put about 30 percent of AI bot scrapes in late 2025 on explicitly disallowed URLs, with ChatGPT-User leading at 42 percent. Bytespider is widely blocked, and if you want it gone, block it by IP or firewall rule.

Cloudflare now sorts crawlers into Search, Agent and Training buckets, and its Bot Preference Sync writes that split into robots.txt, which is the per crawler decision I am arguing for here. Whatever you decide, decide it per crawler and write it down, because a block-all rule from 2023 now blocks the search crawlers and the user fetchers too.

Doing this without the grep

Everything above works with raw logs and an hour. The weakness of the manual version is that it gives you a snapshot, and what you need is the trend, such as which pages the fetchers read this week that they skipped last week, and that is the gap AI Traffic closes, since it watches the recognized AI crawlers hit your pages and keeps the history so the diff is already there. Our team thinks you should start with the manual pass either way, because knowing what the raw lines look like makes every dashboard, ours included, much harder to mislead you.

FAQ

What is the difference between GPTBot and ChatGPT-User?

GPTBot collects pages for training future OpenAI models. ChatGPT-User fetches a page live because a person's question sent ChatGPT there mid conversation. The first one changes what a model knows next year, while the second one changes an answer a buyer is reading today.

Why is there no Google AI crawler in my logs?

Google uses its normal Googlebot crawl for AI Overviews and AI Mode. Google-Extended is only a robots.txt opt-out token for Gemini training, and it never visits pages.

How do I know a crawler is really from OpenAI or Anthropic?

Check the visiting IP against the vendor's published range file: OpenAI publishes gptbot.json, searchbot.json and chatgpt-user.json, Anthropic publishes bots.json, and Perplexity publishes perplexitybot.json and perplexity-user.json. Reverse DNS, resolving the IP to a hostname and back, is the documented method for Googlebot and Bingbot. User agent strings alone are trivially faked.

Do AI crawlers respect robots.txt?

The training and search crawlers from OpenAI, Anthropic, Google, Meta and Amazon honor it. User triggered fetchers such as ChatGPT-User, Perplexity-User and Google-Agent are documented as not bound by it because a person initiated the request, while Anthropic says Claude-User honors it. Bytespider is widely blocked by IP or firewall rule instead.

Which pages should I check first?

The ones user fetchers hit most. Those are the pages being read into live answers, so a wrong price or stale claim there reaches buyers immediately.

Keep reading

See which AI crawlers read your site this week

Ooky captures recognized AI crawler requests as they arrive, groups them into sessions, and keeps the history, so the week over week diff is already there when you go looking. The raw log pass still works, and it is where I would start.

See how AI Traffic works See the delivery options

Sources

  1. Cloudflare. "The crawl before the fall of referrals: understanding AI's impact on content providers." Crawl-to-refer methodology post, July 2025. Retrieved 2026-08-24. https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/
  2. Cloudflare. "The agentic internet: Cloudflare's 2026 bot report." July 1, 2026. Retrieved 2026-08-24. https://blog.cloudflare.com/agentic-internet-bot-report/
  3. Cloudflare. "Attribution and business insights for AI crawl traffic." 2026. Retrieved 2026-08-24. https://blog.cloudflare.com/attribution-business-insights/
  4. Cloudflare. "From Googlebot to GPTBot: who's crawling your site in 2025." 2025. Retrieved 2026-08-24. https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
  5. Cloudflare. "Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives." August 2025. Retrieved 2026-08-24. https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/
  6. Cloudflare. "Cloudflare just changed how AI crawlers scrape the internet at large." Press release, July 1, 2025. Retrieved 2026-08-24. https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/
  7. Cloudflare. "Bot Preference Sync: Search, Agent and Training buckets in robots.txt." 2026. Retrieved 2026-08-24. https://blog.cloudflare.com/bot-preference-sync/
  8. OpenAI. "Overview of OpenAI crawlers." Developer documentation, GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot and IP range files. Retrieved 2026-08-24. https://developers.openai.com/api/docs/bots
  9. Anthropic. "Does Anthropic crawl data from the web, and how can site owners block the crawler?" Help Center. Retrieved 2026-08-24. https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
  10. Perplexity. "Perplexity crawlers." Developer documentation, PerplexityBot and Perplexity-User. Retrieved 2026-08-24. https://docs.perplexity.ai/guides/bots
  11. Google. "Google's common crawlers." Search Central documentation. Retrieved 2026-08-24. https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
  12. Google. "User-triggered fetchers." Search Central documentation, Google-Agent. Retrieved 2026-08-24. https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers
  13. Google. "AI features and your website." Search Central documentation. Retrieved 2026-08-24. https://developers.google.com/search/docs/appearance/ai-features
  14. Meta. "Meta web crawlers." Developer documentation, Meta-ExternalAgent. Retrieved 2026-08-24. https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
  15. Amazon. "Amazonbot." Developer documentation. Retrieved 2026-08-24. https://developer.amazon.com/amazonbot
  16. TollBit. "Google-Agent and robots.txt: what the Q4 2025 data shows." April 2026. Retrieved 2026-08-24. https://tollbit.com/blog/google-agent-robots-txt/
  17. Adobe. "AI traffic surge hits retail sites that are not machine readable." Adobe Digital Insights, April 2026. Retrieved 2026-08-24. https://business.adobe.com/blog/ai-traffic-surge-retail-sites-not-machine-readable
  18. TechCrunch. "AI traffic to US retailers rose 393% in Q1, and it's boosting their revenue too." April 16, 2026. Retrieved 2026-08-24. https://techcrunch.com/2026/04/16/ai-traffic-to-us-retailers-rose-393-in-q1-and-its-boosting-their-revenue-too/
  19. SEOmator. "GEO Data Report 2026: crawl-to-refer ratios for AI crawlers." Reporting Cloudflare Radar's 28 day window ending July 21, 2026, plus its own 500 site panel. Retrieved 2026-08-24. https://seomator.com/blog/crawl-to-refer-ratio-ai-crawlers-llm-bots
  20. Surfaced By. "nginx logs: AI traffic vs referral traffic." April 2026. Retrieved 2026-08-24. https://surfacedby.com/blog/nginx-logs-ai-traffic-vs-referral-traffic
  21. Digital Applied (secondary). "AI crawler and bot traffic statistics 2026." Compiles Cloudflare Radar and Imperva figures. Retrieved 2026-08-24. https://www.digitalapplied.com/blog/ai-crawler-bot-traffic-statistics-2026-data-reference
  22. HoneyB (secondary). "AI crawler user agents reference 2026." User agent strings. Retrieved 2026-08-24. https://www.honeyb.ai/blog/ai-crawler-user-agents-reference-2026