AI search & GEO

AI crawlers

AI crawlers are the bots that fetch web pages for AI companies — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended — for training, for retrieval, or for both.

Abstract 3D render visualizing artificial intelligence and neural networks in digital form, illustrating the SEO definition of AI crawlers
AI crawlers are the bots that fetch web pages for AI companies — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended — for training, for retrieval, or for both.Photo: Google DeepMind / Pexels

They are distinguishable in robots.txt, and the distinction matters: some fetch for model training, others fetch live to answer a question a user just asked. Blocking the first while allowing the second is a coherent position for a publisher; blocking everything is a coherent position for some business models and a costly accident for most others.

Two practical points. Most of these crawlers do not execute JavaScript, so client-side rendered content is invisible to them regardless of your robots.txt. And a blanket block often arrives by default from a security vendor or CDN rule rather than by decision — check what your infrastructure is actually doing before assuming your policy is what you intended.

Related terms

Guides on this subject