AI search
ClaudeBot
Anthropic's crawler. Used for training and Claude product context.
Definition
Anthropic operates ClaudeBot for general crawling, plus anthropic-ai and claude-web for related purposes. Honors robots.txt. Generally does not execute JavaScript — content available only after JS hydration is invisible to it.
More in AI search
Answer Engine Optimization (AEO)
Optimizing content to be cited inside AI-generated answers.
AI Overviews
Google's AI-generated answer block that appears above traditional search results.
llms.txt
A proposed file at site root that hints to AI crawlers what content to prioritize.
GPTBot
OpenAI's training crawler. Distinct from OAI-SearchBot.
Need this concept applied to your stack?
Glossary entries are intentionally short. Real engineering tradeoffs need a scoping call — bring the domain, the stack, and the question.