---
type: Glossary Term
title: "llms.txt"
description: "A proposed file at site root that hints to AI crawlers what content to prioritize."
resource: "https://prerendering.com/glossary/llms-txt"
tags: [ai-search, glossary]
timestamp: 2026-05-11T00:00:00Z
---

# llms.txt

A proposed file at site root that hints to AI crawlers what content to prioritize.

A markdown-formatted file at /llms.txt that lists the most important pages on a site for AI crawlers, with structured descriptions. Inspired by robots.txt and sitemap.xml. Adoption is uneven: some AI engines respect it as a hint; most do not enforce it strongly. Worth shipping for forward compatibility.

## When it matters

- Sites with > 50 indexed pages where AI crawlers struggle to identify canonical content.
- Technical or topical sites where a clear hierarchy (overview, glossary, deep-dives) helps citation accuracy.
- Brands with a distinct knowledge graph worth surfacing to answer engines — products, docs, methodology.
- After major architecture changes (URL migration, content reorg) to signal new canonical structure.

## Common pitfalls

- Listing every URL — defeats the purpose; keep it to the 20-50 entries you most want cited.
- Treating llms.txt as a guarantee — most AI engines treat it as a hint, not a directive.
- Letting it drift out of sync with the sitemap — confuses crawlers that cross-check.
- Forgetting to serve it as `text/plain; charset=utf-8` — some bots reject other content types.

## How to verify

- curl -I https://your-site.com/llms.txt — must return 200 with `Content-Type: text/plain`.
- Server logs: confirm GPTBot, ClaudeBot, PerplexityBot fetch /llms.txt at least once per week.
- Test prompts in ChatGPT and Perplexity that reference the canonical content listed — verify citations resolve to those URLs.
- Compare /llms.txt entries against /sitemap.xml — flagged entries should be a strict subset of indexable URLs.

## External references

- [llmstxt.org — Specification](https://llmstxt.org/)
- [darkvisitors.com — AI crawler catalogue](https://darkvisitors.com/)

## Related

- [LLMs.txt and AI Crawl Directives](/blog/llms-txt-and-ai-crawl-directives.md)