---
type: Glossary Term
title: "robots.txt"
description: "A file at site root that tells crawlers which paths they may or may not fetch."
resource: "https://prerendering.com/glossary/robots-txt"
tags: [crawl, glossary]
timestamp: 2026-05-11T00:00:00Z
---

# robots.txt

A file at site root that tells crawlers which paths they may or may not fetch.

Defined by RFC 9309 (Robots Exclusion Protocol). Lives at /robots.txt. Each crawler reads it and is expected to honor Allow/Disallow rules. Blocking a path in robots.txt prevents crawling but does not guarantee non-indexation — pages can still be indexed from external links unless also marked noindex.