---
type: Glossary Term
title: "RFC 9309 (Robots Exclusion Protocol)"
description: "The IETF standard that formally specifies robots.txt parsing and behavior."
resource: "https://prerendering.com/glossary/rfc-9309"
tags: [standards, glossary]
timestamp: 2026-05-11T00:00:00Z
---

# RFC 9309 (Robots Exclusion Protocol)

The IETF standard that formally specifies robots.txt parsing and behavior.

Published 2022. Codifies the long-de-facto robots.txt format: User-agent grouping, Allow/Disallow rules, longest-match precedence. Defines that robots.txt blocks crawling but not indexation — a page reachable via external link can still be indexed despite a Disallow rule, unless also blocked by noindex.