Standards
RFC 9309 (Robots Exclusion Protocol)
The IETF standard that formally specifies robots.txt parsing and behavior.
Definition
Published 2022. Codifies the long-de-facto robots.txt format: User-agent grouping, Allow/Disallow rules, longest-match precedence. Defines that robots.txt blocks crawling but not indexation — a page reachable via external link can still be indexed despite a Disallow rule, unless also blocked by noindex.
More in Standards
hreflang
A signal that tells search engines which language and region version of a page to serve.
JSON-LD
JSON-formatted structured data that Google and AI engines extract for entity understanding.
RFC 9110 (HTTP Semantics)
The current IETF standard defining HTTP semantics: methods, status codes, headers.
Need this concept applied to your stack?
Glossary entries are intentionally short. Real engineering tradeoffs need a scoping call — bring the domain, the stack, and the question.