Robots Exclusion Protocol — RFC 9309
The standards-track definition of robots.txt retrieval, matching, errors, and caching for automated clients.
Open standardReferenceReviewed 2026-07-25
Open official source How the Fieldbook uses it
We refer to Robots Exclusion Protocol — RFC 9309 in work involving discovery, seo and geo. Its own documentation remains the authority for installation, compatibility, and current behavior. A listing here does not imply endorsement by its maintainers.
Good fit for
- Crawler-access reviews
- Separating crawl controls from authorization
CrawlingSEOWeb standards
Before you adopt it
- Maintainer
- Internet Engineering Task Force
- License or usage boundary
- Internet Standards Track
Check the official source for the current release, support policy, compatibility notes, and license terms. The review date on this page only records when Beseam last checked the listing.
Related guides
No Fieldbook guide links to this entry yet. It is listed because it is still relevant to the category.