IETF - RFC 9309
Robots Exclusion Protocol
active, Most Current
| Organization: | IETF |
| Publication Date: | 1 September 2022 |
| Status: | active |
| Page Count: | 12 |
scope:
This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.
Document History
RFC 9309
September 1, 2022
Robots Exclusion Protocol
This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be...