RFC 9309: Robots Exclusion Protocol

These published guides cite this source. Read the original, or follow a guide to see how it is used.

IETFRead the original source

Publications using this source: 1 publication

  1. AI crawler access is not one switch

    Separate controls for search discovery, possible model-training use and visits triggered by a user. The available controls and their effects differ by provider.

Read the JSON companion