AI crawler access is not one switch
Separate controls for search discovery, possible model-training use and visits triggered by a user. The available controls and their effects differ by provider.
Robots Refresher: robots.txt — a flexible way to control how machines explore your website | Google Search Central Blog | Google for Developers
Google Search Centraldevelopers.google.com