Robots Refresher: robots.txt — a flexible way to control how machines explore your website  |  Google Search Central Blog  |  Google for Developers

Google Search Centraldevelopers.google.com