{"schemaVersion":"ready-scan.discovery-resource.v1","id":"urn:readyscan:discovery:ai-crawler-search-training-controls:zh-Hans","slug":"ai-crawler-search-training-controls","kind":"guide","inLanguage":"zh-Hans","canonicalUrl":"https://readyscan.ai/zh-Hans/insights/ai-crawler-search-training-controls","machineUrl":"https://readyscan.ai/zh-Hans/discovery/resources/ai-crawler-search-training-controls.json","title":"AI 爬虫控制：搜索、训练与用户请求的访问 | Ready Scan","description":"比较各供应商的爬虫控制，了解 robots.txt 的作用和限制，并提供不承诺索引或模型遗忘的检查流程。","h1":"AI 爬虫访问：分别处理搜索发现、模型训练和用户请求的访问","summary":"AI 访问并非单一设置。各供应商的控制可分别处理自动搜索抓取、可能用于未来模型训练的内容，以及因用户请求而进行的访问。Google-Extended 也涵盖指定的 Gemini grounding 用途。robots 指令不是身份验证；抓取记录或政策设置均不保证出现在搜索结果、获得引用，或从已训练模型中移除。","answer":{"question":"网站所有者能否分别控制 AI 爬虫的搜索、训练和用户请求访问？","text":"不少供应商会分别记录自动搜索抓取、可能用于模型训练或 grounding 的内容，以及用户触发的检索控制。控制效果因供应商而异：例如，OpenAI 分别说明 OAI-SearchBot、GPTBot 和 ChatGPT-User；Google 则表示 Google-Extended 控制指定的未来 Gemini 训练和 grounding 用途，但不影响 Google 搜索收录或排名。robots.txt 提供爬虫指令，并非访问授权。应分别核对相关供应商规则和实际页面传送情况；这些控制均不保证收录、引用、从已训练模型中删除内容，或获得所有爬虫遵守。","sourceIds":["openai-bots","anthropic-bots","google-crawlers","google-ai-features","robots-standard"]},"sections":[{"title":"一次页面访问，不能回答所有访问问题","body":"供应商可能为自动搜索发现、可能用于模型开发的内容，以及由个人发起的检索，使用不同爬虫或产品 token。这些用途各不相同，不应把其中一项控制当作通用的 AI 访问开关。Google-Extended 有一项重要区别：Google 说明它用于控制未来 Gemini 模型训练及指定的 grounding 用途，而不是控制 Google 搜索收录或排名。\n\n访问观测与回答结果应分别处理。抓取、检索、提及、引用、推荐和引流是不同的观测；其中一项的证据不能证明另一项，插图也不是有保证的流程。","sourceIds":["openai-bots","anthropic-bots","google-crawlers","google-ai-features"],"figure":{"src":"/assets/insights/ai-visibility-signals-zh-hans-1280.webp","width":1280,"height":720,"alt":"六个独立图格分别标有「抓取」、「检索」、「提及」、「引用」、「推荐」和「引流」，呈现不同的 AI 可见度观察。","caption":"抓取、检索、提及、引用、推荐和引流，是不同的观察。看到其中一项，不代表已出现其他项目。概念示意图。","variants":[{"src":"/assets/insights/ai-visibility-signals-zh-hans-640.webp","width":640}]}},{"title":"供应商控制及其已说明的界限","body":"下表概括各项控制所述的用途，并非适用于所有爬虫或网站的通用规则。HTTP 请求中的 user-agent 字符串，不一定等同 robots.txt 组所用的产品 token。Google-Extended 有 robots.txt 控制 token，但没有独立的 HTTP 请求 user-agent。应按相关主机和用途核对供应商文档；各项已记录的选择及例外不能互换。","sourceIds":["openai-bots","anthropic-bots","google-crawlers","google-ai-features"],"table":{"caption":"已记录的控制及重要限制","columns":["供应商控制","已说明的用途","重要界限"],"rows":[["OpenAI OAI-SearchBot","供 ChatGPT 搜索功能自动发现网站内容。","OpenAI 表示，拒绝 OAI-SearchBot 的网站不会出现在 ChatGPT 搜索回答中，但仍可能以导航链接形式出现。"],["OpenAI GPTBot","抓取可能用于训练 OpenAI 基础模型的内容。","OpenAI 说明 GPTBot 与 OAI-SearchBot 相互独立。禁止 GPTBot 表示内容不应用于这项训练用途；但不能据此确定内容已从已训练模型中移除。"],["OpenAI ChatGPT-User","因 ChatGPT 或 Custom GPT 用户操作而发生的部分访问；并非自动网络抓取。","OpenAI 表示，robots.txt 规则可能不适用于这些由用户发起的请求。此控制不管理搜索退出设置；OAI-SearchBot 才是已说明的搜索控制。"],["Anthropic Claude-SearchBot","改善用户搜索的结果质量及索引。","Anthropic 表示，停用它会阻止为搜索优化而索引内容，并可能降低网站在用户搜索结果中的可见度或准确度。"],["Anthropic ClaudeBot","收集可能用于模型训练的网页内容。","Anthropic 表示，限制访问意味着未来资料应排除在其 AI 模型训练数据集之外；这并非对已训练模型的声明。"],["Anthropic Claude-User","根据用户查询获取网页内容。","Anthropic 表示，停用它会阻止这类检索，并可能降低网站在用户主导的网络搜索中的可见度。Anthropic 说明其爬虫会遵守 robots.txt 指令。"],["Google Googlebot","Google 搜索抓取，包括 AI Overviews 和 AI Mode。","Google 指明 Googlebot 是搜索访问控制。如果把它当作只控制训练，也会影响搜索抓取；允许抓取本身不保证页面会出现在搜索结果。"],["Google Google-Extended","控制已抓取内容能否用于未来 Gemini 模型训练，以及 Gemini Apps 和 Vertex AI 的 Grounding with Google Search 中指定的 grounding 用途。","它不影响 Google 搜索收录或排名，也没有独立的 HTTP 请求 user-agent。Google 说明此 token 控制未来用途，并非将内容从已训练模型中移除。"]]}},{"title":"robots.txt 是爬虫指令，不是门锁","body":"Robots Exclusion Protocol 要求自动爬虫遵守有关路径访问的规则。RFC 9309 明确指出，这些规则并非访问授权。robots.txt 文件可供公众读取；列出敏感路径可能让路径更容易被发现，而不是让其保密。如果内容需要限制访问，应使用身份验证等应用层访问控制。\n\n应分别检查政策文本、身份验证和实际传送情况。即使 robots 规则允许访问，主机或 CDN 设置仍可能阻止爬虫读取页面；robots 指令本身也不会阻止请求发出。供应商文档会说明特定的遵守方式，但不应把指令视为所有客户端都会遵守的保证。请求中的爬虫名称本身，不能证明其身份。","sourceIds":["robots-standard","google-crawlers","openai-bots","anthropic-bots"]},{"title":"修改设置前的检查流程","body":"按以下顺序厘清拟议修改的用途。此流程不会更改网站设置，也不能判定页面会否出现在搜索结果或回答中。","sourceIds":["robots-standard","google-ai-features","openai-bots","anthropic-bots"],"items":[{"title":"列明内容和用途","body":"指出具体的公开页面，以及目标涉及搜索发现、可能的训练用途、grounding，还是用户请求的检索。不要把「AI 访问」视为单一用途。"},{"title":"核对相应的供应商规则","body":"阅读供应商文档，以及相关主机的 robots.txt 组。先检查现有规则，不要照抄全面封锁指令；并核对已说明的控制是 HTTP user-agent 还是 robots 产品 token。"},{"title":"分别检查实际传送和保护措施","body":"检查相关页面和 robots.txt 是否实际可供读取，包括主机或 CDN 的限制。私人或仅供账户使用的资料应以适当访问控制保护；robots 规则不能替代这些措施。"},{"title":"记录实际观测，不要记录推测","body":"记录已检查的规则和实际观测到的访问结果。曾被获取不代表已收录于搜索结果或获得引用。训练退出信号不代表内容已从已训练模型中删除；一家供应商所述的遵守方式也不代表普遍适用。"}]}],"sources":[{"id":"openai-bots","title":"Overview of OpenAI Crawlers","publisher":"OpenAI","url":"https://developers.openai.com/api/docs/bots","accessedOn":"2026-10-10"},{"id":"anthropic-bots","title":"Does Anthropic crawl data from the web, and how can site owners block the crawler?","publisher":"Anthropic","url":"https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler","accessedOn":"2026-10-10"},{"id":"google-crawlers","title":"List of Google common crawlers","publisher":"Google for Developers","url":"https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers","accessedOn":"2026-10-10"},{"id":"google-ai-features","title":"AI features and your website","publisher":"Google Search Central","url":"https://developers.google.com/search/docs/appearance/ai-features","accessedOn":"2026-10-06"},{"id":"robots-standard","title":"RFC 9309: Robots Exclusion Protocol","publisher":"IETF","url":"https://www.rfc-editor.org/rfc/rfc9309.html","accessedOn":"2026-10-10"}],"ownership":{"state":"ready_owned","publisher":{"name":"Ready Scan","url":"https://readyscan.ai"},"verificationMethod":"first_party_publication","verificationSource":"https://readyscan.ai/zh-Hans/insights/ai-crawler-search-training-controls"},"datePublished":"2026-10-10","dateModified":"2026-10-10","related":[{"id":"urn:readyscan:discovery:ai-citation-visibility:zh-Hans","canonicalUrl":"https://readyscan.ai/zh-Hans/ai-citation-visibility","machineUrl":"https://readyscan.ai/zh-Hans/discovery/resources/ai-citation-visibility.json"},{"id":"urn:readyscan:discovery:website-updates-search-ai-answers:zh-Hans","canonicalUrl":"https://readyscan.ai/zh-Hans/insights/website-updates-search-ai-answers","machineUrl":"https://readyscan.ai/zh-Hans/discovery/resources/website-updates-search-ai-answers.json"},{"id":"urn:readyscan:discovery:geo-aeo-seo:zh-Hans","canonicalUrl":"https://readyscan.ai/zh-Hans/insights/geo-aeo-seo","machineUrl":"https://readyscan.ai/zh-Hans/discovery/resources/geo-aeo-seo.json"},{"id":"urn:readyscan:source:source-2d8d883ba8f289ac:zh-Hans","canonicalUrl":"https://readyscan.ai/zh-Hans/sources/source-2d8d883ba8f289ac","machineUrl":"https://readyscan.ai/zh-Hans/discovery/sources/source-2d8d883ba8f289ac.json"},{"id":"urn:readyscan:source:source-0e8eaf1d95acd8d8:zh-Hans","canonicalUrl":"https://readyscan.ai/zh-Hans/sources/source-0e8eaf1d95acd8d8","machineUrl":"https://readyscan.ai/zh-Hans/discovery/sources/source-0e8eaf1d95acd8d8.json"},{"id":"urn:readyscan:source:source-8f6bb09bbba1b2d8:zh-Hans","canonicalUrl":"https://readyscan.ai/zh-Hans/sources/source-8f6bb09bbba1b2d8","machineUrl":"https://readyscan.ai/zh-Hans/discovery/sources/source-8f6bb09bbba1b2d8.json"},{"id":"urn:readyscan:source:source-0a5f4e0044c75068:zh-Hans","canonicalUrl":"https://readyscan.ai/zh-Hans/sources/source-0a5f4e0044c75068","machineUrl":"https://readyscan.ai/zh-Hans/discovery/sources/source-0a5f4e0044c75068.json"},{"id":"urn:readyscan:source:source-d871053673d1c958:zh-Hans","canonicalUrl":"https://readyscan.ai/zh-Hans/sources/source-d871053673d1c958","machineUrl":"https://readyscan.ai/zh-Hans/discovery/sources/source-d871053673d1c958.json"}],"structuredData":{"@context":"https://schema.org","@type":"Article","headline":"AI 爬虫控制：搜索、训练与用户请求的访问 | Ready Scan","description":"比较各供应商的爬虫控制，了解 robots.txt 的作用和限制，并提供不承诺索引或模型遗忘的检查流程。","mainEntityOfPage":"https://readyscan.ai/zh-Hans/insights/ai-crawler-search-training-controls","datePublished":"2026-10-10","dateModified":"2026-10-10","publisher":{"@type":"Organization","name":"Ready Scan","url":"https://readyscan.ai"},"author":{"@type":"Organization","name":"Ready Scan","url":"https://readyscan.ai"},"citation":["https://developers.openai.com/api/docs/bots","https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler","https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers","https://developers.google.com/search/docs/appearance/ai-features","https://www.rfc-editor.org/rfc/rfc9309.html"],"image":{"@type":"ImageObject","url":"https://readyscan.ai/assets/insights/ai-visibility-signals-zh-hans-1280.webp","width":1280,"height":720,"caption":"抓取、检索、提及、引用、推荐和引流，是不同的观察。看到其中一项，不代表已出现其他项目。概念示意图。","description":"六个独立图格分别标有「抓取」、「检索」、「提及」、「引用」、「推荐」和「引流」，呈现不同的 AI 可见度观察。"},"inLanguage":"zh-Hans"}}