An AI crawler robots.txt policy should not begin with a copied list of bot names. It should begin with the outcome the organization wants: visibility in AI-assisted search, exclusion from model development, support for a user-requested fetch, or protection of content that should never be public. Those outcomes are different, and the controls are not interchangeable.
The practical answer is to classify each crawler by purpose, decide what public content it may fetch, and reserve authentication for anything private. A blanket allow can approve uses the team never discussed. A blanket block can remove a useful discovery path without securing a single sensitive page.
The crawler names and vendor guidance below were reviewed on August 4, 2026. They can change, so treat the policy as a maintained production setting rather than a one-time SEO edit.
Start with the outcome, not the bot name
The Robots Exclusion Protocol standard gives crawlers a common way to discover rules for a site. It does not define every business purpose behind a fetch. A site owner therefore needs a policy layer above the syntax.
Separate four questions before editing the file:
- Discovery: should public pages be eligible to appear as links or citations in an AI-assisted search experience?
- Model development: may a vendor collect public pages for possible use in improving or training foundation models?
- User-requested retrieval: may a product fetch a page because a person supplied a URL or asked a tool to visit it?
- Product validation: does a specialized bot need to check a page for a separate service, such as validating an advertising destination?
A May 2026 IAB workshop report published as informational RFC 9969 describes why this coordination remains unsettled: crawler-time controls, inference-time uses, vendor identifiers, and content-owner preferences do not map neatly onto one mechanism. The report is useful evidence of the problem, not a binding standard or a claim that every crawler behaves alike.
One vendor can publish several independent controls
The labels matter because a rule aimed at one purpose may leave another unchanged. Read the current vendor documentation, then record the intended effect beside each token.
OpenAI separates search, model development, user requests, and ads validation
OpenAI’s crawler documentation identifies OAI-SearchBot for ChatGPT search and GPTBot for content that may be used to improve generative AI foundation models. Those controls are independent, so a publisher can allow search discovery while disallowing GPTBot. OpenAI says changes may take about 24 hours to be reflected.
ChatGPT-User is different: it represents certain user-triggered visits, not automatic web crawling, and OpenAI notes that robots.txt rules may not apply. OAI-AdsBot is another separate agent used to validate submitted ad landing pages; OpenAI says that data is not used to train its foundation models.
Anthropic also distinguishes three purposes
Anthropic’s April 7, 2026 crawler guidance names ClaudeBot for potential model training, Claude-SearchBot for search-related retrieval, and Claude-User for user-initiated retrieval. Anthropic says its bots honor robots.txt and supports crawl-delay, but that vendor-specific statement should not be generalized to every user-triggered agent.
Google-Extended is not a Google Search switch
Google documents Google-Extended as a robots.txt product token, not a separate HTTP user-agent string. It controls certain Gemini model-development and grounding uses, but Google says it does not affect inclusion or ranking in Google Search.
For AI Overviews and AI Mode, Google’s Search AI features guidance points site owners back to normal Googlebot access and Search preview controls. Google also says no special AI text file or schema markup is required to appear in those features. That is why a Google-Extended rule should not be described as an AI-search visibility rule.
Build an AI crawler robots.txt policy from outcomes
Create a small decision matrix before writing syntax. For each bot or product token, record the vendor, documented purpose, allowed paths, blocked paths, policy owner, evidence link, decision date, and next review. Add the expected consequence in plain language. ‘Disallow GPTBot’ is less useful than ‘Keep public documentation discoverable in ChatGPT search while declining collection by GPTBot for model development.’
Use four checks for each row:
- Is the content intentionally public, and does the organization own the decision for every affected path?
- Does the control govern search discovery, model development, a user-triggered request, or another product function?
- Will a broader wildcard group or CDN rule override, duplicate, or contradict the planned instruction?
- What observable signal will show whether the change worked: crawler requests, referral traffic, citations, indexing, or a vendor test?
The last question prevents an easy reporting mistake. A crawler rule is an input. It does not guarantee visibility, exclusion from every downstream use, or a ranking outcome.
An illustrative search-yes, training-no file
A publisher that wants selected AI-search discovery but does not want specified model-development crawlers on public pages might begin with rules like these:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
This is an example of policy translation, not a universal paste-in solution. It omits existing site rules, path exceptions, sitemap declarations, other vendors, and infrastructure controls. Google-Extended also covers certain Gemini grounding uses, so its disallow rule is not identical in scope to GPTBot or ClaudeBot. ChatGPT-User and other user-triggered fetchers require separate product and security consideration.
Before deployment, merge the decisions with the site’s actual robots.txt, validate group precedence and path patterns, and test the file from every relevant hostname. A rule on www.example.com does not automatically govern a documentation subdomain or an asset host.
Robots.txt is not a privacy or access-control boundary
RFC 9309 is explicit that robots rules are not access authorization. The file is public, and disallowed paths can reveal the names of resources a visitor might not otherwise know about. Private customer records, unreleased products, paid content, personal data, and confidential files need authentication and authorization at the application or storage layer.
Use robots.txt for cooperative crawler preferences. Use sign-in, permission checks, network controls, expiring URLs, and appropriate cache policy for content that must not be available to unauthorized visitors. If a page is public but should not appear in a search preview, use the preview or indexing controls documented by the relevant search provider rather than treating a training-bot rule as a substitute.
Ship the policy as a production change
Crawler policy can affect discovery, infrastructure load, content licensing positions, and product behavior. Give it the same change discipline as a redirect map or analytics configuration.
- Inventory the production hosts and fetch the live robots.txt from each one before editing.
- Capture a dated copy, the reason for every new rule, the accountable owner, and the vendor documentation reviewed.
- Check CDN, firewall, bot-management, and origin logs; a correct robots file does not prove the infrastructure served or enforced the intended response.
- Test representative allowed and disallowed paths without exposing private URLs in the robots file.
- Monitor crawler identity, path, status code, volume, and referral outcomes separately. A user-agent string alone is not strong identity verification.
- Set a review date. Vendor tokens, product scope, and business preferences can change after deployment.
This is the same principle behind fixing measurement and conversion problems before spending more on acquisition: make the system observable before interpreting the outcome.
Measure discovery, collection, and security separately
A useful dashboard does not compress every consequence into ‘AI traffic.’ Track requests from documented crawler agents, visits from AI-assisted search referrals, landing pages cited or reached, server responses, and changes after a rule deployment. Keep training-policy decisions in a separate governance record because web analytics cannot prove whether already collected material was used, removed, or retained by a model provider.
Likewise, a drop in one crawler’s requests is not evidence that private content is secure. Test access controls without credentials, verify cache behavior, and review data exposure with the teams responsible for security, privacy, and legal policy. This article explains implementation distinctions; it is not legal advice about ownership, licensing, or regulatory obligations.
For teams that need the website, analytics, and infrastructure to support the same policy, The Branding Bull’s Web & Mobile Applications service and Growth Marketing service describe the relevant implementation and measurement work.
Treat crawler policy as product governance
The durable policy is not ‘block AI’ or ‘allow AI.’ It is a documented set of choices about which public content may support discovery, which collection uses the organization accepts, how user-requested retrieval is handled, and which material belongs behind real access controls. Vendor names are implementation details inside that larger decision.
If your current rules grew from copied snippets or emergency blocks, send a project brief to discuss a crawler-policy audit, the technical changes, and the measurement needed to verify them.
