How to check AI crawler access without changing your training policy

Choose the search access you want, check the rule for each relevant crawler, then verify that your server actually returns the page. Training permissions and user-requested visits have their own controls.

By CiteMaybePublished and sources reviewed November 13, 2026

Identify the purpose before changing access

Check the named crawler’s purpose, the rule that matches your page, and the response your server returns. Search discovery, model training, and a visit requested by a user can follow different policies. A site owner can make a deliberate choice for each documented use.

Begin with the exact public page you want customers to find. Record its HTTPS URL, final destination after redirects, and the current robots.txt file on that origin. An apex domain and a www hostname can serve different policies. Preserve the original configuration before making an edit.

Read the documented crawler roles

OpenAI: OAI-SearchBot supports ChatGPT search discovery. GPTBot collects content that may be used for model training. ChatGPT-User handles some user-triggered visits, for which OpenAI says robots rules may not apply. OpenAI crawler documentation.
Anthropic: Claude-SearchBot supports search, ClaudeBot collects potential training content, and Claude-User retrieves content for users. Anthropic states that these bots honor robots directives. Its hosted browser agents have separate documentation. Anthropic crawler documentation.
Google: Googlebot crawl preferences affect Google Search, including its AI features. Google-Extended governs specified Gemini training and grounding uses. It has no separate HTTP user-agent string and does not determine Google Search inclusion or ranking. Google crawler and product tokens.
Microsoft: bingbot is Bing’s principal crawler. Check Bing’s content-use preferences separately when considering training and grounding. Microsoft’s announced replacement controls start taking effect as pages are crawled from February 1, 2027. Bing crawlers · Microsoft’s transition dates.
Perplexity: PerplexityBot supports search discovery. Perplexity-User fetches for user requests and generally ignores robots rules. Perplexity describes both as separate from foundation-model training. Perplexity crawler documentation.

These are documented tokens for specific purposes. For DeepSeek, Doubao, Qwen, or another product, establish the current official token and its meaning before adding a rule. A product name alone supplies insufficient evidence to invent a user-agent entry.

Check the group that applies to the page

Read the whole robots file. A named crawler group can replace the wildcard group for that crawler. Under the Robots Exclusion Protocol, matching groups for a product token are combined, and the wildcard applies when no matching group exists. Google also documents its most-specific user-agent selection. A restriction placed only under User-agent: * may therefore be absent from the applicable named group. Protocol group matching · Google group selection.

Within the applicable rules, inspect the most specific matching path. A blanket disallow can have a narrower allow exception. Test the actual destination URL, including its path case, and record the matching line with the result. Review duplicate groups added by plugins or hosting settings.

A scoped example for a public service page

This fictional site wants OpenAI and Anthropic search crawlers to reach public pages while keeping its existing choice to block their named training crawlers. It also asks compliant crawlers to avoid /account/. The example covers these declared groups and their wildcard fallback. Review your complete existing policy before adapting it.

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /account/

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /
Disallow: /account/

For these groups, /services/repair is available to the two search crawlers and excluded for the two training crawlers. The explicit search group repeats the account restriction because it does not inherit the wildcard rules. User-triggered agents still follow their providers’ documented behavior.

Protect account pages with authentication and authorization. Robots files are public crawl instructions, and listing a path exposes its name. Robots protocol security considerations.

Check the firewall and delivered page

A permitted robots rule can coexist with a firewall challenge, denied request, redirect loop, or server error. Inspect the returned HTTP status and body. An HTML challenge page with status 200 still prevents the crawler from receiving your service description.

Use hosting logs to investigate a claimed crawler visit. Verify its identity through the provider’s published method before granting a firewall exception. A user-agent string alone is easy to copy. Perplexity documents matching each of its bot names with that bot’s current published IP ranges. Keep an exception scoped to the verified traffic and required paths. Perplexity firewall guidance.

Then inspect the page’s source HTML. Text supplied only after JavaScript runs can be absent from a source-based audit. Indexing directives such as noindex require their own review after access is established.

Verify one change at a time

Save the original policy, affected URL, intended crawler, observed response, and time.
Change the specific rule responsible for the unintended outcome, preserving the owner’s other publishing choices.
Fetch the published robots file again. Check the target page, every redirect destination, and any relevant firewall response.
Use the search engine’s URL inspection tools and verified server logs when available. Allow for policy caching and later recrawling.
Run the same audit again and compare its source evidence. Separately inspect actual answers when measuring a citation.

Google’s AI search features use existing search eligibility and SEO foundations. Passing access checks establishes a technical condition. Search indexing and an observed answer citation each need their own evidence. Google AI search requirements.

Start with the page customers need

CiteMaybe’s free audit checks one public HTTPS page, records the source and applicable crawler findings, and leaves inaccessible content checks untested. Approved extended-scan hosts can include up to five pages. Bring the final URL, matching policy line, timestamp, and recommendation to the person managing your site.

After access is understood, review whether the page actually answers the buyer’s question and record any external citations under consistent conditions. Run a free page audit · Check actual AI answer citations.