FREE POLICY GENERATOR / ROBOTS.TXT

Choose what AI systems can do with your public pages.

Search discovery, user-requested retrieval and model training are separate choices. Set each one, then copy a policy block to merge into your existing robots.txt.

TARGET: None requested
PROCESSING: This browser only
OUTPUT: Merge-ready policy

Public evidence can be reviewed. It does not predict indexing, citations or traffic.

01 / LOCAL POLICY

Set three separate purposes.

The default keeps public search and a user-requested visit available while expressing a preference against model training. It does not change a CDN, WAF or provider account setting.

01AI search discovery

May a search crawler use public pages when it builds AI search results?

Applies to OAI-SearchBot, Claude-SearchBot and PerplexityBot.

02User-requested retrieval

May an assistant retrieve a public page because a person asked it to consult that source?

Applies to ChatGPT-User, Claude-User and Perplexity-User.

03Model training preference

May a provider crawl public pages for future model training or related provider-controlled grounding?

Applies to GPTBot, ClaudeBot and Google-Extended.

GENERATED BLOCK

Add this to your existing robots.txt

LOCAL ONLY

Review duplicate rules first. robots.txt expresses crawler preferences; it is not access control and does not guarantee crawling, indexing, citations or compliance.

02 / VERIFY EXISTING RULES

Already have a public site?

Open the free site audit to inspect observable crawler rules, content signals and public machine-facing files. It requests a public URL only when you choose to enter one there.

Audit my public site

03 / READ BEFORE PUBLISHING

Three choices, three different outcomes.

Discovery is not a promise

Allowing a search crawler removes one stated preference barrier. It does not make a page rank, appear in a response or receive a citation.

Protect private routes with real controls

A user-requested fetch can be useful for public pages. Keep accounts, admin areas and personal data behind authentication, not only robots.txt.

Training is a separate preference

A training preference can differ from AI search or a user-requested visit. Recheck each provider's current documentation before publishing.

SOURCES

Choose what AI systems can do with your public pages.

Already have a public site?

Discovery is not a promise

Protect private routes with real controls

Training is a separate preference

Verify provider controls at the source.