What Should robots.txt Allow for AI Crawlers Without Blocking GEO?

GEO teams keep asking the same question in Slack: should we block AI crawlers or let them in? The honest answer is neither blanket rule works. You need a short robots.txt policy that blocks low-value scrapers, allows retrieval bots you want citing you, and keeps admin paths closed.

If you block every bot with "AI" in the name, you may reduce training noise but you also reduce the chance Perplexity or Claude can fetch your canonical FAQ page. If you allow everything, you invite bulk scrapers that never send traffic. White-hat GEO starts with an explicit allow list tied to business goals, not fear headlines.

Which AI crawlers matter for citation in 2026?

The bots that most often affect answer-engine visibility (not just training) include:

Google's core Googlebot still drives classic SEO and Google AI Mode, which our CiteLens citation study breakdown showed tracks organic top-10 overlap near 93%. Do not conflate blocking Google-Extended with blocking Google search indexing; they are separate products.

TLDR: Know the six agents above. Policy differs for training bots vs citation fetchers vs Google search.

Should you block GPTBot if ChatGPT barely uses Google rank?

CiteLens data suggests ChatGPT overlap with Google top-10 can sit near 30%, so rank alone will not save you. That does not mean blocking GPTBot is free. Blocking removes one path for OpenAI systems to ingest your structured facts, product names, and FAQ text.

A practical split:

• Block GPTBot only if you have a legal or licensing reason and you accept reduced OpenAI surface area.
• Allow OAI-SearchBot if your goal is retrieval visibility and your legal team is comfortable.
• Allow PerplexityBot if Perplexity is a target channel (high Google-rank correlation in the same study).

Document the decision in your internal GEO runbook so marketing does not flip policy every quarter.

TLDR: ChatGPT is rank-weak, but blind blocking still costs structured-ingest paths. Decide per bot, in writing.

What does a minimal white-hat robots.txt look like?

Keep the file small and readable. Example pattern for a marketing site that wants GEO visibility on public pages:

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /preview/
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

Add Disallow for staging hosts at the DNS level (do not expose staging.yoursite.com publicly). Pair robots rules with static HTML on money pages so allowed bots see answers in view-source, not empty shells, per our view-source GEO checklist.

Optional: publish llms.txt at /llms.txt with plain-text pointers to your canonical About, pricing, and docs URLs. Robots governs who may crawl; llms.txt governs what you want emphasized. They complement each other.

TLDR: Disallow admin and preview paths globally; explicitly allow citation bots on public marketing URLs.

How do you test robots.txt before you publish?

Run three checks:

1. Fetch https://yoursite.com/robots.txt and confirm status 200 with text/plain, not HTML from a SPA fallback.
2. Validate each Disallow path returns 404 or 401, not public article HTML.
3. Spot-check one article URL with curl.exe -sS and confirm H1 plus JSON-LD appear in the first response.

Log results next to your weekly citation overlap test from the CiteLens playbook. Tools like Searchable ↗ help track whether visibility moves after policy changes, but they cannot fix a robots file that blocks the bots you care about.

TLDR: Test robots.txt returns plain text, blocked paths stay closed, and article HTML is crawl-ready.

What should you change this week?

Export your current robots.txt. Highlight every AI-related user-agent. For each one, mark allow, block, or undecided with a one-line business reason. Remove duplicate or legacy rules from old CMS migrations.

If you are on Cloudflare Pages, serve robots from static dist/robots.txt at build time so it never accidentally becomes a client route. Then redeploy and re-fetch. GEO is mostly boring infrastructure; boring wins citations.

TLDR: Audit AI user-agents with written reasons, serve robots statically, redeploy, re-test.