Is Your Site Invisible to ChatGPT? Check Your robots.txt

Someone asks ChatGPT “what’s the best invoicing tool for freelancers?” and ChatGPT answers — with your competitors’ names. Not because your product is worse, but because ChatGPT’s crawler never saw your site. Your robots.txt told it to go away, probably without you ever knowing.

Over 3% of websites now block OpenAI’s crawlers outright, and many more block them accidentally — through a starter template’s default robots.txt, a CDN “block AI bots” toggle, or a copy-pasted rule from 2024. In 2026, when a meaningful share of product research starts in an AI assistant instead of Google, that little text file quietly decides whether you exist.

I ran this check on my own site while writing this article, and found a subtle gap: my robots.txt explicitly allowed GPTBot, ClaudeBot, and PerplexityBot, but never named OAI-SearchBot or ChatGPT-User — the two bots that actually decide whether ChatGPT search cites me. They were only covered by the fallback wildcard. Explicit beats accidental.

Key Takeaways

• In 2026, roughly 3% of websites block OpenAI’s crawler, and many more do it by accident via templates or CDN toggles (Passionfruit, 2025).
• Blocking GPTBot stops model training; blocking OAI-SearchBot removes you from ChatGPT search citations. These are separate decisions.
• Audit your robots.txt per bot, allow search bots explicitly, and pair it with llms.txt and plain-HTML product details.

What Is robots.txt for AI Crawlers?

robots.txt is a plain text file at your site’s root that tells automated crawlers which parts of your site they may fetch. AI crawlers — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot — follow the same protocol as Googlebot: they look up https://yourdomain.com/robots.txt before crawling, find a User-agent group matching their name (or fall back to User-agent: *), and obey the Allow/Disallow rules in it.

The catch in 2026 is that “AI crawler” is no longer one thing. OpenAI alone operates at least three distinct bots with different jobs:

• GPTBot — collects content for training future models. Blocking it protects your content from training but does not remove you from ChatGPT search.
• OAI-SearchBot — crawls the live web for ChatGPT Search results. OpenAI’s documentation is explicit: if you block this bot, your site will not be shown in ChatGPT Search.
• ChatGPT-User — fetches pages on behalf of a real user (when someone pastes a link into a chat). It is not used for search inclusion and robots.txt rules may not apply.

Anthropic and Perplexity mirror the same split: ClaudeBot (training) vs Claude-SearchBot / Claude-User (search and user actions), and PerplexityBot (search index). One file now manages at least eight decisions that used to be one.

Why Being Blocked Matters More in 2026

Do people really ask ChatGPT instead of Google? Increasingly, yes — one industry analysis estimates ChatGPT is involved in roughly 93% of commercial AI searches (honeyb.ai, 2025). Whether that exact number holds up or not, the direction is not in doubt: AI assistants are now a discovery channel, and they only recommend what they can crawl or what appears in the sources they trust.

Here’s the uncomfortable part: the block is usually unintentional. Common causes I keep seeing:

• A starter template that shipped with User-agent: GPTBot / Disallow: / years ago and nobody re-reviewed.
• A CDN or security plugin toggle (“Block AI bots”) that fires at all AI user-agents at once — taking out OAI-SearchBot along with GPTBot.
• A blanket rule like User-agent: * / Disallow: / left over from a staging environment that went live.

A blanket block is the worst case: it also stops Bingbot, which feeds Copilot, and effectively hides you from every answer engine that respects robots.txt.

Which AI Crawlers Should You Allow?

This is where most guides stop and where your actual decision starts. The right answer depends on what you want to block — training, search, or both. The comparison table below is the decision map:

The single biggest mistake is treating all AI bots as one group. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot) drive citations — the free, high-intent referrals of the AI era. Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) are a long-term strategic question about who learns from your content. You can — and should — decide about each category separately.

How to Write a robots.txt That Allows AI Search (Without Giving Away Training Rights)

If you want to appear in ChatGPT, Claude, and Perplexity answers but keep your content out of model training, this is the pattern the reference community converged on in 2025–2026:

# Allow AI search & user-triggered bots (citation visibility)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Block training-only bots (optional strategic choice)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

# Everyone else (including Googlebot, Bingbot)
User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://yourdomain.com/sitemap.xml

Three rules make robots.txt behave the way you expect:

1. Named groups win. A bot looks for a User-agent block with its exact name first; if found, it ignores the * group entirely. This means a leftover User-agent: GPTBot / Disallow: / overrides your friendly wildcard.
2. Longest path wins when rules conflict — Disallow: /private/ beats Allow: / for anything under /private/.
3. An empty Disallow: means “allow everything.” User-agent: * followed by Disallow: (no path) is a full permit, not a block.

If your goal is simpler — “be visible everywhere, I don’t care about training” — then the whole file collapses to:

User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://yourdomain.com/sitemap.xml

Just make sure no stray named groups below it contradict it.

When I audited my own robots.txt for this article, I found exactly this class of gap: I had explicit Allow: / groups for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and CCBot — but OAI-SearchBot, ChatGPT-User, and the Claude search bots were unnamed, riding on the wildcard. Functionally allowed, but fragile: any future User-agent: * change, a new Disallow, or a CDN rule reclassifying unknown bots would silently drop me from ChatGPT search. I also noticed a policy inconsistency worth fixing — my Content-Signal header declares ai-train=no while my robots.txt says GPTBot: Allow. Signals that contradict each other are worse than either choice alone.

How to Check If Your Site Is Currently Blocked

The check takes two minutes and every site owner should do it once:

1. Open https://yourdomain.com/robots.txt in a browser. Search the file for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and any line Disallow: /. Anything you don’t name explicitly falls under User-agent: *.
2. Look at your CDN/security settings. Cloudflare, Vercel, and various WordPress security plugins now ship “block AI bots” switches. Check whether the toggle distinguishes search bots from training bots or blocks everything.
3. Check your CMS defaults. Some SaaS platforms and static-site templates ship restrictive robots.txt out of the box.
4. Ask the assistants directly. Ask ChatGPT or Perplexity a question your site should answer for. If competitors get cited and you don’t — over weeks, not days — crawlability is one of the first suspects.

Also verify the flip side: some hosts return different responses to AI user-agents. A quick way to spot it is comparing a plain request against one sent with a User-agent: GPTBot header — if the bot gets a 403 or a wall, your robots.txt content doesn’t matter.

Crawlable Is Necessary, Not Sufficient: Optimize for Citation

Fixing robots.txt gets AI crawlers in the door. Whether they cite you depends on what they find. The reference article’s advice holds up, and it matches what the generative engine optimization (GEO) literature converges on:

• Put the essential facts in plain HTML — what it does, who it’s for, pricing. If your product details only exist inside a JS bundle, crawlers may miss them.
• Answer questions upfront. A clear TL;DR or FAQ block on key pages gives answer engines something extractable. (This site adds FAQPage schema to articles for exactly that reason.)
• Keep naming consistent across your site, directories, and profiles. Assistants reconcile identity across sources; inconsistency makes you unquotable.
• Earn mentions on trusted third parties — directories, review sites, well-known communities. AI answers lean heavily on sources that independently describe you.
• Add llms.txt — a 2024-proposed standard that gives LLMs a curated map of your key pages in one file. Support is still uneven (not all bots read it), but it’s a cheap bonus alongside a Not for section that prevents irrelevant recommendations.

Common Mistakes to Avoid

• Blocking everything AI because of one viral post about content scraping. You lose every citation channel to protect training rights you may not care about.
• Copying a 2024 robots.txt before the search-vs-training bot split existed. The bot list changed; your file should too.
• Relying on the wildcard alone. Explicit Allow groups for each search bot survive future wildcard tightening and show your intent to any audit.
• Contradicting signals — robots.txt says one thing, Content-Signal or meta robots say another. Pick one policy per bot and make every layer agree.
• Forgetting the sitemap line. Crawlers — AI ones included — still use it to find your pages efficiently.

Frequently Asked Questions

Does blocking GPTBot remove my site from ChatGPT search results?

No. Blocking GPTBot only stops OpenAI from training on your content. ChatGPT Search relies on OAI-SearchBot — as long as that bot is allowed, you can still be cited in search answers.

Which robots.txt entry is the most important for ChatGPT visibility?

OAI-SearchBot. OpenAI states that sites blocking it will not appear in ChatGPT Search. Allow it explicitly with User-agent: OAI-SearchBot followed by Allow: /.

How do I allow AI search bots but block AI training?

Create explicit Allow: / groups for OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, and PerplexityBot, then Disallow: / for GPTBot, ClaudeBot, and CCBot. Keep a permissive User-agent: * fallback for Googlebot and Bingbot.

Do AI crawlers actually respect robots.txt?

The major ones do — OpenAI, Anthropic, Google, and Perplexity document robots.txt compliance for their named crawlers. Enforcement varies for smaller or scrappier bots, so robots.txt is a strong signal, not a technical guarantee.

What is llms.txt and do I need it?

llms.txt is a proposed standard (from 2024) placing a markdown file at your root that curates your most important pages for LLMs. Support is partial today, so it’s a low-cost supplement to robots.txt and good HTML — not a replacement.

Conclusion

Your robots.txt is now a distribution decision, not just a housekeeping file. The bots that decide whether ChatGPT, Claude, and Perplexity recommend you are distinct from the bots that train on your content — and in 2026 the number of sites getting this wrong, often silently, keeps growing.

Your five-minute checklist:

1. Open /robots.txt and list every AI bot named there.
2. Explicitly allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, and PerplexityBot.
3. Make a conscious, separate decision about training bots (GPTBot, ClaudeBot, Google-Extended, CCBot).
4. Remove contradictions between robots.txt, CDN toggles, and any content-signal headers.
5. Add a sitemap line, then layer on llms.txt and plain-HTML answers on your key pages.

Then ask ChatGPT the questions your customers ask. If you’re not in the answer yet, at least now you’ll know exactly which line to fix.

Sources

• [OpenAI], Crawler documentation (GPTBot, OAI-SearchBot, ChatGPT-User), retrieved 2026-11-05, https://developers.openai.com/api/docs/bots
• [Passionfruit], “What is GPTBot and should you block it”, retrieved 2026-11-05, https://www.getpassionfruit.com/blog/what-is-gptbot-and-should-you-block-it
• [honeyb.ai], “OAI-SearchBot”, retrieved 2026-11-05, https://www.honeyb.ai/blog/oai-searchbot
• [xseek.io], “GPTBot block/allow and robots.txt”, retrieved 2026-11-05, https://www.xseek.io/blogs/articles/gptbot-block-allow-robots-txt
• [ESEOspace], “AI crawlers and robots.txt”, retrieved 2026-11-05, https://eseospace.com/blog/ai-crawlers-robots-txt
• [IndieDaddy], “Is your site invisible to ChatGPT? Check your robots.txt”, retrieved 2026-11-05, https://dev.to/indie_daddy/is-your-site-invisible-to-chatgpt-check-your-robotstxt-ojj