MyClawn AI Search Audit › AI crawlers
GPTBot and the other AI crawlers: what to allow in robots.txt
Blocking the wrong bot can take you out of ChatGPT or Perplexity answers while leaving training untouched, or the reverse. Here is what each crawler does according to its operator, and a robots.txt that keeps you visible.
Training bots vs search bots vs user fetchers
| Crawler | Operator's stated purpose | Respects robots.txt | If you block it |
|---|---|---|---|
| GPTBot | Collects content that may train OpenAI's models (OpenAI) | Yes | Opts out of training; does not by itself remove you from ChatGPT search |
| OAI-SearchBot | Surfaces websites in ChatGPT search features (OpenAI) | Yes | "Will not be shown in ChatGPT search answers", though navigational links can still appear |
| ChatGPT-User | Visits pages when a user asks ChatGPT or a custom GPT something (OpenAI) | Rules "may not apply" | Not used to decide search inclusion |
| PerplexityBot | Surfaces and links sites in Perplexity results; not used for training (Perplexity) | Yes | Out of Perplexity's search results |
| Perplexity-User | Fetches a page when a user asks a question (Perplexity) | Generally ignores it | Little effect |
| Google-Extended | robots.txt token for Gemini training and grounding (Google) | Yes (token only, no separate bot) | Does not affect Google Search inclusion or ranking |
| Googlebot, Bingbot | Search indexing | Yes | Out of search, and out of the AI answers built on it |
A robots.txt that stays visible but opts out of training
If you want to appear in AI search answers but not have your pages used for model training, this is the common pattern:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Change the domain, keep any existing rules for your admin or cart paths, and remember that a User-agent: * block does not override a named block: a bot follows the most specific group that names it. If you are happy to be used for training, leave GPTBot and Google-Extended allowed.
OpenAI notes that search changes can take about 24 hours to take effect after you update robots.txt.
Paste your website. Starter $39/mo · Pro $129/mo. Refund within 14 days if it misses.
Audit my siteThe mistake we see most: the firewall says no
robots.txt is a request; your CDN or firewall is what actually answers. "Block AI bots" switches in CDN and security plugins, bot-fight modes and rate limits can return 403 or 429 to search crawlers that robots.txt allows. From the outside, the site looks open; to OAI-SearchBot, it is closed.
That is why our audit does not stop at reading robots.txt. It fetches your home page with each crawler's user agent and records the HTTP status in a table, so a block shows up as a number, not a guess. (A firewall that checks IP ranges can still treat our requests differently from the real bot; the report says when a result needs confirming in your CDN logs.)
Decisions by business type
| You are | Usually |
|---|---|
| A local service business | Allow search crawlers (OAI-SearchBot, PerplexityBot, Bingbot, Googlebot). Training bots: your call; most local businesses gain nothing from blocking. |
| B2B or SaaS | Allow search crawlers. Block training bots only for gated or proprietary content paths. |
| A publisher with paid content | A real trade-off; decide per section. Not our core audience. |
What the audit gives you
fixes/robots-check.txt with the exact lines to change, or "no change needed", plus the crawler table in the report: each bot, its robots.txt rule, and the HTTP status we got. Read more on how it works.
Why we built this: we run our own software products and measure, in our own analytics, where signups come from. Over 34 days, visitors arriving from ChatGPT produced 46 signups on one product, 27 on a second and 15 on a third; Gemini sent 11. ChatGPT visitors converted from visit to signup at about 31% (internal analytics, November 2026). That is a small channel next to Google, but it converts, and we wanted the same checks for every site we run. This audit is those checks, written up for your site.
Questions
Does blocking GPTBot remove me from ChatGPT?
Not from ChatGPT search, according to OpenAI's docs: search inclusion is controlled by OAI-SearchBot.
Does blocking Google-Extended hurt my Google rankings?
Google says it does not affect inclusion or ranking in Google Search.
Can I verify a request really is GPTBot?
OpenAI publishes IP ranges for its crawlers; check them in your server or CDN logs.
Paste your website. Starter $39/mo · Pro $129/mo. Refund within 14 days if it misses.
Audit my siteMore about AI Search Audit
Sources (checked 2026-11-09): developers.openai.com/api/docs/bots · docs.perplexity.ai/guides/bots · developers.google.com/crawling/docs/crawlers-fetchers/google-common-cr