GPTBot, ClaudeBot, PerplexityBot: Block or Allow?

Two Jobs, Three Robots

A training crawler reads a page to fold it into the dataset behind a future model. An answer crawler reads a page in real time, the moment a user asks a question, to pull a citation into a generated response. In a robots.txt file, both announce themselves the same way, with a plain User-agent line. Economically, they could not be more different: blocking a training crawler protects content from future reuse, while blocking an answer crawler drops a site out of ChatGPT, Claude, or Perplexity answers immediately. Mixing the two up is the most common mistake we see when a company configures robots.txt for the first time.

The instinct, once a founder reads a headline about AI models scraping the open web, is to block anything that looks like an AI bot. That reaction genuinely protects content from being folded into a future model. It also strips away, in the same move, any chance of showing up in an answer generated the same day. The rest of this article works through the calculation crawler by crawler, so the two decisions do not get made at once by accident.

GPTBot, ClaudeBot, PerplexityBot: Who Does What

Each operator actually ships several crawlers with different jobs. Treating an operator as a single entity means blocking answer visibility while believing you are only limiting training.

One confusion is worth flagging on its own: Google-Extended is not Googlebot. Blocking Google-Extended has zero effect on classic indexing in Google Search, which stays entirely governed by Googlebot. A marketing lead who blocks Google-Extended believing it protects search rankings changes nothing about ranking, and only reduces the odds of showing up inside an AI-generated summary.

A Cloudflare network analysis published in August 2026 found that GPTBot is disallowed 2.33 times for every site that allows it, while OAI-SearchBot, the search-facing crawler from the same company, is allowed nearly as often as it is blocked. Sites that took the time to separate the two crawlers are not making the same tradeoff as the ones that block "OpenAI" as a single block.

For a small accounting firm in Geneva or an architecture practice in Lausanne, the difference plays out very concretely. A prospect asks ChatGPT for an accounting firm recommendation in Geneva: if GPTBot is blocked but OAI-SearchBot stays allowed, the page remains eligible for that answer. If both get blocked under one catch-all rule, the firm simply disappears from that conversation, and nobody inside the company notices, since few teams track AI citations the way they track search rankings.

The Real Trade-off: What Blocking Protects, What It Costs

What blocking genuinely protects

Blocking a training crawler limits how your text gets folded into a future language model. That matters for high-value editorial content, a proprietary dataset, or any sector where contractual confidentiality rules out external reuse. The protection is not retroactive, though: content already crawled before the block was set stays inside datasets that have already been assembled.

What blocking costs in AI visibility

Blocking an answer crawler, often out of caution by disallowing every user-agent containing "GPT" or "AI", mechanically removes a site from cited answers. For a company whose stated goal is to be found through answer engines, that choice works directly against the goal. Here is the standalone version worth quoting on its own: a training crawler builds a model, an answer crawler builds a citation; the first gets blocked to protect content, the second gets blocked at the price of the exact visibility a company is trying to earn.

A Five-Step Method

None of these steps require specialized tooling. A text editor, access to server logs, and a recurring calendar reminder cover the whole exercise for a company running a handful of key pages rather than a large content operation.

1. State the objective first: protect proprietary content, aim for citations in AI answers, or both, depending on the section of the site.
2. For each operator, separate the training crawler from the answer crawler instead of blocking the operator as a whole.
3. Write one explicit User-agent line per crawler in robots.txt, never a catch-all rule targeting anything containing "bot" or "AI".
4. Confirm in server logs that crawlers actually honor the published directives, rather than trusting the file alone.
5. Revisit the configuration every quarter: operators regularly ship new user-agents with different jobs attached.

A Working robots.txt Example

Here is a configuration that allows answer crawlers while blocking training crawlers, built for a company that wants AI answer visibility without handing over its entire content library to third-party model training:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Disallow: /

This configuration is one specific trade-off, not a universal recommendation: a company selling high-value documentary content may prefer to block answer crawlers too, while a media site living off direct traffic may prefer to allow everything. Treat the block above as a starting point to edit, not a file to paste unchanged onto every domain.

Confirming the Directives Are Actually Followed

Major operators publicly commit to honoring robots.txt for their named crawlers, but that does not rule out a misconfiguration on the site side, or a third party spoofing a well-known user-agent string. Two checks usually cover it: reviewing server access logs to confirm the requesting IPs match the ranges published by the operator, and running the page through Google Search Console's URL inspection tool to confirm no accidental rule is blocking a legitimate crawler.

A third, less common check is to ask answer engines directly about your own sector, using a small, representative set of questions, and note whether the site shows up. That is the only check that measures the real effect of a robots.txt decision rather than its formal compliance. Even a simple monthly tracking sheet is enough to catch a regression after a configuration change.

When This Fine-Grained Calculation Is Not the Right Call

Two configuration mistakes show up most often once a company starts separating crawlers by hand. The first is copying a list found on a forum without checking the date: operators retire and rename user-agents, and a rule written against a two-year-old list can silently stop matching anything. The second is forgetting that a robots.txt directive is a request, not an enforcement mechanism; a crawler that ignores it entirely will not show up as an error anywhere, only as unexplained traffic in the logs. Both mistakes are reasons to keep the configuration simple rather than reasons to skip the exercise altogether.

Separating crawlers one by one takes ongoing maintenance and requires watching for new user-agents published by each operator. Two cases call for a more radical approach instead. First, a site handling data under contractual or regulatory confidentiality, such as healthcare, finance, or client records: blocking every AI crawler without distinction remains the safest position, even at the cost of total invisibility in AI answers. Second, an editorial site that lives off citation volume and holds no sensitive content: allowing everything, without bothering to separate training from answering, avoids maintenance that outweighs the stakes. Between these two extremes, most small and mid-sized companies are better served by the fine-grained calculation described here, neither the reflex block that sacrifices AI visibility for no measurable benefit, nor blanket access that gives away proprietary content for nothing in return.

One more factor shapes the decision: the size of the team that will maintain the configuration over time. A company without a dedicated technical lead is better off with a simple rule, revisited once a year with a partner's help, than with a ten-crawler matrix updated every time the news cycle shifts. The fine-grained method in this article is a tool, not an obligation: a simple configuration applied correctly beats a precise one that never gets updated.

A robots.txt setting is one technical lever among several: it does not replace the underlying work on page structure and verifiable data covered in our overview of the six levers behind visibility in AI answer engines. robots.txt and llms.txt answer two different questions: the first allows or blocks access, the second summarizes content for crawlers already let in. For a company that wants to know where it stands before adjusting its directives, the GEO Sprint starts from a concrete audit. All of the studio's articles on the topic stay listed on the blog.