robots.txt and AI crawlers: should you block GPTBot?
Usually there is no need, and if you do, block only the training crawlers. GPTBot collects content that may be used to train OpenAI's models; OAI-SearchBot is the separate crawler that lets ChatGPT search show your site. Blocking the search crawlers of ChatGPT, Perplexity or Claude can keep your practice out of their answers, which for most practices costs more than it protects.
What robots.txt can and cannot do
Robots.txt is a plain text file at the root of your site, at your domain followed by /robots.txt, that tells crawlers which parts of the site they may visit. Each group of rules names a crawler in a User-agent line and lists paths it should not visit in Disallow lines.
Google's documentation is clear about its limits. Robots.txt is mainly for managing crawler traffic. It is not a mechanism for keeping a web page out of Google; for that, Google says to use noindex or password protection. A page blocked in robots.txt can still be indexed if other sites link to it. And robots.txt cannot enforce anything: respectable crawlers obey it, but it is up to each crawler to do so.
So robots.txt is a set of requests to well behaved crawlers. That is exactly what you need for the AI companies' crawlers, which document that they respect it, and not what you need for privacy.
The crawlers that matter, sorted by job
Each company separates crawlers by purpose. The names below are the ones they publish.
- OpenAI: GPTBot crawls content that may be used to train its models. OAI-SearchBot surfaces websites in ChatGPT's search features, and sites that opt out of it are not shown in ChatGPT search answers. ChatGPT-User makes visits on behalf of a user, and OpenAI notes that robots.txt rules may not apply to those.
- Anthropic: ClaudeBot collects content that could contribute to model training. Claude-SearchBot crawls to improve search results for Claude users. Claude-User retrieves pages when a user asks Claude a question. Anthropic says blocking either of the last two may reduce your visibility.
- Perplexity: PerplexityBot surfaces and links websites in Perplexity's search results and is not used for training foundation models. Perplexity-User visits pages to answer a user's question and generally ignores robots.txt.
- Google: Googlebot crawls for Google Search, including AI Overviews and AI Mode, which draw on indexed pages. Google-Extended is a robots.txt token, not a separate crawler, that controls whether content can be used for training Gemini models and for grounding; Google says it does not affect inclusion or ranking in Google Search.
Does ChatGPT recommend your practice?
The Surge Score measures five things about your practice website: how often ChatGPT, Gemini and Perplexity recommend you, your PageSpeed, your structured data, your indexing and your reviews, each with its evidence and date. Free.
Training versus answers: the decision for a practice
The useful distinction is between crawlers that collect training data and crawlers that fetch pages so an assistant can answer a patient today.
Blocking training crawlers, such as GPTBot, ClaudeBot or the Google-Extended token, is a reasonable choice if you do not want your content used to train models. It is a values decision. Your practice pages are public marketing content, so the practical stakes are modest either way.
Blocking search crawlers, such as OAI-SearchBot, Claude-SearchBot or PerplexityBot, is different. Their companies say blocking them keeps your site out of, or less visible in, their search answers. For a practice that wants new patients, that is usually a self inflicted wound.
Google offers one more control. The Search generative AI setting in Search Console lets you exclude your site from AI Overviews, AI Mode and generative AI features in Discover, both as links and as a source for answers. Google says it does not affect the rest of Search. Few practices seeking patients will want to turn it on.
How to read your own robots.txt
Type your domain followed by /robots.txt into a browser. You will see a short text file, or a not found page if you have none, which means nothing is blocked by robots.txt.
Look for three things:
- A group that starts User-agent: GPTBot, User-agent: OAI-SearchBot, User-agent: PerplexityBot, User-agent: ClaudeBot or User-agent: Claude-SearchBot, followed by Disallow: /. That pair blocks that crawler from your whole site.
- A group for all crawlers, where the user agent is a single asterisk, followed by Disallow: /. That blocks every crawler that obeys robots.txt, including Google. It is sometimes left over from a staging site.
- Disallow lines for paths that should be public, such as your provider or services folders.
If you change the file, allow time. OpenAI and Perplexity both say changes can take about 24 hours to be reflected in their systems.
Blocks you did not choose
Robots.txt is not the only place AI crawlers get turned away. Some hosting companies, website platforms, content delivery networks and security services offer settings or firewall rules that block AI bots, and those can be switched on by you, by a past vendor or by a default you never saw.
These blocks happen before the crawler ever reads your robots.txt, so the file can look perfectly open while the crawler is refused at the door. If your robots.txt looks fine but your practice never appears in an assistant's answers, ask whoever manages your hosting and security to check for bot management or firewall rules that affect AI crawlers. OpenAI also recommends allowing requests from its published IP ranges. Our guide on JavaScript sites and AI crawlers covers the other common reason a crawler sees nothing.
What to keep out, and how
There are pages you do not want in any search engine or assistant: portal logins, form confirmation pages, staff only areas and anything containing patient information. Robots.txt is the wrong tool for all of them. It is public, so listing a sensitive path advertises it, and it only asks politely.
- For pages that should not appear in search, use a noindex tag.
- For anything private, use a login. Google's guidance names password protection as a way to keep content out of its results.
- Patient information should never be on public pages in the first place, and tracking tools on pages where patients enter health information raise separate privacy questions. See tracking technology and patient privacy.
Use robots.txt for what it was designed for: telling well behaved crawlers where they are welcome.
Checklist
- You have opened your robots.txt and know what it blocks
- OAI-SearchBot, PerplexityBot and Claude-SearchBot are not disallowed
- No rule for all crawlers disallows your entire site
- Any training crawler blocks are a deliberate decision
- Your host or security service is not blocking AI crawlers you want
- Private pages are protected by login or noindex, not by robots.txt
- Search Console's Search generative AI control is set to include your site
Frequently asked
If I block GPTBot, will my practice disappear from ChatGPT?
Not from ChatGPT search, according to OpenAI's documentation. GPTBot is for training. OAI-SearchBot is the crawler that surfaces sites in ChatGPT's search features. You can block GPTBot and allow OAI-SearchBot.
Does blocking Google-Extended remove me from AI Overviews?
Google says Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode draw on pages indexed by Googlebot. To exclude your site from those features, Google provides the Search generative AI control in Search Console.
Can AI crawlers slow down my website?
Any crawler adds some load. Anthropic says it supports the non standard Crawl-delay rule to limit its crawling. For a typical practice site the load is small; if you see a problem, ask your host before blocking crawlers you want.
Why does Perplexity show my page even though I blocked it?
Perplexity says its user triggered fetcher, Perplexity-User, generally ignores robots.txt, since a person asked for the page. OpenAI says the same may be true of ChatGPT-User. Blocking those visits requires server or firewall rules, not robots.txt.
Sources
- Introduction to robots.txt, Google Search Central
- Overview of OpenAI crawlers, OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic
- Perplexity crawlers, Perplexity
- Google's common crawlers, Google Search Central
- Search generative AI control, Google Search Console Help
- AI features and your website, Google Search Central
Book a strategy call with the team.
Twenty minutes. We'll walk through the specific opportunities in your market and what a MedPracticeGrowth engagement would look like for your practice.