Should I block GPTBot? Opt out of AI training, stay in ChatGPT

GPTBot collects pages for AI training, but a different OpenAI bot decides whether you appear in ChatGPT answers. Block one and keep the other.

Published

A robots.txt card with four directive rows, the second row flagged by a yellow Blocked pill
In short

Asking "should I block GPTBot?" usually means you want your pages kept out of AI training without disappearing from ChatGPT. You can have both. OpenAI uses one bot, GPTBot, to collect training content and a separate one, OAI-SearchBot, to find pages for ChatGPT's search answers. Block GPTBot in your robots.txt file (a short text file that tells bots which pages they may visit), leave OAI-SearchBot allowed, and wait about a day.

What GPTBot is, and what OpenAI's other bots do

A bot (or crawler) is a program that visits web pages automatically. Your robots.txt file is a public text file at yoursite.com/robots.txt that tells bots which pages they may visit. Polite bots follow it, but it is a request, not a lock.

OpenAI lists its bots on its crawler documentation page. Each one has its own name, called a user-agent token, and its own job:

BotWhat it doesFollows robots.txt?
GPTBotCollects content that may be used to train OpenAI's AI modelsYes
OAI-SearchBotFinds pages to show in ChatGPT's search answersYes
ChatGPT-UserOpens a page right now because a person asked ChatGPT toNot always

OpenAI says each setting is independent. It also says that if you allow both GPTBot and OAI-SearchBot, it may reuse one visit for both purposes. A fourth bot, OAI-AdsBot, only checks landing pages submitted as ChatGPT ads, and OpenAI says it isn't used for training.

Should you block GPTBot? Start from your goal

The answer depends on what you want, not on GPTBot alone. OpenAI ties appearing in ChatGPT search answers to OAI-SearchBot. Blocking GPTBot doesn't remove you from those answers on its own.

Your goalGPTBotOAI-SearchBotWhat happens
Stay in ChatGPT answers, no trainingBlockAllowYou can still be cited in search answers
Out of training and ChatGPT searchBlockBlockYou drop out of ChatGPT search answers
Fine with bothAllowAllowNo rules needed, unless your file blocks every bot (User-agent: * with Disallow: /)
Only out of ChatGPT searchAllowBlockRarely what anyone wants

So if you only want visibility in ChatGPT, OAI-SearchBot is the bot that matters. Opting out of OAI-SearchBot doesn't hide you completely: OpenAI says your site can still appear as a plain link, without your content quoted in the answer.

Copy the robots.txt rules for your goal

The rules come in groups: a group is a User-agent: line naming a bot, followed by the rules for that bot. User-agent: * holds the rules for every bot not named elsewhere in the file. Your robots.txt file sits at the root of your domain, never in a subfolder. Each subdomain, such as shop.yoursite.com, needs its own file. If your platform writes the file for you, look for a robots.txt setting in its SEO options, or send these lines to your developer.

Block training, stay in ChatGPT answers

text
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Block both training and ChatGPT search

text
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

Allow both

You don't need to name either bot. Only add an explicit Allow: / group for each one if your file blocks everyone with User-agent: * and Disallow: /.

OpenAI says changes can take about 24 hours to show up in ChatGPT search. Don't judge the result on the same day.

The trap: a bot with its own group ignores your other rules

Google's robots.txt guide states the rule plainly: a bot follows only one group, the most specific one that names it. Your User-agent: * rules are not added on top. So the moment you give OAI-SearchBot its own group, it stops reading your User-agent: * rules.

Here is a file that keeps every bot out of the checkout pages, before the change:

text
User-agent: *
Disallow: /checkout/

Add a group for OAI-SearchBot with only Allow: /, and it can now visit /checkout/. Copy your existing rules into the new group instead:

text
User-agent: *
Disallow: /checkout/

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /checkout/

Watch for the opposite trap after a redesign or a move from a test site. A leftover Disallow: / under User-agent: * blocks every bot that has no group of its own, OpenAI's included. Our guide to robots.txt errors shows how to spot it. While you're there, check for a forgotten noindex tag too: a line in a page's code that tells search engines not to list it.

Check that nothing blocks ChatGPT by mistake

Your robots.txt can be perfect while something else turns bots away. Run these checks after any change:

  1. Open yoursite.com/robots.txt in a private browser window. Confirm it loads and read every group that names GPTBot, OAI-SearchBot or *.
  2. In Google Search Console, open the robots.txt report to confirm Google can fetch the file. It tests Google's reading only, not OpenAI's.
  3. Check your CDN or firewall, the service that sits in front of your site. Cloudflare's AI crawler setting can block matching bots across the whole domain, whatever robots.txt says. Its Block action for training bots also hits bots used for both search and training.
  4. If you can read your server logs, compare visitors claiming to be OAI-SearchBot with the IP address lists OpenAI publishes on its crawler page.
  5. Wait about 24 hours, then check again.

UpCrawler's alerts include robots.txt validation, so a file that breaks after an update doesn't go unnoticed. If ChatGPT still never mentions you, work through why a site doesn't show up in ChatGPT.

robots.txt is a request, not a lock

Bots that fetch a page because a person asked for it may skip robots.txt. OpenAI says this of ChatGPT-User, and Perplexity says its Perplexity-User bot generally ignores it. To truly keep a page private, put it behind a login or block it at your server or firewall.

Other AI companies split their bots the same way:

CompanyTrainingSearch answersFetch for a person
OpenAIGPTBotOAI-SearchBotChatGPT-User
AnthropicClaudeBotClaude-SearchBotClaude-User
Perplexitynone listedPerplexityBotPerplexity-User

Anthropic documents its bots on its crawler help page, and Perplexity says PerplexityBot isn't used to train AI models. Google offers Google-Extended to opt out of Gemini training and grounding. Google says it doesn't affect inclusion or ranking in Google Search.

Changed your robots.txt and want a quick health check of your homepage? Run a free audit and get the results in about a minute.

Found this useful?

Share it with whoever runs your site’s SEO.

Ready when you are!Crawly, the UpCrawler mascot

Catch the next drop on day one.

UpCrawler watches your uptime, crawls every page and tracks Search Console, so you see the turn the day it happens.

14 days freeNo cardNothing to install