Updated November 10, 2026.
Over the past two years, many businesses added a line to their robots.txt file to “block AI.” The intention is often good: you don’t want your content used to train models without permission. But depending on the line you added, you may also, without knowing it, disappear from ChatGPT’s answers.
Letting the right crawler read your website is the most technical of the three steps to show up in ChatGPT, and the only one that can block everything on its own. The good news: you can check it in two minutes.
Robots.txt is a small text file at the root of your website (yoursite.com/robots.txt). It tells search engine and AI assistant crawlers which parts of the site they may explore. Each crawler has a name, and you can give it specific rules.
OpenAI uses several crawlers, and that’s where the confusion starts. According to its documentation, each one is set independently of the others.
| Crawler | What it does | If you block it |
|---|---|---|
| OAI-SearchBot | Crawls websites to show them in ChatGPT search. | Your site no longer appears in ChatGPT search answers (it can still appear as a navigational link). |
| GPTBot | Crawls content that may be used to train OpenAI’s models. | Your content shouldn’t be used for training. No effect on search. |
| ChatGPT-User | Visits a page when a user asks for it in ChatGPT. | Robots.txt rules may not apply, since the visit comes from a user request. It isn’t used to decide what appears in search. |
| OAI-AdsBot | Checks pages submitted as ad destinations in ChatGPT. | Only concerns advertisers. |
The key point: blocking GPTBot doesn’t remove you from search, and blocking OAI-SearchBot doesn’t protect you from training. They’re two separate decisions.
If you want to be found in ChatGPT, but don’t want your content used to train models, here are the rules to add to your robots.txt:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /If, on the contrary, you want to allow everything, there’s nothing special to write: a robots.txt that doesn’t block these crawlers lets them in. On devjpt.com, for example, the robots.txt welcomes all crawlers, including AI assistants’, and only excludes the admin area and a few low-value pages.
Watch out for the most dangerous rule, still found on some websites:
User-agent: *
Disallow: /It blocks every crawler without a specific rule, including OAI-SearchBot and Bingbot. It’s sometimes left by mistake when a development site goes live.
Finally, OpenAI says it takes about 24 hours after a robots.txt change for its search systems to adjust.
A perfect robots.txt isn’t always enough. Some hosts, content delivery networks (CDNs) and security plugins block AI crawlers by default, or treat them as suspicious traffic. In that case, the crawler gets an error or a challenge page, and never reads your content.
OpenAI mentions it explicitly: to be eligible for search, your host or CDN must allow traffic from its crawler’s published IP addresses. These addresses are public, in a file provided by OpenAI (openai.com/searchbot.json).
If you don’t manage your hosting yourself, just ask your provider: “Do you block OAI-SearchBot, GPTBot or Bingbot?” Check Bingbot while you’re at it, since Bing is also one of ChatGPT’s search providers.
The fastest way: enter your address in the AI robots.txt checker. It applies your rules to each crawler from OpenAI, Microsoft, Google, Anthropic and Perplexity. To check by hand:
No. GPTBot is used for model training. According to OpenAI, each setting is independent: you can block GPTBot and allow OAI-SearchBot, which powers search.
Not always. ChatGPT-User visits a page when a user asks for it. Because the action comes from the user, OpenAI says robots.txt rules may not apply. This crawler isn’t used to decide what appears in search: OAI-SearchBot is the one that counts for that.
OpenAI mentions about 24 hours after a robots.txt change for its search systems to adjust. Actually being cited then depends on the content of your pages.
Yes. Some firewalls and bot protection services block AI crawlers by default. Ask your host whether it allows the IP addresses OpenAI publishes for OAI-SearchBot.
It isn’t necessary for ChatGPT search. OpenAI’s crawler documentation only mentions robots.txt and its IP addresses. An llms.txt file can help present your website to assistants, but it replaces neither a well-configured robots.txt nor pages that truly answer questions. I explain its real role in the article on the llms.txt file and AI assistants.