Your site can be perfectly visible in Google and completely absent from every AI assistant at the same time. The rules are matched per crawler, and almost nobody has read theirs.
Almost every site with this problem got it by accident. Nobody sat down and decided to be excluded from AI search. Somebody installed a security plugin with a "block aggressive bots" option, or pasted a robots.txt snippet from a forum thread during a scraping incident, or added a firewall rule at three in the morning to stop a traffic spike. The rule stayed. Google kept working, so nothing looked wrong.
How do I know if my site blocks AI crawlers?
Open your own robots.txt at yourdomain.com/robots.txt and read it rule by rule, matching each User-agent block against the crawler tokens the AI operators publish. A rule only applies to the user agents it names, so a site can allow Googlebot and block GPTBot in the same file without any warning. Checking the file by eye once is worthwhile, but the rules drift, which is why the check belongs on a schedule.
Two details trip people up when they read the file for the first time. The first is that a User-agent: * block is a fallback, not a global rule: if a crawler has its own named block anywhere in the file, the wildcard block does not apply to it at all. The second is that the most restrictive-looking line is not always the operative one — matching is by the most specific user-agent block, not by whichever rule appears last.
This is why "we allow everything" is such a common and such an unreliable belief. It is usually based on the wildcard block, which is exactly the block a named crawler ignores.
What is the difference between GPTBot and OAI-SearchBot?
Both are OpenAI, and they do different jobs. GPTBot collects pages that may be used as model training data. OAI-SearchBot builds the search index behind ChatGPT, and ChatGPT-User fetches a page live when someone asks a question that needs it. Blocking GPTBot keeps you out of training while leaving you citable; blocking OAI-SearchBot and ChatGPT-User takes you out of the answers.
Every major operator has this same split, and the naming gives you almost no help in telling them apart. Anthropic uses ClaudeBot for training and Claude-User for live retrieval. Google's training opt-out is Google-Extended, which is not a crawler at all — no request ever arrives with that user agent; it is a switch that Googlebot honours on Gemini's behalf. Apple works the same way with Applebot and Applebot-Extended.
The practical consequence is that a rule written against a name you half-remember is a coin flip. TrustCtrl keeps the catalogue and reports each crawler by operator and purpose, so the question you answer is "do we want to be cited by Perplexity?" rather than "is PerplexityBot the training one?".
Should I block AI crawlers?
It depends which ones, and the answer is rarely all or nothing. Blocking training crawlers keeps your writing out of model training data. Blocking retrieval crawlers removes you from the AI answers that would otherwise cite you and send visitors. Most businesses that think it through end up allowing retrieval and deciding about training separately, because the two have opposite commercial effects.
It helps to separate the two questions rather than answering one and assuming the other:
- Do we want to appear in AI answers? For almost every business selling something, yes. An assistant that cites you is doing what a search result does, with a stronger implicit recommendation. This argues for allowing the retrieval crawlers.
- Do we mind our content becoming training data? A legitimate business decision with no universal answer. Publishers and businesses whose writing is the product often mind a great deal. A shop with product pages usually does not. This is the training question, and it is separate.
- Are we trying to stop scraping? If so, robots.txt is the wrong tool — see below. Rate limiting and bot management are the right ones, and they should be configured not to catch the crawlers you want.
Does robots.txt actually stop AI crawlers?
It is a request, not a control. The major operators state that they honour robots.txt, and in practice the well-known crawlers do. It gives you no protection against a crawler that ignores it or identifies as something else, so robots.txt is the right tool for expressing a preference to legitimate operators and the wrong tool for stopping determined scraping.
Confusing the two leads to the worst of both outcomes. A business worried about scraping blocks everything in robots.txt, the scrapers carry on because they never read it, and the only parties that obey are the assistants that would have sent traffic. The site loses the visibility and keeps the scraping.
The rule that is not in robots.txt
Robots.txt is only half the picture, and the other half is harder to see. Your site can allow every AI crawler by the letter of the file and still serve them something useless, because a layer in front of the site is making its own decision: a bot-protection service issuing a JavaScript challenge, a CDN rule treating unfamiliar user agents as suspicious, a WAF that blocks anything without a browser fingerprint.
From the assistant's side the result is the same as a block. It requests your article and receives a "checking your browser" page, so it cites a competitor. From your side nothing looks wrong at all, because you never visit your own site as GPTBot.
The only reliable way to find this is to ask as each crawler and compare. TrustCtrl runs that probe — fetching your pages as each AI user agent and flagging where the response differs materially from what an ordinary visitor receives. It skips the robots-only tokens like Google-Extended, because probing with a user agent no crawler ever sends would prove nothing either way.
What to do this week
Read your robots.txt with the crawler list in front of you and decide the training question and the retrieval question separately. Check whether anything in front of your site — CDN, WAF, bot protection, security plugin — is filtering by user agent, and confirm the AI crawlers are on the allow side of it. Then make the check repeat, because the thing that breaks this is not the decision you make today; it is the unrelated change six months from now.
TrustCtrl reports every AI crawler as allowed, blocked or unknown on each crawl, splits them by training and retrieval, and probes for responses that differ by user agent. It sits alongside the rest of the site checks — whether your content survives without JavaScript, whether your sitemap is honest, speed, accessibility and security headers — so the whole picture arrives from one visit. The broader strategy sits in how to get cited by AI search.