Who blocks AI · #192 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does chatgpt.com block AI crawlers?
Yes. chatgpt.com’s robots.txt blocks all 19 AI crawlers we track, from GPTBot and ClaudeBot to PerplexityBot. It also blocks Googlebot, Bingbot and 5 more among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- chatgpt.com/robots.txt · 244 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Blocked | line 231 · Disallow: / under User-agent: * | |
Bingbot | Microsoft | Blocked | line 231 · Disallow: / under User-agent: * |
DuckDuckBot | DuckDuckGo | Blocked | line 231 · Disallow: / under User-agent: * |
Applebot | Apple | Blocked | line 231 · Disallow: / under User-agent: * |
PetalBot | Huawei | Blocked | line 231 · Disallow: / under User-agent: * |
YandexBot | Yandex | Blocked | line 231 · Disallow: / under User-agent: * |
Baiduspider | Baidu | Blocked | line 231 · Disallow: / under User-agent: * |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 231 · Disallow: / under User-agent: * |
PerplexityBot | Perplexity | Blocked | line 40 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 231 · Disallow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Blocked | line 231 · Disallow: / under User-agent: * |
Amazonbot | Amazon | Blocked | line 231 · Disallow: / under User-agent: * |
YouBot | You.com | Blocked | line 231 · Disallow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 231 · Disallow: / under User-agent: * |
ClaudeBot | Anthropic | Blocked | line 231 · Disallow: / under User-agent: * |
Google-Extended | Blocked | line 10 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 4 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 231 · Disallow: / under User-agent: * |
GoogleOther | Blocked | line 231 · Disallow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Blocked | line 231 · Disallow: / under User-agent: * |
Bytespider | ByteDance | Blocked | line 34 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 231 · Disallow: / under User-agent: * |
Claude-User | Anthropic | Blocked | line 231 · Disallow: / under User-agent: * |
Perplexity-User | Perplexity | Blocked | line 231 · Disallow: / under User-agent: * |
MistralAI-User | Mistral | Blocked | line 231 · Disallow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Blocked | line 231 · Disallow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 231 · Disallow: / under User-agent: * |
SemrushBot | Semrush | Blocked | line 231 · Disallow: / under User-agent: * |
MJ12bot | Majestic | Blocked | line 231 · Disallow: / under User-agent: * |
DotBot | Moz | Blocked | line 231 · Disallow: / under User-agent: * |
View chatgpt.com/robots.txt · 244 lines
1# https://www.robotstxt.org/robotstxt.html
2
3User-agent: CCBot
4Disallow: /
5
6User-agent: img2dataset
7Disallow: /
8
9User-agent: Google-Extended
10Disallow: /
11
12# Prevent shared pages and GPTs from being used as Google Ads destinations
13User-agent: AdsBot-Google
14User-agent: AdsBot-Google-Mobile
15Disallow: /s/
16Disallow: /g/
17
18User-agent: anthropic-ai
19Disallow: /
20
21User-agent: Claude-Web
22Disallow: /
23
24User-agent: Omgilibot
25Disallow: /
26
27User-agent: Omgili
28Disallow: /
29
30User-agent: FacebookBot
31Disallow: /
32
33User-agent: Bytespider
34Disallow: /
35
36User-agent: magpie-crawler
37Disallow: /
38
39User-agent: PerplexityBot
40Disallow: /
41
42User-agent: Perplexity‑User
43Disallow: /
44
45# General rules for all other bots
46User-agent: *
47# Allow Googlebot to render the indexable homepage; production CSS and JavaScript live here.
48Allow: /unauth-mweb/assets/
49
50# Place allows first to avoid bots skipping after Disallow: /
51# Allow exactly the homepage
52Allow: /$
53# Allow the homepage with any query parameters
54Allow: /?*
55Allow: /api/share/og/
56Allow: /g/
57Allow: /s/
58Allow: /gg/v/
59Allow: /m/
60Allow: /share/
61Allow: /maps$
62Allow: /maps/
63Allow: /*/maps$
64Allow: /*/maps/
65Allow: /canvas/shared/
66Allow: /*/images
67Allow: /images
68Allow: /*/library
69Allow: /library
70Allow: /favicon.ico
71Allow: /assets/favicon
72Allow: /cdn/assets/favicon
73Allow: /cdn/assets/
74Allow: /auth/
75Allow: /gpts$
76Allow: /codex
77Allow: /*/codex
78Allow: /search$
79Allow: /backend-anon/
80Allow: /public-api/
81Allow: /sitemap.xml
82Allow: /marketing-sitemap.xml
83Allow: /writing-tools-sitemap.xml
84Allow: /acquisition-landing-pages-sitemap.xml
85Allow: /100chats
86Allow: /api/public_content/
87Allow: /backend-api/public_content/
88Allow: /?ref=dotcom
89# Static Landing Pages
90Allow: /overview
91Allow: /*/overview
92Allow: /features
93Allow: /*/features
94Allow: /health$
95Allow: /health/$
96Allow: /*/health$
97Allow: /*/health/$
98Allow: /voice$
99Allow: /voice/$
100Allow: /*/voice$
101Allow: /*/voice/$
102Allow: /apps
103Allow: /*/apps
104Allow: /finances
105Allow: /*/finances
106Allow: /shopping
107Allow: /*/shopping
108Allow: /use-cases
109Allow: /*/use-cases
110Allow: /learn
111Allow: /*/learn
112Allow: /business
113Allow: /*/business
114Allow: /pricing
115Allow: /*/pricing
116Allow: /download
117Allow: /*/download
118Allow: /students
119Allow: /students/2026
120Allow: /*/college-students
121Allow: /college-students
122Allow: /contact-sales
123Allow: /*/contact-sales
124Allow: /100chats-project
125Allow: /*/100chats-project
126Allow: /merchants
127Allow: /*/merchants
128Allow: /parent-resources
129Allow: /*/parent-resources
130Allow: /plans
131Allow: /*/plans
132Allow: /plugins
133Allow: /translate
134Allow: /*/translate
135Allow: /writing
136Allow: /*/writing
137Allow: /ask/
138Allow: /futures
139Allow: /*/futures
140Allow: /football
141Allow: /*/football
142Allow: /remote
143Allow: /*/remote
144Allow: /work
145Allow: /*/work
146# Allow crawlers to read the campaign page's noindex directive.
147Allow: /get-dot$
148Allow: /get-dot/$
149Allow: /get-dot?
150Allow: /get-dot/?
151Allow: /*/get-dot$
152Allow: /*/get-dot/$
153Allow: /*/get-dot?
154Allow: /*/get-dot/?
155Allow: /dots
156Allow: /*/dots
157Allow: /sites
158Allow: /*/sites
159Allow: /space
160Allow: /*/space
161Allow: /import-to-chatgpt
162Allow: /*/import-to-chatgpt
163# Exact locale specific homepages
164Allow: /am/$
165Allow: /ar/$
166Allow: /bg-BG/$
167Allow: /bn-BD/$
168Allow: /bs-BA/$
169Allow: /ca-ES/$
170Allow: /cs-CZ/$
171Allow: /da-DK/$
172Allow: /de-DE/$
173Allow: /el-GR/$
174Allow: /es-ES/$
175Allow: /es-419/$
176Allow: /et-EE/$
177Allow: /fa-IR/$
178Allow: /fi-FI/$
179Allow: /fr-FR/$
180Allow: /fr-CA/$
181Allow: /gu-IN/$
182Allow: /he-IL/$
183Allow: /hi-IN/$
184Allow: /hr-HR/$
185Allow: /hu-HU/$
186Allow: /hy-AM/$
187Allow: /id-ID/$
188Allow: /is-IS/$
189Allow: /it-IT/$
190Allow: /ja-JP/$
191Allow: /ka-GE/$
192Allow: /kk/$
193Allow: /kn-IN/$
194Allow: /ko-KR/$
195Allow: /lt/$
196Allow: /lv-LV/$
197Allow: /mk-MK/$
198Allow: /ml/$
199Allow: /mn/$
200Allow: /mr-IN/$
201Allow: /ms-MY/$
202Allow: /my-MM/$
203Allow: /nb-NO/$
204Allow: /nl-NL/$
205Allow: /pa/$
206Allow: /pl-PL/$
207Allow: /pt-BR/$
208Allow: /pt-PT/$
209Allow: /ro-RO/$
210Allow: /ru-RU/$
211Allow: /sk-SK/$
212Allow: /sl-SI/$
213Allow: /so-SO/$
214Allow: /sq-AL/$
215Allow: /sr-RS/$
216Allow: /sv-SE/$
217Allow: /sw-TZ/$
218Allow: /ta-IN/$
219Allow: /te-IN/$
220Allow: /th-TH/$
221Allow: /tl/$
222Allow: /tr-TR/$
223Allow: /uk-UA/$
224Allow: /ur/$
225Allow: /vi-VN/$
226Allow: /zh-CN/$
227Allow: /zh-TW/$
228Allow: /zh-HK/$
229
230# Now block everything else
231Disallow: /
232
233# Specific disallows (redundant for some bots, but still useful for those that respect precedence)
234Disallow: /auth/logout
235Disallow: /auth/login?*
236Disallow: /backend-anon/sentinel/*
237Disallow: /backend-anon/conversation$
238Disallow: /account-link/*
239
240Sitemap: https://chatgpt.com/sitemap.xml
241Sitemap: https://chatgpt.com/marketing-sitemap.xml
242Sitemap: https://chatgpt.com/writing-tools-sitemap.xml
243Sitemap: https://chatgpt.com/acquisition-landing-pages-sitemap.xml
244
What do the bots actually get?
This page reads robots.txt, which states what chatgpt.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on chatgpt.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does chatgpt.com block GPTBot?
Yes. chatgpt.com’s robots.txt disallows GPTBot from the whole site (line 231: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does chatgpt.com block ClaudeBot?
Yes. chatgpt.com’s robots.txt disallows ClaudeBot from the whole site (line 231: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does chatgpt.com block Google-Extended?
Yes. chatgpt.com’s robots.txt disallows Google-Extended from the whole site (line 10: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does chatgpt.com block PerplexityBot?
Yes. chatgpt.com’s robots.txt disallows PerplexityBot from the whole site (line 40: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find chatgpt.com?
Not through its own crawler. chatgpt.com’s robots.txt disallows OAI-SearchBot from the whole site (line 231: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.