Who blocks AI · #149 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does arxiv.org block AI crawlers?
No. arxiv.org’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- arxiv.org/robots.txt · 342 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 51 · Disallow: /user line 52 · Disallow: /e-print and 24 more | |
Bingbot | Microsoft | Partly blocked | line 125 · Disallow: /user line 126 · Disallow: /e-print and 24 more |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Partly blocked | line 312 · Disallow: /user line 313 · Disallow: /e-print and 25 more |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Partly blocked | line 269 · Disallow: /e-print/ line 270 · Disallow: /src/ and 31 more |
Baiduspider | Baidu | Partly blocked | line 161 · Disallow: /user line 162 · Disallow: /e-print and 24 more |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Blocked | line 341 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View arxiv.org/robots.txt · 342 lines
1# robots.txt for http://arxiv.org/ and mirror sites http://*.arxiv.org/
2# Indiscriminate automated downloads from this site are not permitted
3# See also: http://arxiv.org/help/robots
4
5User-agent: *
6Crawl-delay: 15
7Allow: /archive
8Allow: /year
9Allow: /list
10Allow: /abs
11Allow: /pdf
12Allow: /html
13Allow: /catchup
14Disallow: /user
15Disallow: /e-print
16Disallow: /src
17Disallow: /ps
18Disallow: /dvi
19Disallow: /cookies
20Disallow: /form
21Disallow: /find
22Disallow: /view
23Disallow: /ftp
24Disallow: /refs
25Disallow: /cits
26Disallow: /format
27Disallow: /PS_cache
28Disallow: /Stats
29Disallow: /seek-and-destroy
30Disallow: /IgnoreMe
31Disallow: /oai2
32Disallow: /auth
33Disallow: /tb
34Disallow: /tb-recent
35Disallow: /trackback
36Disallow: /prevnext
37Disallow: /ct
38Disallow: /api
39Disallow: /search
40Disallow: /set_author_id
41Disallow: /show-email
42
43User-agent: Googlebot
44Allow: /archive
45Allow: /year
46Allow: /list
47Allow: /abs
48Allow: /pdf
49Allow: /html
50Allow: /catchup
51Disallow: /user
52Disallow: /e-print
53Disallow: /src
54Disallow: /ps
55Disallow: /dvi
56Disallow: /cookies
57Disallow: /form
58Disallow: /find
59Disallow: /view
60Disallow: /ftp
61Disallow: /refs
62Disallow: /cits
63Disallow: /format
64Disallow: /PS_cache
65Disallow: /Stats
66Disallow: /seek-and-destroy
67Disallow: /IgnoreMe
68Disallow: /oai2
69Disallow: /auth
70Disallow: /tb
71Disallow: /tb-recent
72Disallow: /trackback
73Disallow: /prevnext
74Disallow: /ct
75Disallow: /api
76Disallow: /search
77
78User-agent: Yahoo! Slurp
79Crawl-delay: 1
80Allow: /archive
81Allow: /year
82Allow: /list
83Allow: /abs
84Allow: /pdf
85Allow: /html
86Allow: /catchup
87Disallow: /user
88Disallow: /e-print
89Disallow: /src
90Disallow: /ps
91Disallow: /dvi
92Disallow: /cookies
93Disallow: /form
94Disallow: /find
95Disallow: /view
96Disallow: /ftp
97Disallow: /refs
98Disallow: /cits
99Disallow: /format
100Disallow: /PS_cache
101Disallow: /Stats
102Disallow: /seek-and-destroy
103Disallow: /IgnoreMe
104Disallow: /oai2
105Disallow: /auth
106Disallow: /tb
107Disallow: /tb-recent
108Disallow: /trackback
109Disallow: /prevnext
110Disallow: /ct
111Disallow: /api
112Disallow: /search
113
114User-agent: bingbot
115# 2021-10-14 - removed crawl-delay for Bingbot. Needs to be re-added if there are any problems.
116# 2021-10-26 - added back
117Crawl-delay: 1
118Allow: /archive
119Allow: /year
120Allow: /list
121Allow: /abs
122Allow: /pdf
123Allow: /html
124Allow: /catchup
125Disallow: /user
126Disallow: /e-print
127Disallow: /src
128Disallow: /ps
129Disallow: /dvi
130Disallow: /cookies
131Disallow: /form
132Disallow: /find
133Disallow: /view
134Disallow: /ftp
135Disallow: /refs
136Disallow: /cits
137Disallow: /format
138Disallow: /PS_cache
139Disallow: /Stats
140Disallow: /seek-and-destroy
141Disallow: /IgnoreMe
142Disallow: /oai2
143Disallow: /auth
144Disallow: /tb
145Disallow: /tb-recent
146Disallow: /trackback
147Disallow: /prevnext
148Disallow: /ct
149Disallow: /api
150Disallow: /search
151
152User-agent: Baiduspider
153Crawl-delay: 10
154Allow: /archive
155Allow: /year
156Allow: /list
157Allow: /abs
158Allow: /pdf
159Allow: /html
160Allow: /catchup
161Disallow: /user
162Disallow: /e-print
163Disallow: /src
164Disallow: /ps
165Disallow: /dvi
166Disallow: /cookies
167Disallow: /form
168Disallow: /find
169Disallow: /view
170Disallow: /ftp
171Disallow: /refs
172Disallow: /cits
173Disallow: /format
174Disallow: /PS_cache
175Disallow: /Stats
176Disallow: /seek-and-destroy
177Disallow: /IgnoreMe
178Disallow: /oai2
179Disallow: /auth
180Disallow: /tb
181Disallow: /tb-recent
182Disallow: /trackback
183Disallow: /prevnext
184Disallow: /ct
185Disallow: /api
186Disallow: /search
187
188User-agent: ToutiaoSpider
189Crawl-delay: 10
190Allow: /archive
191Allow: /year
192Allow: /list
193Allow: /abs
194Allow: /pdf
195Allow: /html
196Allow: /catchup
197Disallow: /user
198Disallow: /e-print
199Disallow: /src
200Disallow: /ps
201Disallow: /dvi
202Disallow: /cookies
203Disallow: /form
204Disallow: /find
205Disallow: /view
206Disallow: /ftp
207Disallow: /refs
208Disallow: /cits
209Disallow: /format
210Disallow: /PS_cache
211Disallow: /Stats
212Disallow: /seek-and-destroy
213Disallow: /IgnoreMe
214Disallow: /oai2
215Disallow: /auth
216Disallow: /tb
217Disallow: /tb-recent
218Disallow: /trackback
219Disallow: /prevnext
220Disallow: /ct
221Disallow: /api
222Disallow: /search
223
224User-agent: SQUID_configured_as_described_at_/help/faq/cache
225Crawl-delay: 10
226Allow: /list
227Allow: /abs
228Allow: /pdf
229Disallow: /archive
230Disallow: /year
231Disallow: /html
232Disallow: /catchup
233Disallow: /user
234Disallow: /e-print
235Disallow: /src
236Disallow: /ps
237Disallow: /dvi
238Disallow: /cookies
239Disallow: /form
240Disallow: /find
241Disallow: /view
242Disallow: /ftp
243Disallow: /refs
244Disallow: /cits
245Disallow: /format
246Disallow: /PS_cache
247Disallow: /Stats
248Disallow: /seek-and-destroy
249Disallow: /IgnoreMe
250Disallow: /oai2
251Disallow: /auth
252Disallow: /tb
253Disallow: /tb-recent
254Disallow: /trackback
255Disallow: /prevnext
256Disallow: /ct
257Disallow: /api
258Disallow: /search
259
260User-agent: YandexBot
261Crawl-delay: 1
262Allow: /archive
263Allow: /year
264Allow: /list
265Allow: /abs
266Allow: /pdf
267Allow: /html
268Allow: /catchup
269Disallow: /e-print/
270Disallow: /src/
271Disallow: /ps/
272Disallow: /psfigs/
273Disallow: /dvi/
274Disallow: /cookies/
275Disallow: /form/
276Disallow: /find/
277Disallow: /view/
278Disallow: /ftp/
279Disallow: /refs/
280Disallow: /cits/
281Disallow: /format/
282Disallow: /register
283Disallow: /submit
284Disallow: /replace
285Disallow: /cross
286Disallow: /jref
287Disallow: /paper_passwd/
288Disallow: /PS_cache/
289Disallow: /Stats/
290Disallow: /seek-and-destroy
291Disallow: /IgnoreMe
292Disallow: /uploads
293Disallow: /oai2
294Disallow: /auth
295Disallow: /tb
296Disallow: /tb-recent
297Disallow: /trackback
298Disallow: /prevnext
299Disallow: /ct
300Disallow: /api
301Disallow: /search
302
303User-agent: Applebot
304Crawl-delay: 1
305Allow: /archive
306Allow: /year
307Allow: /list
308Allow: /abs
309Allow: /pdf
310Allow: /html
311Allow: /catchup
312Disallow: /user
313Disallow: /e-print
314Disallow: /src
315Disallow: /ps
316Disallow: /dvi
317Disallow: /cookies
318Disallow: /form
319Disallow: /find
320Disallow: /view
321Disallow: /ftp
322Disallow: /refs
323Disallow: /cits
324Disallow: /format
325Disallow: /PS_cache
326Disallow: /Stats
327Disallow: /seek-and-destroy
328Disallow: /IgnoreMe
329Disallow: /oai2
330Disallow: /auth
331Disallow: /tb
332Disallow: /tb/recent
333Disallow: /tb-recent
334Disallow: /trackback
335Disallow: /prevnext
336Disallow: /ct
337Disallow: /api
338Disallow: /search
339
340User-agent: SemrushBot
341Disallow: /
342
What do the bots actually get?
This page reads robots.txt, which states what arxiv.org asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on arxiv.org Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does arxiv.org block GPTBot?
No. arxiv.org’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does arxiv.org block ClaudeBot?
No. arxiv.org’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does arxiv.org block Google-Extended?
No. arxiv.org’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does arxiv.org block PerplexityBot?
No. arxiv.org’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find arxiv.org?
Yes. arxiv.org’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.