Who blocks AI · #1313 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does gandi.net block AI crawlers?
No. gandi.net’s robots.txt lets all 19 AI crawlers we track crawl the site. It also blocks YandexBot among search engines.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.gandi.net/robots.txt · 466 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Blocked | line 426 · Disallow: / under User-agent: YandexBot |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 50 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Blocked | line 114 · Disallow: / under User-agent: dotbot |
View www.gandi.net/robots.txt · 466 lines
1User-agent: *
2
3
4###### Sitemaps ######
5Sitemap: https://www.gandi.net/sitemap.xml
6Sitemap: https://news.gandi.net/sitemaps.xml
7
8###### Disalow ######
9Disallow: /*redirect=*
10Disallow: /*navigation=*
11Disallow: /*client_id=*
12Disallow: /*locale=*
13Disallow: /*tld=*
14Disallow: /*bulk=*
15Disallow: /*bu_id=*
16Disallow: /*opened=*
17Disallow: /*type=*
18Disallow: /*template=*
19Disallow: /*size=*
20Disallow: /*options=*
21Disallow: /*error=*
22Disallow: /*grid=*
23Disallow: /*currency=*
24Disallow: /*?c=*
25Disallow: /*&c=*
26Disallow: /*?sw=*
27Disallow: /*&sw=*
28Disallow: /*taxes=*
29Disallow: /pdf/*
30Disallow: /*/sample/*
31Disallow: */prices$
32Disallow: */rules$
33Disallow: */dates$
34Disallow: */news$
35Allow: */tld/news$
36Disallow: *?country=*
37Disallow: *?l=*
38Disallow: *?lang=*
39Disallow: /gcart/*
40
41
42User-Agent: facebookexternalhit/1.1
43Disallow:
44
45User-Agent: Facebot
46Disallow:
47
48###### Suppression robots ######
49User-agent: AhrefsBot
50Disallow: /
51User-agent: Alexibot
52Disallow: /
53User-agent: Aqua_Products
54Disallow: /
55User-agent: asterias
56Disallow: /
57User-agent: b2w/0.1
58Disallow: /
59User-agent: BackDoorBot/1.0
60Disallow: /
61User-agent: BacklinkCrawler
62Disallow: /
63User-agent: bdbrandprotect
64Disallow: /
65User-agent: Black Hole
66Disallow: /
67User-agent: BlowFish/1.0
68Disallow: /
69User-agent: boitho.com-dc
70Disallow: /
71User-agent: Bookmark search tool
72Disallow: /
73User-agent: BotALot
74Disallow: /
75User-agent: BotRightHere
76Disallow: /
77User-agent: BPImageWalker
78Disallow: /
79User-agent: BuiltBotTough
80Disallow: /
81User-agent: Bullseye/1.0
82Disallow: /
83User-agent: BunnySlippers
84Disallow: /
85User-agent: CazoodleBot
86Disallow: /
87User-agent: Cegbfeieh
88Disallow: /
89User-agent: CheeseBot
90Disallow: /
91User-agent: CherryPicker
92Disallow: /
93User-agent: CherryPickerElite/1.0
94Disallow: /
95User-agent: CherryPickerSE/1.0
96Disallow: /
97User-agent: ConveraCrawler
98Disallow: /
99User-agent: Copernic
100Disallow: /
101User-agent: CopyRightCheck
102Disallow: /
103User-agent: cosmos
104Disallow: /
105User-agent: Crescent
106Disallow: /
107User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
108Disallow: /
109User-agent: discobot
110Disallow: /
111User-agent: DittoSpyder
112Disallow: /
113User-agent: dotbot
114Disallow: /
115User-agent: Download Ninja
116Disallow: /
117User-agent: EmailCollector
118Disallow: /
119User-agent: EmailSiphon
120Disallow: /
121User-agent: EmailWolf
122Disallow: /
123User-agent: EroCrawler
124Disallow: /
125User-agent: ExtractorPro
126Disallow: /
127User-agent: Ezooms
128Disallow: /
129User-agent: FairAd Client
130Disallow: /
131User-agent: Fasterfox
132Disallow: /
133User-agent: Fetch
134Disallow: /
135User-agent: findlinks
136Disallow: /
137User-agent: Flaming AttackBot
138Disallow: /
139User-agent: Foobot
140Disallow: /
141User-agent: Gaisbot
142Disallow: /
143User-agent: GetRight/4.2
144Disallow: /
145User-agent: Gigabot
146Disallow: /
147User-agent: grub
148Disallow: /
149User-agent: grub-client
150Disallow: /
151User-agent: Harvest/1.5
152Disallow: /
153User-agent: heritrix
154Disallow: /
155User-agent: hloader
156Disallow: /
157User-agent: httplib
158Disallow: /
159User-agent: HTTrack
160Disallow: /
161User-agent: HTTrack 3.0
162Disallow: /
163User-agent: HuaweiSymantecSpider
164Disallow: /
165User-agent: humanlinks
166Disallow: /
167User-agent: ichiro
168Disallow: /
169User-agent: InfoNaviRobot
170Disallow: /
171User-agent: IRLbot
172Disallow: /
173User-agent: Iron33/1.0.2
174Disallow: /
175User-agent: JennyBot
176Disallow: /
177User-agent: kalooga
178Disallow: /
179User-agent: Kenjin Spider
180Disallow: /
181User-agent: Keyword Density/0.9
182Disallow: /
183User-agent: larbin
184Disallow: /
185User-agent: LexiBot
186Disallow: /
187User-agent: LexxeBot
188Disallow: /
189User-agent: libWeb/clsHTTP
190Disallow: /
191User-agent: libwww
192Disallow: /
193User-agent: LinkextractorPro
194Disallow: /
195User-agent: linko
196Disallow: /
197User-agent: LinkScan/8.1a Unix
198Disallow: /
199User-agent: LinkWalker
200Disallow: /
201User-agent: LNSpiderguy
202Disallow: /
203User-agent: lwp-trivial
204Disallow: /
205User-agent: lwp-trivial/1.34
206Disallow: /
207User-agent: Mata Hari
208Disallow: /
209User-agent: Microsoft URL Control
210Disallow: /
211User-agent: Microsoft URL Control - 5.01.4511
212Disallow: /
213User-agent: Microsoft URL Control - 6.00.8169
214Disallow: /
215User-agent: Microsoft.URL.Control
216Disallow: /
217User-agent: MIIxpc
218Disallow: /
219User-agent: MIIxpc/4.2
220Disallow: /
221User-agent: Mister PiX
222Disallow: /
223User-agent: MJ12bot
224Disallow: /
225User-agent: MLBot
226Disallow: /
227User-agent: moget
228Disallow: /
229User-agent: moget/2.1
230Disallow: /
231User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
232Disallow: /
233User-agent: MSIECrawler
234Disallow: /
235User-agent: NaverBot
236Disallow: /
237User-agent: NerdyBot
238Disallow: /
239User-agent: Net\ Vampire
240Disallow: /
241User-agent: NetAnts
242Disallow: /
243User-agent: NetMechanic
244Disallow: /
245User-agent: NICErsPRO
246Disallow: /
247User-agent: Nutch
248Disallow: /
249User-agent: Ocelli
250Disallow: /
251User-agent: Offline Explorer
252Disallow: /
253User-agent: Openbot
254Disallow: /
255User-agent: Openfind
256Disallow: /
257User-agent: Openfind data gatherer
258Disallow: /
259User-agent: Oracle Ultra Search
260Disallow: /
261User-agent: penthesilea
262Disallow: /
263User-agent: PerMan
264Disallow: /
265User-agent: PhpDig
266Disallow: /
267User-agent: ProPowerBot/2.14
268Disallow: /
269User-agent: ProWebWalker
270Disallow: /
271User-agent: psbot
272Disallow: /
273User-agent: Python-urllib
274Disallow: /
275User-agent: QueryN Metasearch
276Disallow: /
277User-agent: Radiation Retriever 1.1
278Disallow: /
279User-agent: RepoMonkey
280Disallow: /
281User-agent: RepoMonkey Bait & Tackle/v1.01
282Disallow: /
283User-agent: RMA
284Disallow: /
285User-agent: rogerbot
286Disallow: /
287User-agent: SCOUTjET
288Disallow: /
289User-agent: SearchDaimon.com-dc
290Disallow: /
291User-agent: searchpreview
292Disallow: /
293User-agent: SEOENGBot
294Disallow: /
295User-agent: ShopWiki
296Disallow: /
297User-agent: sitecheck.internetseer.com
298Disallow: /
299User-agent: SiteSnagger
300Disallow: /
301User-agent: Snapbot
302Disallow: /
303User-agent: Sogou\ web\ spider
304Disallow: /
305User-agent: Sosospider
306Disallow: /
307User-agent: SpankBot
308Disallow: /
309User-agent: spanner
310Disallow: /
311User-agent: spbot
312Disallow: /
313User-agent: SurveyBot
314Disallow: /
315User-agent: suzuran
316Disallow: /
317User-agent: Szukacz/1.4
318Disallow: /
319User-agent: Teleport
320Disallow: /
321User-agent: TeleportPro
322Disallow: /
323User-agent: Telesoft
324Disallow: /
325User-agent: Templeton
326Disallow: /
327User-agent: The Intraformant
328Disallow: /
329User-agent: TheNomad
330Disallow: /
331User-agent: TightTwatBot
332Disallow: /
333User-agent: Titan
334Disallow: /
335User-agent: toCrawl/UrlDispatcher
336Disallow: /
337User-agent: True_Robot
338Disallow: /
339User-agent: True_Robot/1.0
340Disallow: /
341User-agent: turingos
342Disallow: /
343User-agent: TurnitinBot
344Disallow: /
345User-agent: TurnitinBot/1.5
346Disallow: /
347User-agent: TwengaBot
348Disallow: /
349User-agent: Twiceler
350Disallow: /
351User-agent: Updownerbot
352Disallow: /
353User-agent: URL Control
354Disallow: /
355User-agent: URL_Spider_Pro
356Disallow: /
357User-agent: URLy Warning
358Disallow: /
359User-agent: VCI
360Disallow: /
361User-agent: VCI WebViewer VCI WebViewer Win32
362Disallow: /
363User-agent: Web Image Collector
364Disallow: /
365User-agent: WebAuto
366Disallow: /
367User-agent: WebBandit
368Disallow: /
369User-agent: WebBandit/3.50
370Disallow: /
371User-agent: WebCapture 2.0
372Disallow: /
373User-agent: WebCopier
374Disallow: /
375User-agent: WebCopier v.2.2
376Disallow: /
377User-agent: WebCopier v3.2a
378Disallow: /
379User-agent: WebEnhancer
380Disallow: /
381User-agent: WebmasterWorldForumBot
382Disallow: /
383User-agent: WebSauger
384Disallow: /
385User-agent: Website Quester
386Disallow: /
387User-agent: Webster Pro
388Disallow: /
389User-agent: WebStripper
390Disallow: /
391User-agent: WebZIP
392Disallow: /
393User-agent: WebZip/4.0
394Disallow: /
395User-agent: WebZIP/4.21
396Disallow: /
397User-agent: WebZIP/5.0
398Disallow: /
399User-agent: wget
400Disallow: /
401User-agent: Wget/1.5.3
402Disallow: /
403User-agent: Wget/1.6
404Disallow: /
405User-agent: woozweb-monitoring
406Disallow: /
407User-agent: woriobot
408Disallow: /
409User-agent: wprocketbot
410Allow: /
411User-agent: WWW-Collector-E
412Disallow: /
413User-agent: Xenu
414Disallow: /
415User-agent: Xenu's
416Disallow: /
417User-agent: Xenu's Link Sleuth 1.1c
418Disallow: /
419User-agent: YACYBIT
420Disallow: /
421User-agent: yacybot
422Disallow: /
423User-agent: Yandex
424Disallow: /
425User-agent: YandexBot
426Disallow: /
427User-agent: Yeti
428Disallow: /
429User-agent: YodaoBot
430Disallow: /
431User-agent: Zealbot
432Disallow: /
433User-agent: Zeus
434Disallow: /
435User-agent: Zeus 32297 Webster Pro V2.9 Win32
436Disallow: /
437User-agent: Zeus Link Scout
438Disallow: /
439User-agent: ZyBorg
440Disallow: /
441User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
442Disallow: /
443User-agent: NerdyBot
444Disallow: /
445User-agent: WebAuto
446Disallow: /
447User-agent: wprocketbot
448Disallow: /
449User-agent: DIIbot
450Disallow: /
451User-agent: eCatch
452Disallow: /
453User-agent: Jetbot
454Disallow: /
455User-agent: Openfind data gathere
456Disallow: /
457User-agent: QuepasaCreep
458Disallow: /
459User-agent: Roverbot
460Disallow: /
461User-agent: Foobot
462Disallow: /
463User-agent: RepoMonkey Bait & Tackle/v1.01
464Disallow: /
465User-agent: ExtractorPro
466Disallow: /
What do the bots actually get?
This page reads robots.txt, which states what gandi.net asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on gandi.net Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does gandi.net block GPTBot?
No. gandi.net’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does gandi.net block ClaudeBot?
No. gandi.net’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does gandi.net block Google-Extended?
No. gandi.net’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does gandi.net block PerplexityBot?
No. gandi.net’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find gandi.net?
Yes. gandi.net’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.