Who blocks AI · #510 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does lemonde.fr block AI crawlers?
Yes. lemonde.fr’s robots.txt blocks 12 of the 19 AI crawlers we track, including ClaudeBot, Google-Extended, CCBot and 9 more. It also blocks PetalBot among search engines.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotAllowedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.lemonde.fr/robots.txt · 1,094 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 688 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Blocked | line 700 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 224 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 212 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 667 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Blocked | line 203 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 57 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 194 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 206 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Blocked | line 712 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Blocked | line 310 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Blocked | line 703 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Blocked | line 233 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Blocked | line 577 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.lemonde.fr/robots.txt · 1,094 lines
1# 16/08/2019
2# Il est interdit d'utiliser des robots d'indexation Web ou d'autres méthodes automatiques de feuilletage ou de navigation sur ce site Web.
3# Nous interdisons de crawler notre site Web en utilisant un agent d'utilisateur volé qui ne correspond pas à votre identité.
4# « Violation du droit du producteur de base de données - article L 342-1 et suivant le Code de la propriété intellectuelle ».
5# Nous vous invitons à nous contacter pour contracter une licence d'utilisation. Seuls les partenaires sont habilités à utiliser nos contenus pour un usage autre que strictement individuel.
6User-agent: *
7Allow: /ws/1/live/*
8Allow: /ws/1/related_content/*
9Disallow: /ajax/
10Disallow: /ajah/
11Disallow: /api/
12Disallow: /beta
13Disallow: /element/commun/afficher/
14Disallow: /petites-annonces/
15Disallow: /qui-sommes-nous/
16Disallow: /txt/
17Disallow: /verification/source/*
18Disallow: /noscript/
19Disallow: /ws/*
20Disallow: /lemonde-beta/*
21Disallow: /_rprt/*
22Disallow: /layout/*
23Disallow: /cgi-bin/*
24Disallow: /envoyer-par-email/*
25Disallow: /lmdgft/*
26Disallow: /article-offert/*
27Disallow: /*?s=43260*
28Disallow: /*?contributions
29Disallow: */reactions/
30Disallow: */mmpub/
31# WordPress
32Disallow: /blog/*/wp-admin/
33Disallow: /blog/*/wp-includes/
34Disallow: /blog/*/wp-content/plugins/
35Disallow: /blog/*/wp-content/themes/
36Disallow: /blog/*/wp-login.php
37Disallow: /blog/*/wp-register.php
38Disallow: /blog/*/author/admin/
39# Recherche
40Disallow: /recherche/?*search_keywords=*
41# Sitemaps
42Sitemap: https://www.lemonde.fr/sitemap_news.xml
43Sitemap: https://www.lemonde.fr/sitemap_index.xml
44# Sitemaps EN
45Sitemap: https://www.lemonde.fr/en/sitemap_news.xml
46Sitemap: https://www.lemonde.fr/en/sitemap_index.xml
47
48User-agent: Googlebot-Image
49Allow: /image/
50
51User-agent: Google-Extended
52Allow: /innovation-et-mobilite/
53Allow: /les-conseils-pour-entreprendre/
54Allow: /mutuelle-sante-bien-choisir-sa-couverture/
55Allow: /prevoir-demain/
56Allow: /de-l-impact-aux-responsabilites/
57Disallow: /
58
59User-agent: Googlebot-News
60Disallow: /archives/
61
62# Robots exclus de toute indexation.
63User-agent: Cision
64Disallow: /
65
66User-agent: Talkwater
67Disallow: /
68
69User-agent: Jetbot
70Disallow: /
71
72User-agent: kbcrawl
73Disallow: /
74
75User-agent: Newzbin
76Disallow: /
77
78User-agent: Qwam content intelligence
79Disallow: /
80
81User-agent: Youmag
82Disallow: /
83
84User-agent: Synthesio
85Disallow: /
86
87User-agent: trendybuzz
88Disallow: /
89
90User-agent: scoop.it
91Disallow: /
92
93User-agent: linkfluence
94Disallow: /
95
96User-agent: grub-client
97Disallow: /
98
99User-agent: ia_archiver-web.archive.org
100Allow: /$
101Disallow: /*
102
103User-agent: k2spider
104Disallow: /
105
106User-agent: libwww
107Disallow: /
108
109User-agent: wget
110Disallow: /
111
112User-agent: 5erue
113Disallow: /
114
115User-agent: adequat
116Disallow: /
117
118User-agent: adequat-systems
119Disallow: /
120
121User-agent: coexel
122Disallow: /
123
124User-agent: leadbox
125Disallow: /
126
127User-agent: mention
128Disallow: /
129
130User-agent: mytwip
131Disallow: /
132
133User-agent: opinion-tracker
134Disallow: /
135
136User-agent: proxem
137Disallow: /
138
139User-agent: score3
140Disallow: /
141
142User-agent: vecteurplus
143Disallow: /
144
145User-agent: verticalsearch
146Disallow: /
147
148User-agent: vsw
149Disallow: /
150
151User-agent: winello
152Disallow: /
153
154User-agent: Fetch
155Disallow: /
156
157User-agent: infoseek
158Disallow: /
159
160User-agent: MSIECrawler
161Disallow: /
162
163User-agent: Offline Explorer
164Disallow: /
165
166User-agent: sitecheck.internetseer.com
167Disallow: /
168
169User-agent: Teleport
170Disallow: /
171
172User-agent: TeleportPro
173Disallow: /
174
175User-agent: WebCopier
176Disallow: /
177
178User-agent: WebStripper
179Disallow: /
180
181User-agent: Zealbot
182Disallow: /
183
184User-agent: asknread.com
185Disallow: /
186
187User-agent: omgilibot
188Disallow: /
189
190User-agent: omgili
191Disallow: /
192
193User-agent: CCBot
194Disallow: /
195
196User-agent: anthropic-ai
197Disallow: /
198
199User-agent: Claude-Web
200Disallow: /
201
202User-agent: ClaudeBot
203Disallow: /
204
205User-agent: Applebot-Extended
206Disallow: /
207
208User-agent: Webzio-Extended
209Disallow: /
210
211User-agent: Amazonbot
212Disallow: /
213
214User-agent: Timpibot
215Disallow: /
216
217User-agent: AI2Bot
218Disallow: /
219
220User-agent: cohere-training-data-crawler
221Disallow: /
222
223User-agent: DuckAssistBot
224Disallow: /
225
226User-agent: Kangaroo Bot
227Disallow: /
228
229User-agent: PanguBot
230Disallow: /
231
232User-agent: MistralAI-User
233Disallow: /
234
235# Cas particulier pour les bots de Facebook
236User-agent: facebookbot
237Disallow: /
238
239User-agent: FacebookBot
240Disallow: /
241
242# LISTEROBOTS1802
243User-agent: 5emeRue
244Disallow: /
245
246User-agent: ACQUIRE MEDIA
247Disallow: /
248
249User-agent: ACTIV Financial (CME Group)
250Disallow: /
251
252User-agent: AlphaSense
253Disallow: /
254
255User-agent: AmiSoftware
256Disallow: /
257
258User-agent: archive.org_bot
259Disallow: /
260
261User-agent: Archive-It
262Disallow: /
263
264User-agent: ArgClrInt
265Disallow: /
266
267User-agent: Ask n read
268Disallow: /
269
270User-agent: Augure
271Disallow: /
272
273User-agent: auramundi
274Disallow: /
275
276User-agent: AwarioRssBot
277Disallow: /
278
279User-agent: AwarioSmartBot
280Disallow: /
281
282User-agent: Barchart.com
283Disallow: /
284
285User-agent: BattleFin
286Disallow: /
287
288User-agent: Bernin IT
289Disallow: /
290
291User-agent: Blackboard Safeassign
292Disallow: /
293
294User-agent: BLP_bbot
295Disallow: /
296
297User-agent: bluematrix
298Disallow: /
299
300User-agent: Brandwatch
301Disallow: /
302
303User-agent: Briefcase.news
304Disallow: /
305
306User-agent: Buck
307Disallow: /
308
309User-agent: Bytespider
310Disallow: /
311
312User-agent: CCBo
313Disallow: /
314
315User-agent: CikisiBot
316Disallow: /
317
318User-agent: Coexel
319Disallow: /
320
321User-agent: cohere-ai
322Disallow: /
323
324User-agent: Comtex News Network
325Disallow: /
326
327User-agent: ConveraCrawler
328Disallow: /
329
330User-agent: Copyright Licensing Agency
331Disallow: /
332
333User-agent: Corporama
334Disallow: /
335
336User-agent: D&B Hoovers
337Disallow: /
338
339User-agent: Data Expression
340Disallow: /
341
342User-agent: Data Observer
343Disallow: /
344
345User-agent: Dataminr
346Disallow: /
347
348User-agent: Dealogic
349Disallow: /
350
351User-agent: Diffbot
352Disallow: /
353
354User-agent: Digimind
355Disallow: /
356
357User-agent: DirectFN
358Disallow: /
359
360User-agent: Dun & Bradstreet
361Disallow: /
362
363User-agent: Dun & Bradstreet - D&B ESG Intelligence
364Disallow: /
365
366User-agent: Dun & Bradstreet Data Marketplace
367Disallow: /
368
369User-agent: Eagle Alpha
370Disallow: /
371
372User-agent: ecoresearch
373Disallow: /
374
375User-agent: ellisphere
376Disallow: /
377
378User-agent: FactSet
379Disallow: /
380
381User-agent: FeedCheck
382Disallow: /
383
384User-agent: FeedReader
385Disallow: /
386
387User-agent: Feedspot
388Disallow: /
389
390User-agent: Fitch Solutions
391Disallow: /
392
393User-agent: Founder Apabi
394Disallow: /
395
396User-agent: Freshbot
397Disallow: /
398
399User-agent: FriendlyCrawler
400Disallow: /
401
402User-agent: Gnowit
403Disallow: /
404
405User-agent: GnowitNewsbot
406Disallow: /
407
408User-agent: Ground News
409Disallow: /
410
411User-agent: ia_archiver
412Disallow: /
413
414User-agent: ICE Connect Desktop Solution
415Disallow: /
416
417User-agent: ICE Data Services
418Disallow: /
419
420User-agent: IHS Markit
421Disallow: /
422
423User-agent: ImageSift
424Disallow: /
425
426User-agent: InMédia Technologies
427Disallow: /
428
429User-agent: Innguma
430Disallow: /
431
432User-agent: Inoreader
433Disallow: /
434
435User-agent: ISI Emerging Markets
436Disallow: /
437
438User-agent: KB Crawl SAS
439Disallow: /
440
441User-agent: Knowings
442Disallow: /
443
444User-agent: Koyfin
445Disallow: /
446
447User-agent: Launchmetrics
448Disallow: /
449
450User-agent: LexisNexis
451Disallow: /
452
453User-agent: Liana
454Disallow: /
455
456User-agent: magpie-crawler
457Disallow: /
458
459User-agent: Make/production
460Disallow: /
461
462User-agent: MarketResearch.com
463Disallow: /
464
465User-agent: MarketWatch
466Disallow: /
467
468User-agent: MarketWise
469Disallow: /
470
471User-agent: Markit Digital
472Disallow: /
473
474User-agent: Mediatoolkitbot
475Disallow: /
476
477User-agent: moduleQ
478Disallow: /
479
480User-agent: MONITIO
481Disallow: /
482
483User-agent: MoodleBot
484Disallow: /
485
486User-agent: Moody's
487Disallow: /
488
489User-agent: Moreover
490Disallow: /
491
492User-agent: MORNINGSTAR
493Disallow: /
494
495User-agent: MuckRack
496Disallow: /
497
498User-agent: Netvibes
499Disallow: /
500
501User-agent: news-api.org
502Disallow: /
503
504User-agent: Newslitbot
505Disallow: /
506
507User-agent: NewsNow
508Disallow: /
509
510User-agent: Northern Light
511Disallow: /
512
513User-agent: Opinion-tracker
514Disallow: /
515
516User-agent: Orbis
517Disallow: /
518
519User-agent: Opoint
520Disallow: /
521
522User-agent: Paqlebot
523Disallow: /
524
525User-agent: Press Monitor Europe
526Disallow: /
527
528User-agent: PressEngineBot D
529Disallow: /
530
531User-agent: PriberamBot
532Disallow: /
533
534User-agent: QuoteMedia
535Disallow: /
536
537User-agent: QWAM CONTENT INTELLIGENCE
538Disallow: /
539
540User-agent: RankurBot
541Disallow: /
542
543User-agent: RavenPack
544Disallow: /
545
546User-agent: ReportLinker
547Disallow: /
548
549User-agent: Research & Markets
550Disallow: /
551
552User-agent: S&P Capital IQ
553Disallow: /
554
555User-agent: S&P Global Market Intelligence
556Disallow: /
557
558User-agent: S&P Global Marketplace
559Disallow: /
560
561User-agent: scoopit
562Disallow: /
563
564User-agent: scoopit-crawler
565Disallow: /
566
567User-agent: scpitspi-rs
568Disallow: /
569
570User-agent: semantic-visions.com
571Disallow: /
572
573User-agent: semantic-visions.com crawler
574Disallow: /
575
576User-agent: SemrushBot
577Disallow: /
578
579User-agent: SentiBot
580Disallow: /
581
582User-agent: SentiOne
583Disallow: /
584
585User-agent: Signal Insights
586Disallow: /
587
588User-agent: Sindup
589Disallow: /
590
591User-agent: smartkarma
592Disallow: /
593
594User-agent: Sociallymap
595Disallow: /
596
597User-agent: spotter
598Disallow: /
599
600User-agent: squirrobot
601Disallow: /
602
603User-agent: Statista
604Disallow: /
605
606User-agent: STOCKBOARD
607Disallow: /
608
609User-agent: telpress.it
610Disallow: /
611
612User-agent: Thomson Reuters
613Disallow: /
614
615User-agent: Thomson Reuters WestLaw
616Disallow: /
617
618User-agent: TraderPlanet
619Disallow: /
620
621User-agent: Tradingcharts.com
622Disallow: /
623
624User-agent: TradingView
625Disallow: /
626
627User-agent: Transform Your API
628Disallow: /
629
630User-agent: trendeo
631Disallow: /
632
633User-agent: trendictionbot
634Disallow: /
635
636User-agent: TurnitinBot
637Disallow: /
638
639User-agent: um-FC
640Disallow: /
641
642User-agent: um-IC
643Disallow: /
644
645User-agent: um-LN
646Disallow: /
647
648User-agent: UptimeRobot
649Disallow: /
650
651User-agent: Vable Ltd
652Disallow: /
653
654User-agent: VISIBRAIN
655Disallow: /
656
657User-agent: webzio
658Disallow: /
659
660User-agent: YaK
661Disallow: /
662
663User-agent: YCHARTS
664Disallow: /
665
666User-agent: YouBot
667Disallow: /
668
669User-agent: Zite
670Disallow: /
671
672User-agent: Cotoyogi
673Disallow: /
674
675User-agent: Datenbank Crawler
676Disallow: /
677
678User-agent: Devin
679Disallow: /
680
681User-agent: Factset_spyderbot
682Disallow: /
683
684User-agent: ICC-Crawler
685Disallow: /
686
687User-agent: PetalBot
688Disallow: /
689
690User-agent: SemrushBot-OCOB
691Disallow: /
692
693User-agent: AddSearchBot
694Disallow: /
695
696User-agent: bigsur.ai
697Disallow: /
698
699User-agent: Claude-SearchBot
700Disallow: /
701
702User-agent: Claude-User
703Disallow: /
704
705User-agent: CloudVertexBot
706Disallow: /
707
708User-agent: Gemini-Deep-Research
709Disallow: /
710
711User-agent: GoogleOther
712Disallow: /
713
714User-agent: LinerBot
715Disallow: /
716
717User-agent: netEstate Imprint Crawler
718Disallow: /
719
720User-agent: QualifiedBot
721Disallow: /
722
723User-agent: AI2Bot-DeepResearchEval
724Disallow: /
725
726User-agent: Ai2Bot-Dolma
727Disallow: /
728
729User-agent: Anomura
730Disallow: /
731
732User-agent: ApifyBot
733Disallow: /
734
735User-agent: ApifyWebsiteContentCrawler
736Disallow: /
737
738User-agent: Aranet-SearchBot
739Disallow: /
740
741User-agent: atlassian-bot
742Disallow: /
743
744User-agent: AzureAI-SearchBot
745Disallow: /
746
747User-agent: Bravebot
748Disallow: /
749
750User-agent: Channel3Bot
751Disallow: /
752
753User-agent: ChatGLM-Spider
754Disallow: /
755
756User-agent: Cloudflare-AutoRAG
757Disallow: /
758
759User-agent: Crawl4AI
760Disallow: /
761
762User-agent: DeepSeekBot
763Disallow: /
764
765User-agent: ExaBot
766Disallow: /
767
768User-agent: FirecrawlAgent
769Disallow: /
770
771User-agent: Google-CloudVertexBot
772Disallow: /
773
774User-agent: Google-NotebookLM
775Disallow: /
776
777User-agent: iAskBot
778Disallow: /
779
780User-agent: iaskspider
781Disallow: /
782
783User-agent: imageSpider
784Disallow: /
785
786User-agent: kagi-fetcher
787Disallow: /
788
789User-agent: KlaviyoAIBot
790Disallow: /
791
792User-agent: KunatoCrawler
793Disallow: /
794
795User-agent: laion-huggingface-processor
796Disallow: /
797
798User-agent: LCC
799Disallow: /
800
801User-agent: LinkupBot
802Disallow: /
803
804User-agent: meta-webindexer
805Disallow: /
806
807User-agent: PhindBot
808Disallow: /
809
810User-agent: Poggio-Citations
811Disallow: /
812
813User-agent: SBIntuitionsBot
814Disallow: /
815
816User-agent: Spider
817Disallow: /
818
819User-agent: TavilyBot
820Disallow: /
821
822User-agent: TerraCotta
823Disallow: /
824
825User-agent: VelenPublicWebCrawler
826Disallow: /
827
828User-agent: webzio-extended
829Disallow: /
830
831User-agent: WRTNBot
832Disallow: /
833
834User-agent: ZanistaBot
835Disallow: /
836
837User-agent: AmazonBuyForMe
838Disallow: /
839
840User-agent: Google-Agent
841Disallow: /
842
843User-agent: GoogleAgent-Mariner
844Disallow: /
845
846User-agent: Manus-User
847Disallow: /
848
849User-agent: NovaAct
850Disallow: /
851
852User-agent: Shap-User
853Disallow: /
854
855User-agent: TwinAgent
856Disallow: /
857
858User-agent: ZenphonyBot
859Disallow: /
860
861User-agent: Brightbot
862Disallow: /
863
864User-agent: HenkBot
865Disallow: /
866
867User-agent: ShapBot
868Disallow: /
869
870User-agent: Terra Cotta
871Disallow: /
872
873User-agent: CohereBot
874Disallow: /
875
876User-agent: CragCrawler
877Disallow: /
878
879User-agent: GeistHaus-PageFetcher
880Disallow: /
881
882User-agent: GoogleAgent-URLContext
883Disallow: /
884
885User-agent: Kimi-SearchBot
886Disallow: /
887
888User-agent: Kimi-User
889Disallow: /
890
891User-agent: KimiBot
892Disallow: /
893
894User-agent: MistralAI-Index
895Disallow: /
896
897User-agent: MistralBot
898Disallow: /
899
900User-agent: Querit-SearchBot
901Disallow: /
902
903User-agent: QueritBot
904Disallow: /
905
906User-agent: UseAI
907Disallow: /
908
909User-agent: xAI-SearchBot
910Disallow: /
911
912User-agent: AIWebIndex
913Disallow: /
914
915User-agent: amazon-kendra
916Disallow: /
917
918User-agent: amazon-QBusiness
919Disallow: /
920
921User-agent: bedrockbot
922Disallow: /
923
924User-agent: Crawlspace
925Disallow: /
926
927User-agent: Doubaobot
928Disallow: /
929
930User-agent: ERNIEBot
931Disallow: /
932
933User-agent: KimiCrawler
934Disallow: /
935
936User-agent: micro-crawl
937Disallow: /
938
939User-agent: MoonshotBot
940Disallow: /
941
942User-agent: MoonSpider
943Disallow: /
944
945User-agent: Mozilla-Tabstack
946Disallow: /
947
948User-agent: QwenBot
949Disallow: /
950
951User-agent: SalamandraVLM
952Disallow: /
953
954User-agent: TongyiBot
955Disallow: /
956
957User-agent: YandexAdditional
958Disallow: /
959
960User-agent: YandexAdditionalBot
961Disallow: /
962
963User-agent: YiyanBot
964Disallow: /
965
966User-agent: AgentDataBot
967Disallow: /
968
969User-agent: AGI Agent
970Disallow: /
971
972User-agent: AIWebIndex-Agent
973Disallow: /
974
975User-agent: AndiAgent
976Disallow: /
977
978User-agent: artsdata-crawler
979Disallow: /
980
981User-agent: BeansLLM-CorpusBot
982Disallow: /
983
984User-agent: BixelBot
985Disallow: /
986
987User-agent: CartAI Agentic Commerce
988Disallow: /
989
990User-agent: CledaraBot
991Disallow: /
992
993User-agent: Cloudflare-AI-Search
994Disallow: /
995
996User-agent: Cloudflare-AI-Search-External
997Disallow: /
998
999User-agent: CloudflareBrowserRenderingCrawler
1000Disallow: /
1001
1002User-agent: DesearchBot
1003Disallow: /
1004
1005User-agent: Dola-SearchBot
1006Disallow: /
1007
1008User-agent: Dot
1009Disallow: /
1010
1011User-agent: ElevenlabsBot
1012Disallow: /
1013
1014User-agent: ElevenlabsCrawler
1015Disallow: /
1016
1017User-agent: ExaSearchBot
1018Disallow: /
1019
1020User-agent: ExaUser
1021Disallow: /
1022
1023User-agent: Grok Bot
1024Disallow: /
1025
1026User-agent: grok-agent
1027Disallow: /
1028
1029User-agent: GrokBot
1030Disallow: /
1031
1032User-agent: HarkBot
1033Disallow: /
1034
1035User-agent: Hunyuan
1036Disallow: /
1037
1038User-agent: ImageBot
1039Disallow: /
1040
1041User-agent: Instinct
1042Disallow: /
1043
1044User-agent: Keenable-User
1045Disallow: /
1046
1047User-agent: KeenableBot
1048Disallow: /
1049
1050User-agent: Kimi-Agent
1051Disallow: /
1052
1053User-agent: Muse
1054Disallow: /
1055
1056User-agent: Payhawk Invoice Fetching Agent
1057Disallow: /
1058
1059User-agent: Quartr Crawler
1060Disallow: /
1061
1062User-agent: recraft-image-dataset
1063Disallow: /
1064
1065User-agent: ReflectionBot
1066Disallow: /
1067
1068User-agent: ScryBot
1069Disallow: /
1070
1071User-agent: SofyaBot
1072Disallow: /
1073
1074User-agent: SomethinggBot
1075Disallow: /
1076
1077User-agent: TalarionFrontierCrawler
1078Disallow: /
1079
1080User-agent: TalarionSearchBot
1081Disallow: /
1082
1083User-agent: xAI-Grok
1084Disallow: /
1085
1086User-agent: YepBot
1087Disallow: /
1088
1089User-agent: YiBot
1090Disallow: /
1091
1092User-agent: Claude-Agent
1093Disallow: /
1094
What do the bots actually get?
This page reads robots.txt, which states what lemonde.fr asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on lemonde.fr Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does lemonde.fr block GPTBot?
No. lemonde.fr’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does lemonde.fr block ClaudeBot?
Yes. lemonde.fr’s robots.txt disallows ClaudeBot from the whole site (line 203: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does lemonde.fr block Google-Extended?
Yes. lemonde.fr’s robots.txt disallows Google-Extended from the whole site (line 57: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does lemonde.fr block PerplexityBot?
No. lemonde.fr’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find lemonde.fr?
Yes. lemonde.fr’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.