Who blocks AI · #785 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does lefigaro.fr block AI crawlers?
Yes. lefigaro.fr’s robots.txt blocks 11 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 8 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotAllowedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.lefigaro.fr/robots.txt · 846 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 89 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Blocked | line 71 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Blocked | line 581 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 836 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 152 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 143 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 155 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 140 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 65 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 95 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Blocked | line 125 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Blocked | line 722 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Blocked | line 620 · Disallow: / under User-agent: dotbot |
View www.lefigaro.fr/robots.txt · 846 lines
1# :.
2# ++.
3# .+**:
4# .+****.
5# =******..
6# .********:.
7# -*********-.
8# =***********#FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF#.
9# .-************#FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF#.
10# .++=*************#FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF#.
11# .+****************FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF#.
12# -****************#FFFFFFFF# .#FFFFFFFF#.
13# .+***************#FFFFFFF# .=+++++++++.
14# .+**************#FFFFFF#
15# .-+************FFFFFF#
16# .-**************FFFFFFFFFFFFFFFFF:
17# .-************FFFFFFFFFFFFFFFFF:
18# ..=*********FFFFFFFFFFFFFFFFF:
19# .##******FFFFFFFFFFFFFFFFF:
20# #FFFF##*FFFFFF#*+.
21# #FFFFFFFFFFFFF#***=.
22# #FFFFFFFFFFFFF# .:+*-.
23# .#FFFFFFFFFFFFF#. -*-
24# +FFFFFFFFFFFFFFFFFFFFFFF- :+.
25# +FFFFFFFFFFFFFFFFFFFFFFF- .-=.
26# +FFFFFFFFFFFFFFFFFFFFFFF- .=-.
27# +FFFFFFFFFFFFFFFFFFFFFFF- .+.
28
29# Toute utilisation de nos contenus protégés autre qu'un usage strictement individuel (incluant notamment l'entraînement des grands modèles de langage (LLM),
30# l'entraînement des outils d'intelligence artificielle, la veille web ou le media monitoring) est conditionnée à la conclusion d'une licence d'utilisation
31# avec la SOCIETE DU FIGARO. Nous vous invitons à cette fin à contacter syndication-service@lefigaro.fr.
32#
33# Toute utilisation non-autorisée de nos contenus protégés est constitutive d'une contrefaçon de droit d'auteur et/ou du droit de producteur
34# de base de données et susceptible d'être poursuivie.
35#
36# Il est interdit de crawler notre site web en utilisant un agent d'utilisateur (user agent) volé qui ne correspond pas à votre identité.
37# L'utilisation des robots d'indexation web ou d'autres méthodes automatiques de feuilletage ou de navigation sur ce site Web n'est pas autorisée.
38
39#Généralités
40
41User-agent: *
42Disallow: /async/
43Disallow: /*?noheader*
44Disallow: /*?*&noheader*
45Disallow: /*?sfdebug*
46Disallow: /*?*&sfdebug*
47Disallow: /brouillon/
48Disallow: /proxy/
49Disallow: /synsearch/
50Disallow: /a-savoir-en-france/
51Disallow: /vos-questions
52
53#LLM
54
55User-agent: ChatGPT-User
56Allow: /voyages
57Allow: /culture
58Allow: /style
59Allow: /beaute
60Allow: /joaillerie
61Allow: /mode-homme
62Allow: /horlogerie
63Allow: /bons-plans
64Allow: /elections/resultats
65Disallow: /
66
67User-agent: Claude-SearchBot
68Allow: /voyages
69Allow: /bons-plans
70Allow: /elections/resultats
71Disallow: /
72
73User-agent: Claude-Web
74Allow: /voyages
75Allow: /bons-plans
76Allow: /elections/resultats
77Disallow: /
78
79User-agent: OAI-SearchBot
80Allow: /voyages
81Allow: /culture
82Allow: /style
83Allow: /beaute
84Allow: /joaillerie
85Allow: /mode-homme
86Allow: /horlogerie
87Allow: /bons-plans
88Allow: /elections/resultats
89Disallow: /
90
91User-agent: Claude-User
92Allow: /voyages
93Allow: /bons-plans
94Allow: /elections/resultats
95Disallow: /
96
97User-agent: ChatGPT Agent
98Allow: /voyages
99Allow: /bons-plans
100Allow: /elections/resultats
101Disallow: /
102
103User-agent: GoogleAgent-Mariner
104Allow: /voyages
105Allow: /bons-plans
106Allow: /elections/resultats
107Disallow: /
108
109User-agent: Google-NotebookLM
110Allow: /voyages
111Allow: /bons-plans
112Allow: /elections/resultats
113Disallow: /
114
115User-agent: Google-CloudVertexBot
116Allow: /voyages
117Allow: /bons-plans
118Allow: /elections/resultats
119Disallow: /
120
121User-agent: MistralAI-User
122Allow: /voyages
123Allow: /bons-plans
124Allow: /elections/resultats
125Disallow: /
126
127User-agent: Gemini-Deep-Research
128Allow: /voyages
129Allow: /bons-plans
130Allow: /elections/resultats
131Disallow: /
132
133User-agent: CloudVertexBot
134Disallow: /
135
136User-agent: anthropic-ai
137Disallow: /
138
139User-agent: CCBot
140Disallow: /
141
142User-agent: ClaudeBot
143Disallow: /
144
145User-agent: cohere-ai
146Disallow: /
147
148User-agent: CohereBot
149Disallow: /
150
151User-agent: GPTBot
152Disallow: /
153
154User-agent: Google-Extended
155Disallow: /
156
157User-agent: Kimi-SearchBot
158Disallow: /
159
160User-agent: Kimi-User
161Disallow: /
162
163User-agent: KimiBot
164Disallow: /
165
166#Crawler
167
168User-agent: Alexibot
169Disallow: /
170
171User-agent: AlvinetSpider
172Disallow: /
173
174User-agent: AmiSoftware
175Disallow: /
176
177User-agent: Antenne Hatena
178Disallow: /
179
180User-agent: ApifyBot
181Disallow: /
182
183User-agent: ApifyWebsiteContentCrawler
184Disallow: /
185
186User-agent: ApocalXExplorerBot
187Disallow: /
188
189User-agent: Argus
190Disallow: /
191
192User-agent: Ask n read
193Disallow: /
194
195User-agent: asknread.com
196Disallow: /
197
198User-agent: asterias
199Disallow: /
200
201User-agent: BlowFish/1.0
202Disallow: /
203
204User-agent: BotALot
205Disallow: /
206
207User-agent: Brightbot
208Disallow: /
209
210User-agent: BuiltBotTough
211Disallow: /
212
213User-agent: Bullseye/1.0
214Disallow: /
215
216User-agent: BunnySlippers
217Disallow: /
218
219User-agent: Cegbfeieh
220Disallow: /
221
222User-agent: CheeseBot
223Disallow: /
224
225User-agent: ConveraCrawler
226Disallow: /
227
228User-agent: cosmos
229Disallow: /
230
231User-agent: CragCrawler
232Disallow: /
233
234User-agent: Crescent
235Disallow: /
236
237User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
238Disallow: /
239
240User-agent: cydralspider
241Disallow: /
242
243User-agent: Diffbot
244Disallow: /
245
246User-agent: Diffbot-User
247Disallow: /
248
249User-agent: DISCo Pump 3.1
250Disallow: /
251
252User-agent: DittoSpyder
253Disallow: /
254
255User-agent: EroCrawler
256Disallow: /
257
258User-agent: eureka
259Disallow: /
260
261User-agent: ExaBot
262Disallow: /
263
264User-agent: ExaSearchBot
265Disallow: /
266
267User-agent: Explore
268Disallow: /
269
270User-agent: Fetch
271Disallow: /
272
273User-agent: FirecrawlAgent
274Disallow: /
275
276User-agent: Flamingo_SearchEngine
277Disallow: /
278
279User-agent: Foobot
280Disallow: /
281
282User-agent: gammaSpider
283Disallow: /
284
285User-agent: GeistHaus-PageFetcher
286Disallow: /
287
288User-agent: grub-client
289Disallow: /
290
291User-agent: hloader
292Disallow: /
293
294User-agent: httplib
295Disallow: /
296
297User-agent: humanlinks
298Disallow: /
299
300User-agent: ia_archiver
301Allow: /$
302Disallow: /
303
304User-agent: ia_archiver-web.archive.org
305Allow: /$
306Disallow: /
307
308User-agent: indexer
309Disallow: /
310
311User-agent: InfoNaviRobot
312Disallow: /
313
314User-agent: infoseek
315Disallow: /
316
317User-agent: JennyBot
318Disallow: /
319
320User-agent: Jetbot
321Disallow: /
322
323User-agent: JikeSpider
324Disallow: /
325
326User-agent: k2spider
327Disallow: /
328
329User-agent: Kenjin Spider
330Disallow: /
331
332User-agent: larbin
333Disallow: /
334
335User-agent: LexiBot
336Disallow: /
337
338User-agent: libWeb/clsHTTP
339Disallow: /
340
341User-agent: libwww
342Disallow: /
343
344User-agent: linko
345Disallow: /
346
347User-agent: LinkScan/8.1a Unix
348Disallow: /
349
350User-agent: LinkWalker
351Disallow: /
352
353User-agent: lwp-trivial
354Disallow: /
355
356User-agent: lwp-trivial/1.34
357Disallow: /
358
359User-agent: Mata Hari
360Disallow: /
361
362User-agent: Mister PiX
363Disallow: /
364
365User-agent: MLBot
366Disallow: /
367
368User-agent: moget
369Disallow: /
370
371User-agent: moget/2.1
372Disallow: /
373
374User-agent: Naverbot
375Disallow: /
376
377User-agent: NetAttache
378Disallow: /
379
380User-agent: Newzbin
381Disallow: /
382
383User-agent: NICErsPRO
384Disallow: /
385
386User-agent: NPBot
387Disallow: /
388
389User-agent: ObjectsSearch
390Disallow: /
391
392User-agent: omgili
393Disallow: /
394
395User-agent: omgilibot
396Disallow: /
397
398User-agent: Openfind
399Disallow: /
400
401User-agent: OpenindexSpider
402Disallow: /
403
404User-agent: Pimptrain
405Disallow: /
406
407User-agent: ProPowerBot/2.14
408Disallow: /
409
410User-agent: ProWebWalker
411Disallow: /
412
413User-agent: psbot
414Disallow: /
415
416User-agent: QuepasaCreep
417Disallow: /
418
419User-agent: Querit-SearchBot
420Disallow: /
421
422User-agent: QueritBot
423Disallow: /
424
425User-agent: QueryN Metasearch
426Disallow: /
427
428User-agent: Raven
429Disallow: /
430
431User-agent: RepoMonkey
432Disallow: /
433
434User-agent: RMA
435Disallow: /
436
437User-agent: Scrapy
438Disallow: /
439
440User-agent: ShapBot
441Disallow: /
442
443User-agent: SiteBot
444Disallow: /
445
446User-agent: Sogou web spider
447Disallow: /
448
449User-agent: sosospider
450Disallow: /
451
452User-agent: SpankBot
453Disallow: /
454
455User-agent: Speedy
456Disallow: /
457
458User-agent: spotter
459Disallow: /
460
461User-agent: suggybot
462Disallow: /
463
464User-agent: SuperBot
465Disallow: /
466
467User-agent: SuperBot/2.6
468Disallow: /
469
470User-agent: suzuran
471Disallow: /
472
473User-agent: Szukacz/1.4
474Disallow: /
475
476User-agent: Telesoft
477Disallow: /
478
479User-agent: The Intraformant
480Disallow: /
481
482User-agent: TheNomad
483Disallow: /
484
485User-agent: TightTwatBot
486Disallow: /
487
488User-agent: Titan
489Disallow: /
490
491User-agent: toCrawl/UrlDispatcher
492Disallow: /
493
494User-agent: TosCrawler
495Disallow: /
496
497User-agent: True_Robot
498Disallow: /
499
500User-agent: True_Robot/1.0
501Disallow: /
502
503User-agent: turingos
504Disallow: /
505
506User-agent: URLy Warning
507Disallow: /
508
509User-agent: VCI
510Disallow: /
511
512User-agent: vsw
513Disallow: /
514
515User-agent: wapspider
516Disallow: /
517
518User-agent: WebBandit
519Disallow: /
520
521User-agent: WebBandit/3.50
522Disallow: /
523
524User-agent: WebEnhancer
525Disallow: /
526
527User-agent: WebmasterWorldForumBot
528Disallow: /
529
530User-agent: Webster Pro
531Disallow: /
532
533User-agent: WebZinger
534Disallow: /
535
536User-agent: Webzio-Extended
537Disallow: /
538
539User-agent: WWW-Collector-E
540Disallow: /
541
542User-agent: yacy
543Disallow: /
544
545User-agent: yandex
546Disallow: /
547
548User-agent: YRSPider
549Disallow: /
550
551User-agent: Zealbot
552Disallow: /
553
554User-agent: Zeus
555Disallow: /
556
557User-agent: Zite
558Disallow: /
559
560User-agent: Zookabot
561Disallow: /
562
563User-agent: ZyBORG
564Disallow: /
565
566#Outil
567
568User-agent: 5emeRue
569Disallow: /
570
571User-agent: 5erue
572Disallow: /
573
574User-agent: adequat
575Disallow: /
576
577User-agent: adequat-systems
578Disallow: /
579
580User-agent: Amazonbot
581Disallow: /
582
583User-agent: Augure
584Disallow: /
585
586User-agent: auramundi
587Disallow: /
588
589User-agent: BizInformation
590Disallow: /
591
592User-agent: Bloomberg
593Disallow: /
594
595User-agent: CherryPicker
596Disallow: /
597
598User-agent: CherryPickerElite/1.0
599Disallow: /
600
601User-agent: CherryPickerSE/1.0
602Disallow: /
603
604User-agent: Cision
605Disallow: /
606
607User-agent: coexel
608Disallow: /
609
610User-agent: CopyRightCheck
611Disallow: /
612
613User-agent: Corporama
614Disallow: /
615
616User-agent: Digimind
617Disallow: /
618
619User-agent: dotbot
620Disallow: /
621
622User-agent: Download Ninja
623Disallow: /
624
625User-agent: downloadexpress
626Disallow: /
627
628User-agent: ellisphere
629Disallow: /
630
631User-agent: EmailCollector
632Disallow: /
633
634User-agent: EmailSiphon
635Disallow: /
636
637User-agent: EmailWolf
638Disallow: /
639
640User-agent: Europresse
641Disallow: /
642
643User-agent: ExtractorPro
644Disallow: /
645
646User-agent: Fasterfox
647Disallow: /
648
649User-agent: HTTrack
650Disallow: /
651
652User-agent: HTTrack 3.0
653Disallow: /
654
655User-agent: Igentia
656Disallow: /
657
658User-agent: Kantar
659Disallow: /
660
661User-agent: kbcrawl
662Disallow: /
663
664User-agent: Knowings
665Disallow: /
666
667User-agent: leadbox
668Disallow: /
669
670User-agent: LinkextractorPro
671Disallow: /
672
673User-agent: linkfluence
674Disallow: /
675
676User-agent: manageo
677Disallow: /
678
679User-agent: mediacompil
680Disallow: /
681
682User-agent: mention
683Disallow: /
684
685User-agent: Moreover
686Disallow: /
687
688User-agent: mytwip
689Disallow: /
690
691User-agent: NetAnts
692Disallow: /
693
694User-agent: NetMechanic
695Disallow: /
696
697User-agent: newscan-online
698Disallow: /
699
700User-agent: NewsNow
701Disallow: /
702
703User-agent: Offline Explorer
704Disallow: /
705
706User-agent: opinion-tracker
707Disallow: /
708
709User-agent: Qwam content intelligence
710Disallow: /
711
712User-agent: readability.com
713Disallow: /
714
715User-agent: scoop.it
716Disallow: /
717
718User-agent: score3
719Disallow: /
720
721User-agent: SemrushBot
722Disallow: /
723
724User-agent: SightupBot
725Disallow: /
726
727User-agent: Sindup
728Disallow: /
729
730User-agent: SirdataBot
731Disallow: /
732
733User-agent: SiteSnagger
734Disallow: /
735
736User-agent: SiteSucker
737Disallow: /
738
739User-agent: Synthesio
740Disallow: /
741
742User-agent: Talkwalker
743Disallow: /
744
745User-agent: TavilyBot
746Disallow: /
747
748User-agent: Teleport
749Disallow: /
750
751User-agent: TeleportPro
752Disallow: /
753
754User-agent: trendeo
755Disallow: /
756
757User-agent: trendybuzz
758Disallow: /
759
760User-agent: TurnitinBot
761Disallow: /
762
763User-agent: up2news
764Disallow: /
765
766User-agent: UrlPouls
767Disallow: /
768
769User-agent: UseAI
770Disallow: /
771
772User-agent: vecteurplus
773Disallow: /
774
775User-agent: Verif
776Disallow: /
777
778User-agent: verticalsearch
779Disallow: /
780
781User-agent: Web Image Collector
782Disallow: /
783
784User-agent: WebAuto
785Disallow: /
786
787User-agent: WebCopier
788Disallow: /
789
790User-agent: webcopy
791Disallow: /
792
793User-agent: Webedia
794Disallow: /
795
796User-agent: webmirror
797Disallow: /
798
799User-agent: WebReaper
800Disallow: /
801
802User-agent: WebSauger
803Disallow: /
804
805User-agent: website extractor
806Disallow: /
807
808User-agent: Website Quester
809Disallow: /
810
811User-agent: WebStripper
812Disallow: /
813
814User-agent: WebStripper/2.02
815Disallow: /
816
817User-agent: WebZIP
818Disallow: /
819
820User-agent: Wget
821Disallow: /
822
823User-agent: WikioFeedBot
824Disallow: /
825
826User-agent: winello
827Disallow: /
828
829User-agent: WinHTTrack
830Disallow: /
831
832User-agent: Xenu Link Sleuth/1.3.8
833Disallow: /
834
835User-agent: YouBot
836Disallow: /
837
838User-agent: Youmag
839Disallow: /
840
841Sitemap: https://sitemaps.lefigaro.fr/lefigaro.fr/articles.xml
842Sitemap: https://sitemaps.lefigaro.fr/lefigaro.fr/sections.xml
843Sitemap: https://sitemaps.lefigaro.fr/lefigaro.fr/topics.xml
844Sitemap: https://www.lefigaro.fr/sitemap_news.xml
845Sitemap: https://sitemaps.lefigaro.fr/lefigaro.fr/elections/resultats/sitemap.xml
846Sitemap: https://static.lefigaro.fr/f1/lefigaro/sitemaps/robots-service.txt
What do the bots actually get?
This page reads robots.txt, which states what lefigaro.fr asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on lefigaro.fr Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does lefigaro.fr block GPTBot?
Yes. lefigaro.fr’s robots.txt disallows GPTBot from the whole site (line 152: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does lefigaro.fr block ClaudeBot?
Yes. lefigaro.fr’s robots.txt disallows ClaudeBot from the whole site (line 143: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does lefigaro.fr block Google-Extended?
Yes. lefigaro.fr’s robots.txt disallows Google-Extended from the whole site (line 155: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does lefigaro.fr block PerplexityBot?
No. lefigaro.fr’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find lefigaro.fr?
Not through its own crawler. lefigaro.fr’s robots.txt disallows OAI-SearchBot from the whole site (line 89: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.