Who blocks AI · #780 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does france24.com block AI crawlers?
Yes. france24.com’s robots.txt blocks 13 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 10 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.france24.com/robots.txt · 1,024 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 21 · Disallow: /*/_ws/urgent | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Partly blocked | line 21 · Disallow: /*/_ws/urgent |
PetalBot | Huawei | Allowed | line 990 · Allow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | line 1018 · Allow: / under User-agent: YandexBot |
Baiduspider | Baidu | Allowed | line 974 · Allow: / under User-agent: Baiduspider |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 134 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 146 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 59 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 83 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Blocked | line 167 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 95 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 68 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 92 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 270 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 53 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 62 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 140 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 125 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 189 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Blocked | line 669 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Blocked | line 330 · Disallow: / under User-agent: DotBot |
View www.france24.com/robots.txt · 1,024 lines
1# France Medias Monde [2026-02-26] - francemediasmonde.com
2## FRANCE 24 - france24.com
3
4### Sitemaps
5Sitemap: https://www.france24.com/sitemaps/fr/index.xml
6Sitemap: https://www.france24.com/sitemaps/en/index.xml
7Sitemap: https://www.france24.com/sitemaps/es/index.xml
8Sitemap: https://www.france24.com/sitemaps/ar/index.xml
9
10### News Sitemaps
11Sitemap: https://www.france24.com/sitemaps/fr/news.xml
12Sitemap: https://www.france24.com/sitemaps/en/news.xml
13Sitemap: https://www.france24.com/sitemaps/es/news.xml
14Sitemap: https://www.france24.com/sitemaps/ar/news.xml
15
16### General rules
17User-agent: *
18Disallow:
19
20User-agent: Googlebot
21Disallow: /*/_ws/urgent
22
23### AI Bots blocking
24### >> Contact us at syndication[@]francemm.com for partnership opportunities
25User-agent: AI2Bot
26Disallow: /
27
28User-agent: Ai2Bot-Dolma
29Disallow: /
30
31User-agent: AlibabaBot
32Disallow: /
33
34User-agent: AliyunBot
35Disallow: /
36
37User-agent: anthropic-ai
38Disallow: /
39
40User-agent: Baidu-YunGuanCe
41Disallow: /
42
43User-agent: Big Sur AI
44Disallow: /
45
46User-agent: ByteDance-AI
47Disallow: /
48
49User-agent: ChatGPT Operator
50Disallow: /
51
52User-agent: ChatGPT-User
53Disallow: /
54
55User-agent: Claude-Code
56Disallow: /
57
58User-agent: Claude-SearchBot
59Disallow: /
60
61User-agent: Claude-User
62Disallow: /
63
64User-agent: Claude-Web
65Disallow: /
66
67User-agent: ClaudeBot
68Disallow: /
69
70User-agent: cohere-ai
71Disallow: /
72
73User-agent: cohere-training-data-crawler
74Disallow: /
75
76User-agent: DeepSeekBot
77Disallow: /
78
79User-agent: DigitalOceanGenAICrawler
80Disallow: /
81
82User-agent: DuckAssistBot
83Disallow: /
84
85User-agent: ERNIEBot
86Disallow: /
87
88User-agent: ERNIE-Bot
89Disallow: /
90
91User-agent: Google-Extended
92Disallow: /
93
94User-agent: GPTBot
95Disallow: /
96
97User-agent: Grok
98Disallow: /
99
100User-agent: img2dataset
101Disallow: /
102
103User-agent: Kimibot
104Disallow: /
105
106User-agent: KimiCrawler
107Disallow: /
108
109User-agent: LINER Bot
110Disallow: /
111
112User-agent: Meta Llama
113Disallow: /
114
115User-agent: MetaAIBot
116Disallow: /
117
118User-agent: Moonshot-AI
119Disallow: /
120
121User-agent: MoonshotBot
122Disallow: /
123
124User-agent: MistralAI-User
125Disallow: /
126
127User-agent: MistralAI-User/1.0
128Disallow: /
129
130User-agent: MyCentralAIScraperBot
131Disallow: /
132
133User-agent: OAI-SearchBot
134Disallow: /
135
136User-agent: PanguBot
137Disallow: /
138
139User-agent: Perplexity-User
140Disallow: /
141
142User-agent: Perplexity-User/1.0
143Disallow: /
144
145User-agent: PerplexityBot
146Disallow: /
147
148User-agent: quillbot.com
149Disallow: /
150
151User-agent: QwenBot
152Disallow: /
153
154User-agent: SBIntuitionsBot
155Disallow: /
156
157User-agent: TongyiBot
158Disallow: /
159
160User-agent: Tongyi-Crawler
161Disallow: /
162
163User-agent: YiyanBot
164Disallow: /
165
166User-agent: YouBot
167Disallow: /
168
169### General Bots blocking
170User-agent: 008
171Disallow: /
172
173User-agent: 5emeRue
174Disallow: /
175
176User-agent: 5erue
177Disallow: /
178
179User-agent: adequat
180Disallow: /
181
182User-agent: adequat-systems
183Disallow: /
184
185User-agent: ADmantX bot
186Disallow: /
187
188User-agent: AhrefsBot
189Disallow: /
190
191User-agent: Alexibot
192Disallow: /
193
194User-agent: AlvinetSpider
195Disallow: /
196
197User-agent: AmiSoftware
198Disallow: /
199
200User-agent: Antenne Hatena
201Disallow: /
202
203User-agent: ApocalXExplorerBot
204Disallow: /
205
206User-agent: Argus
207Disallow: /
208
209User-agent: Ask n read
210Disallow: /
211
212User-agent: asknread.com
213Disallow: /
214
215User-agent: asterias
216Disallow: /
217
218User-agent: Augure
219Disallow: /
220
221User-agent: augure
222Disallow: /
223
224User-agent: auramundi
225Disallow: /
226
227User-agent: Babya Discoverer
228Disallow: /
229
230User-agent: BackDoorBot/1.0
231Disallow: /
232
233User-agent: BacklinkCrawler
234Disallow: /
235
236User-agent: bitlybot
237Disallow: /
238
239User-agent: BizInformation
240Disallow: /
241
242User-agent: Black Hole
243Disallow: /
244
245User-agent: Blekkobot
246Disallow: /
247
248User-agent: BLEXBot
249Disallow: /
250
251User-agent: Bloodhound
252Disallow: /
253
254User-agent: BlowFish/1.0
255Disallow: /
256
257User-agent: BotALot
258Disallow: /
259
260User-agent: BuiltBotTough
261Disallow: /
262
263User-agent: Bullseye/1.0
264Disallow: /
265
266User-agent: BunnySlippers
267Disallow: /
268
269User-agent: CCBot
270Disallow: /
271
272User-agent: Cegbfeieh
273Disallow: /
274
275User-agent: CheeseBot
276Disallow: /
277
278User-agent: CherryPicker
279Disallow: /
280
281User-agent: CherryPickerElite/1.0
282Disallow: /
283
284User-agent: CherryPickerSE/1.0
285Disallow: /
286
287User-agent: Cision
288Disallow: /
289
290User-agent: Cliqzbot
291Disallow: /
292
293User-agent: coexel
294Disallow: /
295
296User-agent: ConveraCrawler
297Disallow: /
298
299User-agent: CopyRightCheck
300Disallow: /
301
302User-agent: Corporama
303Disallow: /
304
305User-agent: cosmos
306Disallow: /
307
308User-agent: Crescent
309Disallow: /
310
311User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
312Disallow: /
313
314User-agent: cydralspider
315Disallow: /
316
317User-agent: Digimind
318Disallow: /
319
320User-agent: DISCo Pump 3.1
321Disallow: /
322
323User-agent: discobot
324Disallow: /
325
326User-agent: DittoSpyder
327Disallow: /
328
329User-agent: DotBot
330Disallow: /
331
332User-agent: dotbot
333Disallow: /
334
335User-agent: Dotmic DotBot
336Disallow: /
337
338User-agent: Download Ninja
339Disallow: /
340
341User-agent: downloadexpress
342Disallow: /
343
344User-agent: EDD
345Disallow: /
346
347User-agent: ellisphere
348Disallow: /
349
350User-agent: EmailCollector
351Disallow: /
352
353User-agent: EmailSiphon
354Disallow: /
355
356User-agent: EmailWolf
357Disallow: /
358
359User-agent: EroCrawler
360Disallow: /
361
362User-agent: EtaoSpider
363Disallow: /
364
365User-agent: eureka
366Disallow: /
367
368User-agent: Europresse
369Disallow: /
370
371User-agent: Exabot
372Disallow: /
373
374User-agent: Explore
375Disallow: /
376
377User-agent: ExtractorPro
378Disallow: /
379
380User-agent: Ezooms Robot
381Disallow: /
382
383User-agent: Fasterfox
384Disallow: /
385
386User-agent: Fetch
387Disallow: /
388
389User-agent: FirstRain
390Disallow: /
391
392User-agent: Flamingo_SearchEngine
393Disallow: /
394
395User-agent: Foobot
396Disallow: /
397
398User-agent: gammaSpider
399Disallow: /
400
401User-agent: GnowitNewsbot
402Disallow: /
403
404User-agent: grub-client
405Disallow: /
406
407User-agent: Harvest/1.5
408Disallow: /
409
410User-agent: hloader
411Disallow: /
412
413User-agent: houzzbot
414Disallow: /
415
416User-agent: httplib
417Disallow: /
418
419User-agent: HTTrack
420Disallow: /
421
422User-agent: HTTrack 3.0
423Disallow: /
424
425User-agent: humanlinks
426Disallow: /
427
428User-agent: Igentia
429Disallow: /
430
431User-agent: indexer
432Disallow: /
433
434User-agent: InfoNaviRobot
435Disallow: /
436
437User-agent: infoseek
438Disallow: /
439
440User-agent: JennyBot
441Disallow: /
442
443User-agent: Jetbot
444Disallow: /
445
446User-agent: JikeSpider
447Disallow: /
448
449User-agent: k2spider
450Disallow: /
451
452User-agent: kbcrawl
453Disallow: /
454
455User-agent: Kenjin Spider
456Disallow: /
457
458User-agent: Knowings
459Disallow: /
460
461User-agent: LamarkBot
462Disallow: /
463
464User-agent: larbin
465Disallow: /
466
467User-agent: leadbox
468Disallow: /
469
470User-agent: LexiBot
471Disallow: /
472
473User-agent: libWeb/clsHTTP
474Disallow: /
475
476User-agent: libwww
477Disallow: /
478
479User-agent: LinkextractorPro
480Disallow: /
481
482User-agent: linkfluence
483Disallow: /
484
485User-agent: linko
486Disallow: /
487
488User-agent: LinkScan/8.1a Unix
489Disallow: /
490
491User-agent: LinkWalker
492Disallow: /
493
494User-agent: lwp-trivial
495Disallow: /
496
497User-agent: lwp-trivial/1.34
498Disallow: /
499
500User-agent: manageo
501Disallow: /
502
503User-agent: Mata Hari
504Disallow: /
505
506User-agent: MaxPointCrawler
507Disallow: /
508
509User-agent: mediacompil
510Disallow: /
511
512User-agent: Mediatoolkitbot
513Disallow: /
514
515User-agent: Meltawer
516Disallow: /
517
518User-agent: Meltwater
519Disallow: /
520
521User-agent: mention
522Disallow: /
523
524User-agent: Microsoft.URL.Control
525Disallow: /
526
527User-agent: MIIxpc
528Disallow: /
529
530User-agent: MIIxpc/4.2
531Disallow: /
532
533User-agent: Mister PiX
534Disallow: /
535
536User-agent: MJ12bot
537Disallow: /
538
539User-agent: MLBot
540Disallow: /
541
542User-agent: moget
543Disallow: /
544
545User-agent: moget/2.1
546Disallow: /
547
548User-agent: Moreover
549Disallow: /
550
551User-agent: MS Search 4.0 Robot
552Disallow: /
553
554User-agent: MS Search 5.0 Robot
555Disallow: /
556
557User-agent: MSIECrawler
558Disallow: /
559
560User-agent: mytwip
561Disallow: /
562
563User-agent: Naverbot
564Disallow: /
565
566User-agent: NetAnts
567Disallow: /
568
569User-agent: NetAttache
570Disallow: /
571
572User-agent: netEstate NE Crawler
573Disallow: /
574
575User-agent: NetMechanic
576Disallow: /
577
578User-agent: newscan-online
579Disallow: /
580
581User-agent: Newzbin
582Disallow: /
583
584User-agent: NICErsPRO
585Disallow: /
586
587User-agent: NPBot
588Disallow: /
589
590User-agent: Nutch
591Disallow: /
592
593User-agent: ObjectsSearch
594Disallow: /
595
596User-agent: Offline Explorer
597Disallow: /
598
599User-agent: omgili
600Disallow: /
601
602User-agent: omgilibot
603Disallow: /
604
605User-agent: Openfind
606Disallow: /
607
608User-agent: OpenindexSpider
609Disallow: /
610
611User-agent: opinion-tracker
612Disallow: /
613
614User-agent: Perl LWP
615Disallow: /
616
617User-agent: Pimptrain
618Disallow: /
619
620User-agent: ProPowerBot/2.14
621Disallow: /
622
623User-agent: ProWebWalker
624Disallow: /
625
626User-agent: proxem
627Disallow: /
628
629User-agent: psbot
630Disallow: /
631
632User-agent: QuepasaCreep
633Disallow: /
634
635User-agent: QueryN Metasearch
636Disallow: /
637
638User-agent: Qwam content intelligence
639Disallow: /
640
641User-agent: Raven
642Disallow: /
643
644User-agent: readability.com
645Disallow: /
646
647User-agent: RepoMonkey
648Disallow: /
649
650User-agent: Riddler
651Disallow: /
652
653User-agent: RMA
654Disallow: /
655
656User-agent: rogerbot
657Disallow: /
658
659User-agent: scoop.it
660Disallow: /
661
662User-agent: score3
663Disallow: /
664
665User-agent: SearchmetricsBot
666Disallow: /
667
668User-agent: SemrushBot
669Disallow: /
670
671User-agent: sentibot
672Disallow: /
673
674User-agent: SEOkicks
675Disallow: /
676
677User-agent: SEOkicks-Robot
678Disallow: /
679
680User-agent: SeznamBot
681Disallow: /
682
683User-agent: SightupBot
684Disallow: /
685
686User-agent: Sindup
687Disallow: /
688
689User-agent: SISTRIX Crawler
690Disallow: /
691
692User-agent: SiteBot
693Disallow: /
694
695User-agent: sitecheck.internetseer.com
696Disallow: /
697
698User-agent: SiteSnagger
699Disallow: /
700
701User-agent: SiteSucker
702Disallow: /
703
704User-agent: Sogou web spider
705Disallow: /
706
707User-agent: sosospider
708Disallow: /
709
710User-agent: SpankBot
711Disallow: /
712
713User-agent: spanner
714Disallow: /
715
716User-agent: Speedy
717Disallow: /
718
719User-agent: spotter
720Disallow: /
721
722User-agent: SputnikBot
723Disallow: /
724
725User-agent: suggybot
726Disallow: /
727
728User-agent: SuperBot
729Disallow: /
730
731User-agent: SuperBot/2.6
732Disallow: /
733
734User-agent: SurveyBot
735Disallow: /
736
737User-agent: suzuran
738Disallow: /
739
740User-agent: Synthesio
741Disallow: /
742
743User-agent: Szukacz/1.4
744Disallow: /
745
746User-agent: Talkwater
747Disallow: /
748
749User-agent: Teleport
750Disallow: /
751
752User-agent: TeleportPro
753Disallow: /
754
755User-agent: Telesoft
756Disallow: /
757
758User-agent: The Intraformant
759Disallow: /
760
761User-agent: TheNomad
762Disallow: /
763
764User-agent: TightTwatBot
765Disallow: /
766
767User-agent: Titan
768Disallow: /
769
770User-agent: toCrawl/UrlDispatcher
771Disallow: /
772
773User-agent: TosCrawler
774Disallow: /
775
776User-agent: trendeo
777Disallow: /
778
779User-agent: trendictionbot
780Disallow: /
781
782User-agent: trendybuzz
783Disallow: /
784
785User-agent: True_Robot
786Disallow: /
787
788User-agent: True_Robot/1.0
789Disallow: /
790
791User-agent: turingos
792Disallow: /
793
794User-agent: up2news
795Disallow: /
796
797User-agent: Updownerbot
798Disallow: /
799
800User-agent: UptimeRobot/2.0
801Disallow: /
802
803User-agent: UrlPouls
804Disallow: /
805
806User-agent: URLy Warning
807Disallow: /
808
809User-agent: VCI
810Disallow: /
811
812User-agent: vecteurplus
813Disallow: /
814
815User-agent: Verif
816Disallow: /
817
818User-agent: verticalsearch
819Disallow: /
820
821User-agent: vsw
822Disallow: /
823
824User-agent: wapspider
825Disallow: /
826
827User-agent: Web Image Collector
828Disallow: /
829
830User-agent: WebAuto
831Disallow: /
832
833User-agent: WebBandit
834Disallow: /
835
836User-agent: WebBandit/3.50
837Disallow: /
838
839User-agent: WebCopier
840Disallow: /
841
842User-agent: webcopy
843Disallow: /
844
845User-agent: WebEnhancer
846Disallow: /
847
848User-agent: WebmasterWorldForumBot
849Disallow: /
850
851User-agent: webmirror
852Disallow: /
853
854User-agent: WebReaper
855Disallow: /
856
857User-agent: WebSauger
858Disallow: /
859
860User-agent: website extractor
861Disallow: /
862
863User-agent: Website Quester
864Disallow: /
865
866User-agent: Webster Pro
867Disallow: /
868
869User-agent: WebStripper
870Disallow: /
871
872User-agent: WebStripper/2.02
873Disallow: /
874
875User-agent: WebZinger
876Disallow: /
877
878User-agent: WebZIP
879Disallow: /
880
881User-agent: wget
882Disallow: /
883
884User-agent: Wget
885Disallow: /
886
887User-agent: WikioFeedBot
888Disallow: /
889
890User-agent: winello
891Disallow: /
892
893User-agent: WinHTTrack
894Disallow: /
895
896User-agent: WiseGuys Robot
897Disallow: /
898
899User-agent: WWW-Collector-E
900Disallow: /
901
902User-agent: Xenu Link Sleuth/1.3.8
903Disallow: /
904
905User-agent: XoviBot
906Disallow: /
907
908User-agent: yacy
909Disallow: /
910
911User-agent: YisouSpider
912Disallow: /
913
914User-agent: Youmag
915Disallow: /
916
917User-agent: YRSPider
918Disallow: /
919
920User-agent: Zealbot
921Disallow: /
922
923User-agent: Zeus
924Disallow: /
925
926User-agent: Zite
927Disallow: /
928
929User-agent: Zookabot
930Disallow: /
931
932User-agent: ZyBORG
933Disallow: /
934
935### Partners
936User-agent: CriteoBot/0.1
937Allow: /
938Crawl-delay: 5
939
940User-agent: grapeshot
941Allow: /
942Crawl-delay: 5
943
944User-agent: INA dlweb
945Allow: /
946Crawl-delay: 5
947
948User-agent: inkl
949Allow: /
950Crawl-delay: 5
951
952User-agent: Kantar
953Allow: /
954Crawl-delay: 5
955
956User-agent: outbrain
957Allow: /
958Crawl-delay: 5
959
960User-agent: proximic
961Allow: /
962Crawl-delay: 10
963
964User-agent: weborama-fetcher
965Allow: /
966Crawl-delay: 5
967
968### Too fast bots
969User-agent: archive.org_bot
970Allow: /
971Crawl-delay: 10
972
973User-agent: Baiduspider
974Allow: /
975Crawl-delay: 10
976
977User-agent: coccocbot-web
978Allow: /
979Crawl-delay: 10
980
981User-agent: Factiva
982Allow: /
983Crawl-delay: 10
984
985User-agent: NewsNow
986Allow: /
987Crawl-delay: 10
988
989User-agent: PetalBot
990Allow: /
991Crawl-delay: 10
992
993User-agent: proximic
994Allow: /
995Crawl-delay: 10
996
997User-agent: Soso Spider
998Allow: /
999Crawl-delay: 10
1000
1001User-agent: Sogou Spider
1002Allow: /
1003Crawl-delay: 10
1004
1005User-agent: TurnitinBot
1006Allow: /
1007Crawl-delay: 10
1008
1009User-agent: Turnitin Robot
1010Allow: /
1011Crawl-delay: 10
1012
1013User-agent: Yandex
1014Allow: /
1015Crawl-delay: 10
1016
1017User-agent: YandexBot
1018Allow: /
1019Crawl-delay: 10
1020
1021User-agent: Yeti
1022Allow: /
1023Crawl-delay: 10
1024
What do the bots actually get?
This page reads robots.txt, which states what france24.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on france24.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does france24.com block GPTBot?
Yes. france24.com’s robots.txt disallows GPTBot from the whole site (line 95: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does france24.com block ClaudeBot?
Yes. france24.com’s robots.txt disallows ClaudeBot from the whole site (line 68: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does france24.com block Google-Extended?
Yes. france24.com’s robots.txt disallows Google-Extended from the whole site (line 92: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does france24.com block PerplexityBot?
Yes. france24.com’s robots.txt disallows PerplexityBot from the whole site (line 146: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find france24.com?
Not through its own crawler. france24.com’s robots.txt disallows OAI-SearchBot from the whole site (line 134: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.