Who blocks AI · #1332 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does rfi.fr block AI crawlers?

Yes. rfi.fr’s robots.txt blocks 13 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 10 more. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: www.rfi.fr/robots.txt · 1,051 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Partly blocked	line 48 · `Disallow: /*/_ws/urgent`
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Partly blocked	line 48 · `Disallow: /*/_ws/urgent`
`PetalBot`	Huawei	Allowed	line 1017 · `Allow: /` under `User-agent: PetalBot`
`YandexBot`	Yandex	Allowed	line 1045 · `Allow: /` under `User-agent: YandexBot`
`Baiduspider`	Baidu	Allowed	line 1001 · `Allow: /` under `User-agent: Baiduspider`

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 161 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 173 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Blocked	line 86 · `Disallow: /` under `User-agent: Claude-SearchBot`
`DuckAssistBot`	DuckDuckGo	Blocked	line 110 · `Disallow: /` under `User-agent: DuckAssistBot`
`Amazonbot`	Amazon	Allowed	no rule matches the home page
`YouBot`	You.com	Blocked	line 194 · `Disallow: /` under `User-agent: YouBot`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 122 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 95 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Blocked	line 119 · `Disallow: /` under `User-agent: Google-Extended`
`CCBot`	Common Crawl	Blocked	line 297 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Allowed	no rule matches the home page
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Allowed	no rule matches the home page
`Bytespider`	ByteDance	Allowed	no rule matches the home page

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 80 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Blocked	line 89 · `Disallow: /` under `User-agent: Claude-User`
`Perplexity-User`	Perplexity	Blocked	line 167 · `Disallow: /` under `User-agent: Perplexity-User`
`MistralAI-User`	Mistral	Blocked	line 152 · `Disallow: /` under `User-agent: MistralAI-User`
`Meta-ExternalFetcher`	Meta	Allowed	no rule matches the home page

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 216 · `Disallow: /` under `User-agent: AhrefsBot`
`SemrushBot`	Semrush	Blocked	line 696 · `Disallow: /` under `User-agent: SemrushBot`
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Blocked	line 357 · `Disallow: /` under `User-agent: DotBot`

View www.rfi.fr/robots.txt · 1,051 lines

1# France Medias Monde [2026-02-26] - francemediasmonde.com

2## RFI - rfi.fr - HTTPS

4### Sitemaps

5Sitemap: https://www.rfi.fr/sitemaps/fr/index.xml

6Sitemap: https://www.rfi.fr/sitemaps/en/index.xml

7Sitemap: https://www.rfi.fr/sitemaps/es/index.xml

8Sitemap: https://www.rfi.fr/sitemaps/pt/index.xml

9Sitemap: https://www.rfi.fr/sitemaps/br/index.xml

10Sitemap: https://www.rfi.fr/sitemaps/cn/index.xml

11Sitemap: https://www.rfi.fr/sitemaps/tw/index.xml

12Sitemap: https://www.rfi.fr/sitemaps/km/index.xml

13Sitemap: https://www.rfi.fr/sitemaps/ha/index.xml

14Sitemap: https://www.rfi.fr/sitemaps/fa/index.xml

15Sitemap: https://www.rfi.fr/sitemaps/sw/index.xml

16Sitemap: https://www.rfi.fr/sitemaps/ma/index.xml

17Sitemap: https://www.rfi.fr/sitemaps/ff/index.xml

18Sitemap: https://www.rfi.fr/sitemaps/ro/index.xml

19Sitemap: https://www.rfi.fr/sitemaps/ru/index.xml

20Sitemap: https://www.rfi.fr/sitemaps/uk/index.xml

21Sitemap: https://www.rfi.fr/sitemaps/vi/index.xml

23### News Sitemaps

24Sitemap: https://www.rfi.fr/sitemaps/fr/news.xml

25Sitemap: https://www.rfi.fr/sitemaps/en/news.xml

26Sitemap: https://www.rfi.fr/sitemaps/es/news.xml

27Sitemap: https://www.rfi.fr/sitemaps/pt/news.xml

28Sitemap: https://www.rfi.fr/sitemaps/br/news.xml

29Sitemap: https://www.rfi.fr/sitemaps/cn/news.xml

30Sitemap: https://www.rfi.fr/sitemaps/tw/news.xml

31Sitemap: https://www.rfi.fr/sitemaps/km/news.xml

32Sitemap: https://www.rfi.fr/sitemaps/ha/news.xml

33Sitemap: https://www.rfi.fr/sitemaps/fa/news.xml

34Sitemap: https://www.rfi.fr/sitemaps/sw/news.xml

35Sitemap: https://www.rfi.fr/sitemaps/ro/news.xml

36Sitemap: https://www.rfi.fr/sitemaps/ru/news.xml

37Sitemap: https://www.rfi.fr/sitemaps/uk/news.xml

38Sitemap: https://www.rfi.fr/sitemaps/vi/news.xml

40### General rules

41User-agent: *

42Disallow:

44User-agent: *

45Disallow: /ro/taxonomy/

47User-agent: Googlebot

48Disallow: /*/_ws/urgent

50### AI Bots blocking

51### >> Contact us at syndication[@]francemm.com for partnership opportunities

52User-agent: AI2Bot

53Disallow: /

55User-agent: Ai2Bot-Dolma

56Disallow: /

58User-agent: AlibabaBot

59Disallow: /

61User-agent: AliyunBot

62Disallow: /

64User-agent: anthropic-ai

65Disallow: /

67User-agent: Baidu-YunGuanCe

68Disallow: /

70User-agent: Big Sur AI

71Disallow: /

73User-agent: ByteDance-AI

74Disallow: /

76User-agent: ChatGPT Operator

77Disallow: /

79User-agent: ChatGPT-User

80Disallow: /

82User-agent: Claude-Code

83Disallow: /

85User-agent: Claude-SearchBot

86Disallow: /

88User-agent: Claude-User

89Disallow: /

91User-agent: Claude-Web

92Disallow: /

94User-agent: ClaudeBot

95Disallow: /

97User-agent: cohere-ai

98Disallow: /

100User-agent: cohere-training-data-crawler

101Disallow: /

102

103User-agent: DeepSeekBot

104Disallow: /

105

106User-agent: DigitalOceanGenAICrawler

107Disallow: /

108

109User-agent: DuckAssistBot

110Disallow: /

111

112User-agent: ERNIEBot

113Disallow: /

114

115User-agent: ERNIE-Bot

116Disallow: /

117

118User-agent: Google-Extended

119Disallow: /

120

121User-agent: GPTBot

122Disallow: /

123

124User-agent: Grok

125Disallow: /

126

127User-agent: img2dataset

128Disallow: /

129

130User-agent: Kimibot

131Disallow: /

132

133User-agent: KimiCrawler

134Disallow: /

135

136User-agent: LINER Bot

137Disallow: /

138

139User-agent: Meta Llama

140Disallow: /

141

142User-agent: MetaAIBot

143Disallow: /

144

145User-agent: Moonshot-AI

146Disallow: /

147

148User-agent: MoonshotBot

149Disallow: /

150

151User-agent: MistralAI-User

152Disallow: /

153

154User-agent: MistralAI-User/1.0

155Disallow: /

156

157User-agent: MyCentralAIScraperBot

158Disallow: /

159

160User-agent: OAI-SearchBot

161Disallow: /

162

163User-agent: PanguBot

164Disallow: /

165

166User-agent: Perplexity-User

167Disallow: /

168

169User-agent: Perplexity-User/1.0

170Disallow: /

171

172User-agent: PerplexityBot

173Disallow: /

174

175User-agent: quillbot.com

176Disallow: /

177

178User-agent: QwenBot

179Disallow: /

180

181User-agent: SBIntuitionsBot

182Disallow: /

183

184User-agent: TongyiBot

185Disallow: /

186

187User-agent: Tongyi-Crawler

188Disallow: /

189

190User-agent: YiyanBot

191Disallow: /

192

193User-agent: YouBot

194Disallow: /

195

196### General Bots blocking

197User-agent: 008

198Disallow: /

199

200User-agent: 5emeRue

201Disallow: /

202

203User-agent: 5erue

204Disallow: /

205

206User-agent: adequat

207Disallow: /

208

209User-agent: adequat-systems

210Disallow: /

211

212User-agent: ADmantX bot

213Disallow: /

214

215User-agent: AhrefsBot

216Disallow: /

217

218User-agent: Alexibot

219Disallow: /

220

221User-agent: AlvinetSpider

222Disallow: /

223

224User-agent: AmiSoftware

225Disallow: /

226

227User-agent: Antenne Hatena

228Disallow: /

229

230User-agent: ApocalXExplorerBot

231Disallow: /

232

233User-agent: Argus

234Disallow: /

235

236User-agent: Ask n read

237Disallow: /

238

239User-agent: asknread.com

240Disallow: /

241

242User-agent: asterias

243Disallow: /

244

245User-agent: Augure

246Disallow: /

247

248User-agent: augure

249Disallow: /

250

251User-agent: auramundi

252Disallow: /

253

254User-agent: Babya Discoverer

255Disallow: /

256

257User-agent: BackDoorBot/1.0

258Disallow: /

259

260User-agent: BacklinkCrawler

261Disallow: /

262

263User-agent: bitlybot

264Disallow: /

265

266User-agent: BizInformation

267Disallow: /

268

269User-agent: Black Hole

270Disallow: /

271

272User-agent: Blekkobot

273Disallow: /

274

275User-agent: BLEXBot

276Disallow: /

277

278User-agent: Bloodhound

279Disallow: /

280

281User-agent: BlowFish/1.0

282Disallow: /

283

284User-agent: BotALot

285Disallow: /

286

287User-agent: BuiltBotTough

288Disallow: /

289

290User-agent: Bullseye/1.0

291Disallow: /

292

293User-agent: BunnySlippers

294Disallow: /

295

296User-agent: CCBot

297Disallow: /

298

299User-agent: Cegbfeieh

300Disallow: /

301

302User-agent: CheeseBot

303Disallow: /

304

305User-agent: CherryPicker

306Disallow: /

307

308User-agent: CherryPickerElite/1.0

309Disallow: /

310

311User-agent: CherryPickerSE/1.0

312Disallow: /

313

314User-agent: Cision

315Disallow: /

316

317User-agent: Cliqzbot

318Disallow: /

319

320User-agent: coexel

321Disallow: /

322

323User-agent: ConveraCrawler

324Disallow: /

325

326User-agent: CopyRightCheck

327Disallow: /

328

329User-agent: Corporama

330Disallow: /

331

332User-agent: cosmos

333Disallow: /

334

335User-agent: Crescent

336Disallow: /

337

338User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0

339Disallow: /

340

341User-agent: cydralspider

342Disallow: /

343

344User-agent: Digimind

345Disallow: /

346

347User-agent: DISCo Pump 3.1

348Disallow: /

349

350User-agent: discobot

351Disallow: /

352

353User-agent: DittoSpyder

354Disallow: /

355

356User-agent: DotBot

357Disallow: /

358

359User-agent: dotbot

360Disallow: /

361

362User-agent: Dotmic DotBot

363Disallow: /

364

365User-agent: Download Ninja

366Disallow: /

367

368User-agent: downloadexpress

369Disallow: /

370

371User-agent: EDD

372Disallow: /

373

374User-agent: ellisphere

375Disallow: /

376

377User-agent: EmailCollector

378Disallow: /

379

380User-agent: EmailSiphon

381Disallow: /

382

383User-agent: EmailWolf

384Disallow: /

385

386User-agent: EroCrawler

387Disallow: /

388

389User-agent: EtaoSpider

390Disallow: /

391

392User-agent: eureka

393Disallow: /

394

395User-agent: Europresse

396Disallow: /

397

398User-agent: Exabot

399Disallow: /

400

401User-agent: Explore

402Disallow: /

403

404User-agent: ExtractorPro

405Disallow: /

406

407User-agent: Ezooms Robot

408Disallow: /

409

410User-agent: Fasterfox

411Disallow: /

412

413User-agent: Fetch

414Disallow: /

415

416User-agent: FirstRain

417Disallow: /

418

419User-agent: Flamingo_SearchEngine

420Disallow: /

421

422User-agent: Foobot

423Disallow: /

424

425User-agent: gammaSpider

426Disallow: /

427

428User-agent: GnowitNewsbot

429Disallow: /

430

431User-agent: grub-client

432Disallow: /

433

434User-agent: Harvest/1.5

435Disallow: /

436

437User-agent: hloader

438Disallow: /

439

440User-agent: houzzbot

441Disallow: /

442

443User-agent: httplib

444Disallow: /

445

446User-agent: HTTrack

447Disallow: /

448

449User-agent: HTTrack 3.0

450Disallow: /

451

452User-agent: humanlinks

453Disallow: /

454

455User-agent: Igentia

456Disallow: /

457

458User-agent: indexer

459Disallow: /

460

461User-agent: InfoNaviRobot

462Disallow: /

463

464User-agent: infoseek

465Disallow: /

466

467User-agent: JennyBot

468Disallow: /

469

470User-agent: Jetbot

471Disallow: /

472

473User-agent: JikeSpider

474Disallow: /

475

476User-agent: k2spider

477Disallow: /

478

479User-agent: kbcrawl

480Disallow: /

481

482User-agent: Kenjin Spider

483Disallow: /

484

485User-agent: Knowings

486Disallow: /

487

488User-agent: LamarkBot

489Disallow: /

490

491User-agent: larbin

492Disallow: /

493

494User-agent: leadbox

495Disallow: /

496

497User-agent: LexiBot

498Disallow: /

499

500User-agent: libWeb/clsHTTP

501Disallow: /

502

503User-agent: libwww

504Disallow: /

505

506User-agent: LinkextractorPro

507Disallow: /

508

509User-agent: linkfluence

510Disallow: /

511

512User-agent: linko

513Disallow: /

514

515User-agent: LinkScan/8.1a Unix

516Disallow: /

517

518User-agent: LinkWalker

519Disallow: /

520

521User-agent: lwp-trivial

522Disallow: /

523

524User-agent: lwp-trivial/1.34

525Disallow: /

526

527User-agent: manageo

528Disallow: /

529

530User-agent: Mata Hari

531Disallow: /

532

533User-agent: MaxPointCrawler

534Disallow: /

535

536User-agent: mediacompil

537Disallow: /

538

539User-agent: Mediatoolkitbot

540Disallow: /

541

542User-agent: Meltawer

543Disallow: /

544

545User-agent: Meltwater

546Disallow: /

547

548User-agent: mention

549Disallow: /

550

551User-agent: Microsoft.URL.Control

552Disallow: /

553

554User-agent: MIIxpc

555Disallow: /

556

557User-agent: MIIxpc/4.2

558Disallow: /

559

560User-agent: Mister PiX

561Disallow: /

562

563User-agent: MJ12bot

564Disallow: /

565

566User-agent: MLBot

567Disallow: /

568

569User-agent: moget

570Disallow: /

571

572User-agent: moget/2.1

573Disallow: /

574

575User-agent: Moreover

576Disallow: /

577

578User-agent: MS Search 4.0 Robot

579Disallow: /

580

581User-agent: MS Search 5.0 Robot

582Disallow: /

583

584User-agent: MSIECrawler

585Disallow: /

586

587User-agent: mytwip

588Disallow: /

589

590User-agent: Naverbot

591Disallow: /

592

593User-agent: NetAnts

594Disallow: /

595

596User-agent: NetAttache

597Disallow: /

598

599User-agent: netEstate NE Crawler

600Disallow: /

601

602User-agent: NetMechanic

603Disallow: /

604

605User-agent: newscan-online

606Disallow: /

607

608User-agent: Newzbin

609Disallow: /

610

611User-agent: NICErsPRO

612Disallow: /

613

614User-agent: NPBot

615Disallow: /

616

617User-agent: Nutch

618Disallow: /

619

620User-agent: ObjectsSearch

621Disallow: /

622

623User-agent: Offline Explorer

624Disallow: /

625

626User-agent: omgili

627Disallow: /

628

629User-agent: omgilibot

630Disallow: /

631

632User-agent: Openfind

633Disallow: /

634

635User-agent: OpenindexSpider

636Disallow: /

637

638User-agent: opinion-tracker

639Disallow: /

640

641User-agent: Perl LWP

642Disallow: /

643

644User-agent: Pimptrain

645Disallow: /

646

647User-agent: ProPowerBot/2.14

648Disallow: /

649

650User-agent: ProWebWalker

651Disallow: /

652

653User-agent: proxem

654Disallow: /

655

656User-agent: psbot

657Disallow: /

658

659User-agent: QuepasaCreep

660Disallow: /

661

662User-agent: QueryN Metasearch

663Disallow: /

664

665User-agent: Qwam content intelligence

666Disallow: /

667

668User-agent: Raven

669Disallow: /

670

671User-agent: readability.com

672Disallow: /

673

674User-agent: RepoMonkey

675Disallow: /

676

677User-agent: Riddler

678Disallow: /

679

680User-agent: RMA

681Disallow: /

682

683User-agent: rogerbot

684Disallow: /

685

686User-agent: scoop.it

687Disallow: /

688

689User-agent: score3

690Disallow: /

691

692User-agent: SearchmetricsBot

693Disallow: /

694

695User-agent: SemrushBot

696Disallow: /

697

698User-agent: sentibot

699Disallow: /

700

701User-agent: SEOkicks

702Disallow: /

703

704User-agent: SEOkicks-Robot

705Disallow: /

706

707User-agent: SeznamBot

708Disallow: /

709

710User-agent: SightupBot

711Disallow: /

712

713User-agent: Sindup

714Disallow: /

715

716User-agent: SISTRIX Crawler

717Disallow: /

718

719User-agent: SiteBot

720Disallow: /

721

722User-agent: sitecheck.internetseer.com

723Disallow: /

724

725User-agent: SiteSnagger

726Disallow: /

727

728User-agent: SiteSucker

729Disallow: /

730

731User-agent: Sogou web spider

732Disallow: /

733

734User-agent: sosospider

735Disallow: /

736

737User-agent: SpankBot

738Disallow: /

739

740User-agent: spanner

741Disallow: /

742

743User-agent: Speedy

744Disallow: /

745

746User-agent: spotter

747Disallow: /

748

749User-agent: SputnikBot

750Disallow: /

751

752User-agent: suggybot

753Disallow: /

754

755User-agent: SuperBot

756Disallow: /

757

758User-agent: SuperBot/2.6

759Disallow: /

760

761User-agent: SurveyBot

762Disallow: /

763

764User-agent: suzuran

765Disallow: /

766

767User-agent: Synthesio

768Disallow: /

769

770User-agent: Szukacz/1.4

771Disallow: /

772

773User-agent: Talkwater

774Disallow: /

775

776User-agent: Teleport

777Disallow: /

778

779User-agent: TeleportPro

780Disallow: /

781

782User-agent: Telesoft

783Disallow: /

784

785User-agent: The Intraformant

786Disallow: /

787

788User-agent: TheNomad

789Disallow: /

790

791User-agent: TightTwatBot

792Disallow: /

793

794User-agent: Titan

795Disallow: /

796

797User-agent: toCrawl/UrlDispatcher

798Disallow: /

799

800User-agent: TosCrawler

801Disallow: /

802

803User-agent: trendeo

804Disallow: /

805

806User-agent: trendictionbot

807Disallow: /

808

809User-agent: trendybuzz

810Disallow: /

811

812User-agent: True_Robot

813Disallow: /

814

815User-agent: True_Robot/1.0

816Disallow: /

817

818User-agent: turingos

819Disallow: /

820

821User-agent: up2news

822Disallow: /

823

824User-agent: Updownerbot

825Disallow: /

826

827User-agent: UptimeRobot/2.0

828Disallow: /

829

830User-agent: UrlPouls

831Disallow: /

832

833User-agent: URLy Warning

834Disallow: /

835

836User-agent: VCI

837Disallow: /

838

839User-agent: vecteurplus

840Disallow: /

841

842User-agent: Verif

843Disallow: /

844

845User-agent: verticalsearch

846Disallow: /

847

848User-agent: vsw

849Disallow: /

850

851User-agent: wapspider

852Disallow: /

853

854User-agent: Web Image Collector

855Disallow: /

856

857User-agent: WebAuto

858Disallow: /

859

860User-agent: WebBandit

861Disallow: /

862

863User-agent: WebBandit/3.50

864Disallow: /

865

866User-agent: WebCopier

867Disallow: /

868

869User-agent: webcopy

870Disallow: /

871

872User-agent: WebEnhancer

873Disallow: /

874

875User-agent: WebmasterWorldForumBot

876Disallow: /

877

878User-agent: webmirror

879Disallow: /

880

881User-agent: WebReaper

882Disallow: /

883

884User-agent: WebSauger

885Disallow: /

886

887User-agent: website extractor

888Disallow: /

889

890User-agent: Website Quester

891Disallow: /

892

893User-agent: Webster Pro

894Disallow: /

895

896User-agent: WebStripper

897Disallow: /

898

899User-agent: WebStripper/2.02

900Disallow: /

901

902User-agent: WebZinger

903Disallow: /

904

905User-agent: WebZIP

906Disallow: /

907

908User-agent: wget

909Disallow: /

910

911User-agent: Wget

912Disallow: /

913

914User-agent: WikioFeedBot

915Disallow: /

916

917User-agent: winello

918Disallow: /

919

920User-agent: WinHTTrack

921Disallow: /

922

923User-agent: WiseGuys Robot

924Disallow: /

925

926User-agent: WWW-Collector-E

927Disallow: /

928

929User-agent: Xenu Link Sleuth/1.3.8

930Disallow: /

931

932User-agent: XoviBot

933Disallow: /

934

935User-agent: yacy

936Disallow: /

937

938User-agent: YisouSpider

939Disallow: /

940

941User-agent: Youmag

942Disallow: /

943

944User-agent: YRSPider

945Disallow: /

946

947User-agent: Zealbot

948Disallow: /

949

950User-agent: Zeus

951Disallow: /

952

953User-agent: Zite

954Disallow: /

955

956User-agent: Zookabot

957Disallow: /

958

959User-agent: ZyBORG

960Disallow: /

961

962### Partners

963User-agent: CriteoBot/0.1

964Allow: /

965Crawl-delay: 5

966

967User-agent: grapeshot

968Allow: /

969Crawl-delay: 5

970

971User-agent: INA dlweb

972Allow: /

973Crawl-delay: 5

974

975User-agent: inkl

976Allow: /

977Crawl-delay: 5

978

979User-agent: Kantar

980Allow: /

981Crawl-delay: 5

982

983User-agent: outbrain

984Allow: /

985Crawl-delay: 5

986

987User-agent: proximic

988Allow: /

989Crawl-delay: 10

990

991User-agent: weborama-fetcher

992Allow: /

993Crawl-delay: 5

994

995### Too fast bots

996User-agent: archive.org_bot

997Allow: /

998Crawl-delay: 10

999

1000User-agent: Baiduspider

1001Allow: /

1002Crawl-delay: 10

1003

1004User-agent: coccocbot-web

1005Allow: /

1006Crawl-delay: 10

1007

1008User-agent: Factiva

1009Allow: /

1010Crawl-delay: 10

1011

1012User-agent: NewsNow

1013Allow: /

1014Crawl-delay: 10

1015

1016User-agent: PetalBot

1017Allow: /

1018Crawl-delay: 10

1019

1020User-agent: proximic

1021Allow: /

1022Crawl-delay: 10

1023

1024User-agent: Soso Spider

1025Allow: /

1026Crawl-delay: 10

1027

1028User-agent: Sogou Spider

1029Allow: /

1030Crawl-delay: 10

1031

1032User-agent: TurnitinBot

1033Allow: /

1034Crawl-delay: 10

1035

1036User-agent: Turnitin Robot

1037Allow: /

1038Crawl-delay: 10

1039

1040User-agent: Yandex

1041Allow: /

1042Crawl-delay: 10

1043

1044User-agent: YandexBot

1045Allow: /

1046Crawl-delay: 10

1047

1048User-agent: Yeti

1049Allow: /

1050Crawl-delay: 10

1051

What do the bots actually get?

This page reads robots.txt, which states what rfi.fr asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on rfi.fr Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does rfi.fr block GPTBot?

Yes. rfi.fr’s robots.txt disallows GPTBot from the whole site (line 122: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does rfi.fr block ClaudeBot?

Yes. rfi.fr’s robots.txt disallows ClaudeBot from the whole site (line 95: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does rfi.fr block Google-Extended?

Yes. rfi.fr’s robots.txt disallows Google-Extended from the whole site (line 119: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does rfi.fr block PerplexityBot?

Yes. rfi.fr’s robots.txt disallows PerplexityBot from the whole site (line 173: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find rfi.fr?

Not through its own crawler. rfi.fr’s robots.txt disallows OAI-SearchBot from the whole site (line 161: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.