Who blocks AI · #1248 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does foreignpolicy.com block AI crawlers?

Yes. foreignpolicy.com’s robots.txt blocks 13 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 10 more. It also blocks PetalBot among search engines.

GPTBotBlocked OAI-SearchBotAllowed ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: foreignpolicy.com/robots.txt · 94 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	line 28 · `Allow: /` under `User-agent: *`
`Bingbot`	Microsoft	Allowed	line 28 · `Allow: /` under `User-agent: *`
`DuckDuckBot`	DuckDuckGo	Allowed	line 28 · `Allow: /` under `User-agent: *`
`Applebot`	Apple	Allowed	line 28 · `Allow: /` under `User-agent: *`
`PetalBot`	Huawei	Blocked	line 65 · `Disallow: /` under `User-agent: PetalBot`
`YandexBot`	Yandex	Allowed	line 28 · `Allow: /` under `User-agent: *`
`Baiduspider`	Baidu	Allowed	line 28 · `Allow: /` under `User-agent: *`

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Allowed	line 28 · `Allow: /` under `User-agent: *`
`PerplexityBot`	Perplexity	Blocked	line 65 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Allowed	line 28 · `Allow: /` under `User-agent: *`
`DuckAssistBot`	DuckDuckGo	Blocked	line 65 · `Disallow: /` under `User-agent: DuckAssistBot`
`Amazonbot`	Amazon	Allowed	line 28 · `Allow: /` under `User-agent: *`
`YouBot`	You.com	Allowed	line 28 · `Allow: /` under `User-agent: *`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 65 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 65 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Blocked	line 65 · `Disallow: /` under `User-agent: Google-Extended`
`CCBot`	Common Crawl	Blocked	line 65 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 65 · `Disallow: /` under `User-agent: Applebot-Extended`
`GoogleOther`	Google	Allowed	line 28 · `Allow: /` under `User-agent: *`
`Meta-ExternalAgent`	Meta	Blocked	line 65 · `Disallow: /` under `User-agent: Meta-ExternalAgent`
`Bytespider`	ByteDance	Blocked	line 65 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 65 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Blocked	line 65 · `Disallow: /` under `User-agent: Claude-User`
`Perplexity-User`	Perplexity	Allowed	line 28 · `Allow: /` under `User-agent: *`
`MistralAI-User`	Mistral	Blocked	line 65 · `Disallow: /` under `User-agent: MistralAI-User`
`Meta-ExternalFetcher`	Meta	Blocked	line 65 · `Disallow: /` under `User-agent: Meta-ExternalFetcher`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	line 28 · `Allow: /` under `User-agent: *`
`SemrushBot`	Semrush	Allowed	line 28 · `Allow: /` under `User-agent: *`
`MJ12bot`	Majestic	Allowed	line 28 · `Allow: /` under `User-agent: *`
`DotBot`	Moz	Allowed	line 28 · `Allow: /` under `User-agent: *`

View foreignpolicy.com/robots.txt · 94 lines

1# As a condition of accessing this website, you agree to abide by the following

2# content signals:

4# (a) If a Content-Signal = yes, you may collect content for the corresponding

5# use.

6# (b) If a Content-Signal = no, you may not collect content for the

7# corresponding use.

8# (c) If the website operator does not include a Content-Signal for a

9# corresponding use, the website operator neither grants nor restricts

10# permission via Content-Signal with respect to the corresponding use.

12# The content signals and their meanings are:

14# search: building a search index and providing search results (e.g., returning

15# hyperlinks and short excerpts from your website's contents). Search does not

16# include providing AI-generated search summaries.

17# ai-input: inputting content into one or more AI models (e.g., retrieval

18# augmented generation, grounding, or other real-time taking of content for

19# generative AI search answers).

20# ai-train: training or fine-tuning AI models.

22# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF

23# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT

24# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.

26User-agent: *

27Content-Signal: search=yes,ai-train=no

28Allow: /

30# AI Assistants

31User-agent: DuckAssistBot

32User-agent: Gemini-Deep-Research

33User-agent: Meta-ExternalFetcher

34User-agent: MistralAI-User

35User-agent: ChatGPT-User

36User-agent: PerplexityBot

37User-agent: Perplexity‑User

38User-agent: Claude-User

40# AI Data Scrapers

41User-agent: AI2Bot

42User-agent: Applebot-Extended

43User-agent: Bytespider

44User-agent: CCBot

45User-agent: ClaudeBot

46User-agent: cohere-training-data-crawler

47User-agent: Cotoyogi

48User-agent: Datenbank Crawler

49User-agent: Diffbot

50User-agent: FacebookBot

51User-agent: Factset_spyderbot

52User-agent: Google-Extended

53User-agent: GPTBot

54User-agent: ICC-Crawler

55User-agent: Kangaroo Bot

56User-agent: Meta-ExternalAgent

57User-agent: netEstate Imprint Crawler

58User-agent: omgili

59User-agent: PanguBot

60User-agent: PetalBot

61User-agent: SemrushBot-OCOB

62User-agent: Timpibot

63User-agent: Webzio-Extended

65Disallow: /

66Allow: /$

67Allow: /staff/

68Allow: /author/

69Allow: /projects/

70Allow: /category/

71Allow: /tag/

73# General Disallow

74User-agent: *

75Disallow: /*?*wp_login_redirect=

76Disallow: /*?*view_post_comments=

77Disallow: */feed/

78Disallow: /search/*

79Disallow: /cdn-cgi/*

80Disallow: /r/api/ip-access/*

81Disallow: /newsletter-ad/*

82Disallow: /endpoint/

83Disallow: /subscribe/?*

84Disallow: /subscribe?*

85Allow: /feed/$

86Sitemap: https://foreignpolicy.com/sitemap.xml

87Sitemap: https://foreignpolicy.com/news-sitemap.xml

88User-agent: *

89Disallow: /wp-admin/

90Allow: /wp-admin/admin-ajax.php

91# Sitemap archive

92Sitemap: https://foreignpolicy.com/sitemap.xml

What do the bots actually get?

This page reads robots.txt, which states what foreignpolicy.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on foreignpolicy.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does foreignpolicy.com block GPTBot?

Yes. foreignpolicy.com’s robots.txt disallows GPTBot from the whole site (line 65: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does foreignpolicy.com block ClaudeBot?

Yes. foreignpolicy.com’s robots.txt disallows ClaudeBot from the whole site (line 65: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does foreignpolicy.com block Google-Extended?

Yes. foreignpolicy.com’s robots.txt disallows Google-Extended from the whole site (line 65: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does foreignpolicy.com block PerplexityBot?

Yes. foreignpolicy.com’s robots.txt disallows PerplexityBot from the whole site (line 65: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find foreignpolicy.com?

Yes. foreignpolicy.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.