Who blocks AI · #82 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does theguardian.com block AI crawlers?

Yes. theguardian.com’s robots.txt blocks 11 of the 19 AI crawlers we track, including ClaudeBot, PerplexityBot, CCBot and 8 more. It also blocks PetalBot among search engines.

GPTBotAllowed OAI-SearchBotAllowed ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedAllowed CCBotBlocked

robots.txt: www.theguardian.com/robots.txt · 105 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Blocked	line 102 · `Disallow: /` under `User-agent: PetalBot`
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Allowed	no rule matches the home page
`PerplexityBot`	Perplexity	Blocked	line 102 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Blocked	line 102 · `Disallow: /` under `User-agent: Claude-SearchBot`
`DuckAssistBot`	DuckDuckGo	Blocked	line 102 · `Disallow: /` under `User-agent: DuckAssistBot`
`Amazonbot`	Amazon	Blocked	line 102 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Blocked	line 102 · `Disallow: /` under `User-agent: YouBot`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Allowed	no rule matches the home page
`ClaudeBot`	Anthropic	Blocked	line 102 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Allowed	no rule matches the home page
`CCBot`	Common Crawl	Blocked	line 102 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 102 · `Disallow: /` under `User-agent: Applebot-Extended`
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Blocked	line 102 · `Disallow: /` under `User-agent: meta-externalagent`
`Bytespider`	ByteDance	Blocked	line 102 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Allowed	no rule matches the home page
`Claude-User`	Anthropic	Blocked	line 102 · `Disallow: /` under `User-agent: Claude-User`
`Perplexity-User`	Perplexity	Allowed	no rule matches the home page
`MistralAI-User`	Mistral	Allowed	no rule matches the home page
`Meta-ExternalFetcher`	Meta	Allowed	no rule matches the home page

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 102 · `Disallow: /` under `User-agent: AhrefsBot`
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View www.theguardian.com/robots.txt · 105 lines

1# This is the robots.txt file for theguardian.com

3# Guardian content is made available under our terms and conditions of use.

4# Any other uses are not permitted, incl. but not limited to: for large language

5# models (LLMs), machine learning and/or artificial intelligence-related

6# purposes; with any of the aforementioned technologies; and/or for any

7# commercial purposes. Contact licensing@theguardian.com for assistance

9User-agent: *

10Disallow: /sendarticle/

11Disallow: /Users/

12Disallow: /users/

13Disallow: /*/print$

14Disallow: /email/

15Disallow: /contactus/

16Disallow: /share/

17Disallow: /websearch

18Disallow: /*?commentpage=

19Disallow: /whsmiths/

20Disallow: /external/overture/

21Disallow: /discussion/report-abuse/*

22Disallow: /discussion/report-abuse-ajax/*

23Disallow: /discussion/comment-permalink/*

24Disallow: /discussion/report-abuse/*

25Disallow: /discussion/user-report-abuse/*

26Disallow: /discussion/handlers/*

27Disallow: /discussion/your-profile

28Disallow: /discussion/your-comments

29Disallow: /discussion/edit-profile

30Disallow: /discussion/search/comments

31Disallow: /discussion/*

32Disallow: /search

33Disallow: /music/artist/*

34Disallow: /music/album/*

35Disallow: /books/data/*

36Disallow: /settings/

37Disallow: /embed/

38Disallow: /*styles/js-on.css$

39Disallow: /sport/olympics/2008/events/*

40Disallow: /sport/olympics/2008/medals/*

41Disallow: /f/healthcheck

42Disallow: /sections

43Disallow: /top-stories

44Disallow: /most-read/sport

45Disallow: /articles

46Disallow: /global$

47Disallow: /*/feedarticle/*

48Disallow: /travel/2013/aug/22/been-there-readers-competition?*

49Disallow: /preference/*

50Disallow: /59666047/

51Disallow: /print/

52Disallow: /info/tech-feedback

53Disallow: /production-monitoring/

54Disallow: *.emailjson

55Disallow: *.emailtxt

56Disallow: /headline.txt

57Disallow: *?*dcr=apps*

59User-agent: Mediapartners-Google

60Disallow:

62Sitemap: http://www.theguardian.com/sitemaps/news.xml

63Sitemap: http://www.theguardian.com/sitemaps/video.xml

65User-agent: NewsNow

66User-agent: CCBot

67User-agent: TurnitinBot

68User-agent: PetalBot

69User-agent: MoodleBot

70User-agent: FacebookBot

71User-agent: Bytespider

72User-agent: Mojeek

73User-agent: JenkersBot

74User-agent: Seekr

75User-agent: YouBot

76User-agent: Arquivo-web-crawler

77User-agent: coccocbot-web

78User-agent: SeznamBot

79User-Agent: PerplexityBot

80User-Agent: yacy

81User-agent: anthropic-ai

82User-agent: ClaudeBot

83User-agent: Claude-SearchBot

84User-agent: Claude-User

85User-agent: AwarioRssBot

86User-agent: AwarioSmartBot

87User-agent: SentiOne

88User-agent: ImageSift

89User-agent: Applebot-Extended

90User-agent: YandexAdditional

91User-agent: YandexAdditionalBot

92User-agent: scalepostAI

93User-agent: Buck

94User-agent: meta-externalagent

95User-agent: Amazonbot

96User-agent: amazon-QBusiness

97User-agent: DuckAssistBot

98User-agent: Google-CloudVertexBot

99User-agent: Amzn-SearchBot

100User-agent: AhrefsBot

101User-agent: AhrefsSiteAudit

102Disallow: /

103

104License: https://theguardian.com/license.xml

105

What do the bots actually get?

This page reads robots.txt, which states what theguardian.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on theguardian.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does theguardian.com block GPTBot?

No. theguardian.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does theguardian.com block ClaudeBot?

Yes. theguardian.com’s robots.txt disallows ClaudeBot from the whole site (line 102: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does theguardian.com block Google-Extended?

No. theguardian.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does theguardian.com block PerplexityBot?

Yes. theguardian.com’s robots.txt disallows PerplexityBot from the whole site (line 102: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find theguardian.com?

Yes. theguardian.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.