Who blocks AI · #416 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does dailymail.com block AI crawlers?

Yes. dailymail.com’s robots.txt blocks 18 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 15 more. It also blocks PetalBot among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: www.dailymail.com/robots.txt · 189 lines
llms.txt: found
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Blocked	line 169 · `Disallow: /` under `User-agent: PetalBot`
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 169 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 169 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Blocked	line 169 · `Disallow: /` under `User-agent: Claude-SearchBot`
`DuckAssistBot`	DuckDuckGo	Blocked	line 169 · `Disallow: /` under `User-agent: DuckAssistBot`
`Amazonbot`	Amazon	Blocked	line 169 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Blocked	line 169 · `Disallow: /` under `User-agent: YouBot`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 169 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 169 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Blocked	line 169 · `Disallow: /` under `User-agent: Google-Extended`
`CCBot`	Common Crawl	Blocked	line 169 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 169 · `Disallow: /` under `User-agent: Applebot-Extended`
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Blocked	line 169 · `Disallow: /` under `User-agent: Meta-ExternalAgent`
`Bytespider`	ByteDance	Blocked	line 169 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 169 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Blocked	line 169 · `Disallow: /` under `User-agent: Claude-User`
`Perplexity-User`	Perplexity	Blocked	line 169 · `Disallow: /` under `User-agent: Perplexity-User`
`MistralAI-User`	Mistral	Blocked	line 169 · `Disallow: /` under `User-agent: MistralAI-User`
`Meta-ExternalFetcher`	Meta	Blocked	line 169 · `Disallow: /` under `User-agent: Meta-ExternalFetcher`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 169 · `Disallow: /` under `User-agent: AhrefsBot`
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View www.dailymail.com/robots.txt · 189 lines

1# Robots.txt for https://www.dailymail.com/ updated 04/11/2026

4# Daily Mail content is made available for your personal and non-commercial

5# use subject to our Terms and Conditions of use:

6# https://www.dailymail.com/terms

7# Use of any device, tool, or process designed to data mine or scrape the content

8# using automated means is prohibited without prior written permission from

9# DMG MEDIA LIMITED. Prohibited uses include but are not limited to:

10# (1) text and data mining activities under Art. 4 of the EU Directive on Copyright in

11# the Digital Single Market;

12# (2) the development of any software, machine learning, artificial intelligence (AI),

13# and/or large language models (LLMs);

14# (3) creating or providing archived or cached data sets containing our content to others; and/or

15# (4) any commercial purposes.

16# For any licensing queries please contact partnerships@dmgmedia.co.uk.

17#

18#

19# All Robots

20User-agent: *

21#

22# Begin Standard Rules

23Disallow: /*readcommentshtml*

24Disallow: /*login?redirectPath=

25Disallow: /*logout?redirectPath=

26Disallow: /*nextThread.html

27Disallow: /*previousThread.html

28Disallow: /*questionId*

29Disallow: /*selectedImage*

30Disallow: /*threadIndex=*

31Disallow: /*topGallery*

32Disallow: /ce/item.cms*

33Disallow: /guide/*

34Disallow: /registration/*

35Disallow: /best-buys/clickout/*

36Disallow: /best-buys/m/clickout/*

37Disallow: /weather.html?latitude=*

38Disallow: /weather.html?old=*

39Disallow: /home/search.html*

40Disallow: /best-buys/mercury/out*

41Disallow: /api/infinite-list.html

42Disallow: /api/infinite-list.html

43Disallow: /mobileapps/*

45#

46# Disallow Money for Google News

47User-agent: Googlebot-News

48Disallow: /tmoney/*

49#

50# Allow Adsense

51User-agent: Mediapartners-Google

52Disallow:

53#

54#

55User-agent: CrystalSemanticsBot

56Disallow: /

57#

58User-agent: archive-it.org

59User-agent: archive.org_bot

60User-agent: ArchiveBot

61User-agent: Arquivo-web-crawler

62User-agent: Authory

63User-agent: babel

64User-agent: bl.uk_ldfc_bot

65User-agent: bl.uk_ldfc_renderbot

66User-agent: bne.es_bot

67User-agent: bnf.fr_bot

68User-agent: ecoresearch

69User-agent: ecoresearchCrawler

70User-agent: Europarchive.org

71User-agent: kb.dk_bot

72User-agent: Kulturarw3

73User-agent: LivelapBot

74User-agent: mediacloud

75User-agent: mirrorweb

76User-agent: MSIECrawler

77User-agent: nationalarchives.gov.uk

78User-agent: netarkivet.dk

79User-agent: netEstate NE Crawler

80User-agent: Nicecrawler

81User-agent: nl-israel_iaharvester2026

82User-agent: nlnbot

83User-agent: nlncrawler

84User-agent: PageFreezer

85User-agent: SafeDNS

86User-agent: SmarshBot

87User-agent: special_archiver

88User-agent: tkl.iis.u-tokyo.ac.jp

89User-agent: TurnitinBot

90User-agent: webarchiv.cz

91User-agent: WordPress.com mShots

92User-agent: XY-Archive-Compliance-Archiver

93User-agent: XY-Archive-Compliance-Crawler

94Disallow: /

95#

96User-agent: AhrefsBot

97User-agent: AI2Bot

98User-agent: Ai2Bot-Dolma

99User-agent: Amazonbot

100User-agent: amazon-kendra-

101User-agent: anthropic-ai

102User-agent: Applebot-Extended

103User-agent: bedrockbot

104User-agent: Bytespider

105User-agent: CloudflareBrowserRenderingCrawler

106User-agent: CCBot

107User-agent: ChatGLM-Spider

108User-agent: ChatGPT-User

109User-agent: ClaudeBot

110User-agent: Claude-SearchBot

111User-agent: Claude-User

112User-agent: Claude-Web

113User-agent: cohere-ai

114User-agent: cohere-training-data-crawler

115User-agent: Cotoyogi

116User-agent: DeepSeekBot

117User-agent: Diffbot

118User-agent: DuckAssistBot

119User-agent: EchoboxBot

120User-agent: ExaSearchBot

121User-agent: FacebookBot

122User-agent: FriendlyCrawler

123User-agent: Gemini-Deep-Research

124User-agent: Google-CloudVertexBot

125User-agent: Google-Extended

126User-agent: GoogleOther-Image

127User-agent: GoogleOther-Video

128User-agent: GPTBot

129User-agent: GPTBot-Search

130User-agent: Grok

131User-agent: iaskspider/2.0

132User-agent: ICC-Crawler

133User-agent: ImagesiftBot

134User-agent: img2dataset

135User-agent: ISSCyberRiskCrawler

136User-agent: Kangaroo Bot

137User-agent: KunatoCrawler

138User-agent: KeenableBot

139User-agent: LinerBot

140User-agent: Meltwater

141User-agent: Meta-ExternalAgent

142User-agent: Meta-ExternalFetcher

143User-agent: Meta-WebIndexer

144User-agent: MistralAI-User

145User-agent: OAI-Operator

146User-agent: OAI-SearchBot

147User-agent: omgili

148User-agent: omgilibot

149User-agent: PanguBot

150User-agent: PerplexityBot

151User-agent: Perplexity-User

152User-agent: PetalBot

153User-agent: QualifiedBot

154User-agent: Scrapy

155User-agent: Seekr

156User-agent: Sidetrade indexer bot

157User-agent: TaraGroup Intelligent Bot

158User-agent: TikTokSpider

159User-agent: Timpibot

160User-agent: VelenPublicWebCrawler

161User-agent: WARDBot

162User-agent: Webzio-Extended

163User-agent: wpbot

164User-agent: WRTNBot

165User-agent: xAI-SearchBot

166User-agent: YouBot

167Allow: /*_.html

168Allow: /llms.txt

169Disallow: /

170#

171#Allow Google sandbox privacy testing

172User-agent: GoogleOther

173Allow: /.well-known/privacy-sandbox-attestations.json

174#

175# Sitemap Files

176Sitemap: https://www.dailymail.com/google-news-sitemap.xml

177Sitemap: https://www.dailymail.com/video-new-sitemap.xml

178Sitemap: https://www.dailymail.com/sitemap-articles-year~2021.xml

179Sitemap: https://www.dailymail.com/sitemap-articles-year~2022.xml

180Sitemap: https://www.dailymail.com/sitemap-articles-year~2023.xml

181Sitemap: https://www.dailymail.com/sitemap-articles-year~2024.xml

182Sitemap: https://www.dailymail.com/sitemap-articles-year~2025.xml

183Sitemap: https://www.dailymail.com/sitemap-articles-year~2026.xml

184Sitemap: https://www.dailymail.com/sitemap-videos-year~2021.xml

185Sitemap: https://www.dailymail.com/sitemap-videos-year~2022.xml

186Sitemap: https://www.dailymail.com/sitemap-videos-year~2023.xml

187Sitemap: https://www.dailymail.com/sitemap-videos-year~2024.xml

188Sitemap: https://www.dailymail.com/sitemap-videos-year~2025.xml

189Sitemap: https://www.dailymail.com/sitemap-videos-year~2026.xml

What do the bots actually get?

This page reads robots.txt, which states what dailymail.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on dailymail.com Test a path in the robots.txt tester

History

19 October 2026: first checked.
26 October 2026: robots.txt readable again.

Questions

Does dailymail.com block GPTBot?

Yes. dailymail.com’s robots.txt disallows GPTBot from the whole site (line 169: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does dailymail.com block ClaudeBot?

Yes. dailymail.com’s robots.txt disallows ClaudeBot from the whole site (line 169: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does dailymail.com block Google-Extended?

Yes. dailymail.com’s robots.txt disallows Google-Extended from the whole site (line 169: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does dailymail.com block PerplexityBot?

Yes. dailymail.com’s robots.txt disallows PerplexityBot from the whole site (line 169: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find dailymail.com?

Not through its own crawler. dailymail.com’s robots.txt disallows OAI-SearchBot from the whole site (line 169: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.