Who blocks AI · #670 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does newscientist.com block AI crawlers?

Yes. newscientist.com’s robots.txt blocks all 19 AI crawlers we track, from GPTBot and ClaudeBot to PerplexityBot. It also blocks PetalBot among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: www.newscientist.com/robots.txt · 214 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Blocked	line 88 · `Disallow: /` under `User-agent: PetalBot`
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 88 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 88 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Blocked	line 88 · `Disallow: /` under `User-agent: Claude-SearchBot`
`DuckAssistBot`	DuckDuckGo	Blocked	line 88 · `Disallow: /` under `User-agent: DuckAssistBot`
`Amazonbot`	Amazon	Blocked	line 88 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Blocked	line 88 · `Disallow: /` under `User-agent: YouBot`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 88 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 88 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Blocked	line 88 · `Disallow: /` under `User-agent: Google-Extended`
`CCBot`	Common Crawl	Blocked	line 88 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 88 · `Disallow: /` under `User-agent: Applebot-Extended`
`GoogleOther`	Google	Blocked	line 88 · `Disallow: /` under `User-agent: GoogleOther`
`Meta-ExternalAgent`	Meta	Blocked	line 88 · `Disallow: /` under `User-agent: Meta-ExternalAgent`
`Bytespider`	ByteDance	Blocked	line 88 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 88 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Blocked	line 88 · `Disallow: /` under `User-agent: Claude-User`
`Perplexity-User`	Perplexity	Blocked	line 88 · `Disallow: /` under `User-agent: Perplexity-User`
`MistralAI-User`	Mistral	Blocked	line 88 · `Disallow: /` under `User-agent: MistralAI-User`
`Meta-ExternalFetcher`	Meta	Blocked	line 88 · `Disallow: /` under `User-agent: Meta-ExternalFetcher`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 88 · `Disallow: /` under `User-agent: AhrefsBot`
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View www.newscientist.com/robots.txt · 214 lines

1# Robots.txt for https://www.newscientist.com/ updated 04/11/2026

2# New Scientist content is made available for your personal and non-commercial

3# use subject to our Terms and Conditions of use:

4# https://www.newscientist.com/terms/

5# Use of any device, tool, or process designed to data mine or scrape the content

6# using automated means is prohibited without prior written permission from

7# NEW SCIENTIST LIMITED. Prohibited uses include but are not limited to:

8# (1) text and data mining activities under Art. 4 of the EU Directive on Copyright in

9# the Digital Single Market;

10# (2) the development of any software, machine learning, artificial intelligence (AI),

11# and/or large language models (LLMs);

12# (3) creating or providing archived or cached data sets containing our content to others; and/or

13# (4) any commercial purposes.

14# For any licensing queries please contact partnerships@dmgmedia.co.uk.

15#

16#

18User-agent: AhrefsBot

19User-agent: AI2Bot

20User-agent: Ai2Bot-Dolma

21User-agent: Amazonbot

22User-agent: amazon-kendra-

23User-agent: anthropic-ai

24User-agent: Applebot-Extended

25User-agent: bedrockbot

26User-agent: Bytespider

27User-agent: CloudflareBrowserRenderingCrawler

28User-agent: CCBot

29User-agent: ChatGLM-Spider

30User-agent: ChatGPT-User

31User-agent: ClaudeBot

32User-agent: Claude-SearchBot

33User-agent: Claude-User

34User-agent: Claude-Web

35User-agent: cohere-ai

36User-agent: cohere-training-data-crawler

37User-agent: Cotoyogi

38User-agent: DeepSeekBot

39User-agent: Diffbot

40User-agent: DuckAssistBot

41User-agent: EchoboxBot

42User-agent: FacebookBot

43User-agent: FriendlyCrawler

44User-agent: Gemini-Deep-Research

45User-agent: Google-CloudVertexBot

46User-agent: Google-Extended

47User-agent: GoogleOther

48User-agent: GoogleOther-Image

49User-agent: GoogleOther-Video

50User-agent: GPTBot

51User-agent: GPTBot-Search

52User-agent: Grok

53User-agent: iaskspider/2.0

54User-agent: ICC-Crawler

55User-agent: ImagesiftBot

56User-agent: img2dataset

57User-agent: ISSCyberRiskCrawler

58User-agent: Kangaroo Bot

59User-agent: KunatoCrawler

60User-agent: LinerBot

61User-agent: Meltwater

62User-agent: Meta-ExternalAgent

63User-agent: Meta-ExternalFetcher

64User-agent: Meta-WebIndexer

65User-agent: MistralAI-User

66User-agent: OAI-Operator

67User-agent: OAI-SearchBot

68User-agent: omgili

69User-agent: omgilibot

70User-agent: PanguBot

71User-agent: PerplexityBot

72User-agent: Perplexity-User

73User-agent: PetalBot

74User-agent: QualifiedBot

75User-agent: Scrapy

76User-agent: Seekr

77User-agent: Sidetrade indexer bot

78User-agent: TaraGroup Intelligent Bot

79User-agent: TikTokSpider

80User-agent: Timpibot

81User-agent: VelenPublicWebCrawler

82User-agent: WARDBot

83User-agent: Webzio-Extended

84User-agent: wpbot

85User-agent: WRTNBot

86User-agent: xAI-SearchBot

87User-agent: YouBot

88Disallow: /

90User-agent: *

91Disallow: /21632812681/

92Disallow: /activate-subscription/

93Disallow: /feed/

94Disallow: /login/

95Disallow: /logout/

96Disallow: /lost-password/

97Disallow: /my-account/

98Disallow: /registration/

99Disallow: /search/

100Disallow: /upfront/

101Disallow: /wp-admin/

102Disallow: /api/

103Disallow: /build/

104Disallow: /nsj/jobsjson/

105Disallow: /nsj/logon/

106Disallow: /nsj/newalert/

107Disallow: /nsj/analytics/

108Disallow: /nsj/apply-profile/

109Disallow: /nsj/document/

110Disallow: /nsj/emailjob/

111Disallow: /nsj/external-redirect-registration/

112Disallow: /nsj/invalid-request/

113Disallow: /nsj/jbequicksignup/

114Disallow: /nsj/jobsrss/

115Disallow: /nsj/previewjob/

116Disallow: /nsj/searchjobs/

117Disallow: /nsj/session-img/

118Disallow: /nsj/your-jobs/

119Disallow: /nsj/profile/

120Disallow: /nsj/remindme/

121Disallow: /nsj/searchjobs/

122#

123

124User-agent: archive-it.org

125User-agent: archive.org_bot

126User-agent: ArchiveBot

127User-agent: Arquivo-web-crawler

128User-agent: Authory

129User-agent: babel

130User-agent: bl.uk_ldfc_bot

131User-agent: bl.uk_ldfc_renderbot

132User-agent: bne.es_bot

133User-agent: bnf.fr_bot

134User-agent: ecoresearch

135User-agent: ecoresearchCrawler

136User-agent: Europarchive.org

137User-agent: ExaSearchBot

138User-agent: kb.dk_bot

139User-agent: Kulturarw3

140User-agent: KeenableBot

141User-agent: LivelapBot

142User-agent: mediacloud

143User-agent: mirrorweb

144User-agent: MSIECrawler

145User-agent: nationalarchives.gov.uk

146User-agent: netarkivet.dk

147User-agent: netEstate NE Crawler

148User-agent: Nicecrawler

149User-agent: nl-israel_iaharvester2026

150User-agent: nlnbot

151User-agent: nlncrawler

152User-agent: PageFreezer

153User-agent: SafeDNS

154User-agent: SmarshBot

155User-agent: special_archiver

156User-agent: tkl.iis.u-tokyo.ac.jp

157User-agent: TurnitinBot

158User-agent: webarchiv.cz

159User-agent: WordPress.com mShots

160User-agent: XY-Archive-Compliance-Archiver

161User-agent: XY-Archive-Compliance-Crawler

162Disallow: /

163

164Sitemap: https://www.newscientist.com/sitemap.xml

165Sitemap: https://www.newscientist.com/nsj/sitemapindex.xml# Sitemap archive

166Sitemap: https://www.newscientist.com/sitemap-1987.xml

167Sitemap: https://www.newscientist.com/sitemap-1988.xml

168Sitemap: https://www.newscientist.com/sitemap-1989.xml

169Sitemap: https://www.newscientist.com/sitemap-1990.xml

170Sitemap: https://www.newscientist.com/sitemap-1991.xml

171Sitemap: https://www.newscientist.com/sitemap-1992.xml

172Sitemap: https://www.newscientist.com/sitemap-1993.xml

173Sitemap: https://www.newscientist.com/sitemap-1994.xml

174Sitemap: https://www.newscientist.com/sitemap-1995.xml

175Sitemap: https://www.newscientist.com/sitemap-1996.xml

176Sitemap: https://www.newscientist.com/sitemap-1997.xml

177Sitemap: https://www.newscientist.com/sitemap-1998.xml

178Sitemap: https://www.newscientist.com/sitemap-1999.xml

179Sitemap: https://www.newscientist.com/sitemap-2000.xml

180Sitemap: https://www.newscientist.com/sitemap-2001.xml

181Sitemap: https://www.newscientist.com/sitemap-2002.xml

182Sitemap: https://www.newscientist.com/sitemap-2003.xml

183Sitemap: https://www.newscientist.com/sitemap-2004.xml

184Sitemap: https://www.newscientist.com/sitemap-2005.xml

185Sitemap: https://www.newscientist.com/sitemap-2006.xml

186Sitemap: https://www.newscientist.com/sitemap-2007.xml

187Sitemap: https://www.newscientist.com/sitemap-2008.xml

188Sitemap: https://www.newscientist.com/sitemap-2009.xml

189Sitemap: https://www.newscientist.com/sitemap-2010.xml

190Sitemap: https://www.newscientist.com/sitemap-2011.xml

191Sitemap: https://www.newscientist.com/sitemap-2012.xml

192Sitemap: https://www.newscientist.com/sitemap-2013.xml

193Sitemap: https://www.newscientist.com/sitemap-2014.xml

194Sitemap: https://www.newscientist.com/sitemap-2015.xml

195Sitemap: https://www.newscientist.com/sitemap-2016.xml

196Sitemap: https://www.newscientist.com/sitemap-2017.xml

197Sitemap: https://www.newscientist.com/sitemap-2018.xml

198Sitemap: https://www.newscientist.com/sitemap-2019.xml

199Sitemap: https://www.newscientist.com/sitemap-2020.xml

200Sitemap: https://www.newscientist.com/sitemap-2021.xml

201Sitemap: https://www.newscientist.com/sitemap-2022.xml

202Sitemap: https://www.newscientist.com/sitemap-2023.xml

203Sitemap: https://www.newscientist.com/sitemap-2024.xml

204Sitemap: https://www.newscientist.com/sitemap-2025.xml

205Sitemap: https://www.newscientist.com/sitemap-2026.xml

206

207

208# START YOAST BLOCK

209# ---------------------------

210User-agent: *

211Disallow:

212

213# ---------------------------

214# END YOAST BLOCK

What do the bots actually get?

This page reads robots.txt, which states what newscientist.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on newscientist.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does newscientist.com block GPTBot?

Yes. newscientist.com’s robots.txt disallows GPTBot from the whole site (line 88: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does newscientist.com block ClaudeBot?

Yes. newscientist.com’s robots.txt disallows ClaudeBot from the whole site (line 88: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does newscientist.com block Google-Extended?

Yes. newscientist.com’s robots.txt disallows Google-Extended from the whole site (line 88: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does newscientist.com block PerplexityBot?

Yes. newscientist.com’s robots.txt disallows PerplexityBot from the whole site (line 88: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find newscientist.com?

Not through its own crawler. newscientist.com’s robots.txt disallows OAI-SearchBot from the whole site (line 88: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.