Who blocks AI · #924 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does theregister.com block AI crawlers?

Yes. theregister.com’s robots.txt blocks 14 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 11 more. It also blocks PetalBot among search engines.

GPTBotBlocked OAI-SearchBotAllowed ClaudeBotBlocked PerplexityBotAllowed Google-ExtendedBlocked CCBotBlocked

robots.txt: www.theregister.com/robots.txt · 304 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	line 24 · `Allow: /` under `User-agent: Googlebot`
`Bingbot`	Microsoft	Allowed	line 27 · `Allow: /` under `User-agent: Bingbot`
`DuckDuckBot`	DuckDuckGo	Allowed	line 33 · `Allow: /` under `User-agent: DuckDuckBot`
`Applebot`	Apple	Allowed	line 42 · `Allow: /` under `User-agent: Applebot`
`PetalBot`	Huawei	Blocked	line 298 · `Disallow: /` under `User-agent: *`
`YandexBot`	Yandex	Allowed	line 39 · `Allow: /` under `User-agent: YandexBot`
`Baiduspider`	Baidu	Allowed	line 36 · `Allow: /` under `User-agent: Baiduspider`

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Allowed	line 51 · `Allow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Allowed	line 57 · `Allow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Allowed	line 66 · `Allow: /` under `User-agent: Claude-SearchBot`
`DuckAssistBot`	DuckDuckGo	Blocked	line 252 · `Disallow: /` under `User-agent: DuckAssistBot`
`Amazonbot`	Amazon	Blocked	line 228 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Blocked	line 248 · `Disallow: /` under `User-agent: YouBot`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 192 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 197 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Blocked	line 207 · `Disallow: /` under `User-agent: Google-Extended`
`CCBot`	Common Crawl	Blocked	line 236 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 224 · `Disallow: /` under `User-agent: Applebot-Extended`
`GoogleOther`	Google	Blocked	line 298 · `Disallow: /` under `User-agent: *`
`Meta-ExternalAgent`	Meta	Blocked	line 214 · `Disallow: /` under `User-agent: Meta-ExternalAgent`
`Bytespider`	ByteDance	Blocked	line 232 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Allowed	line 54 · `Allow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Allowed	line 63 · `Allow: /` under `User-agent: Claude-User`
`Perplexity-User`	Perplexity	Blocked	line 298 · `Disallow: /` under `User-agent: *`
`MistralAI-User`	Mistral	Blocked	line 298 · `Disallow: /` under `User-agent: *`
`Meta-ExternalFetcher`	Meta	Blocked	line 217 · `Disallow: /` under `User-agent: Meta-ExternalFetcher`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 298 · `Disallow: /` under `User-agent: *`
`SemrushBot`	Semrush	Blocked	line 298 · `Disallow: /` under `User-agent: *`
`MJ12bot`	Majestic	Blocked	line 298 · `Disallow: /` under `User-agent: *`
`DotBot`	Moz	Blocked	line 298 · `Disallow: /` under `User-agent: *`

View www.theregister.com/robots.txt · 304 lines

1# =============================================================================

2# robots.txt

3# Last updated: 18 June 2026

5# POLICY: Default-deny. Any crawler not explicitly allowed below is BLOCKED.

6# - Traditional search engines ............ ALLOWED

7# - AI search crawlers that cite + refer ... ALLOWED (conditional on attribution)

8# - Operational bots (ads, social, uptime) . ALLOWED

9# - AI training / scraping crawlers ........ BLOCKED (licence required)

10# - Anything else, including unknown bots .. BLOCKED by default

11#

12#

13# AI companies: to licence our content, see https://situationpublishing.com/ai-licensing/

14# Licensing enquiries: content-licensing@sitpub.com

15#

16# If you are an AI company and want to use our content, contact us about

17# a licensing arrangement. See: https://situationpublishing.com/ai-licensing/

18# =============================================================================

20# ---------------------------------------------------------------------------

21# Traditional search engines — ALLOWED

22# ---------------------------------------------------------------------------

23User-agent: Googlebot

24Allow: /

26User-agent: Bingbot

27Allow: /

29User-agent: Slurp

30Allow: /

32User-agent: DuckDuckBot

33Allow: /

35User-agent: Baiduspider

36Allow: /

38User-agent: YandexBot

39Allow: /

41User-agent: Applebot

42Allow: /

45# ---------------------------------------------------------------------------

46# AI search crawlers that cite and return referral traffic — ALLOWED

47# These power AI search answers and link back to source content.

48# Permission is conditional on continued referral traffic and attribution.

49# ---------------------------------------------------------------------------

50User-agent: OAI-SearchBot

51Allow: /

53User-agent: ChatGPT-User

54Allow: /

56User-agent: PerplexityBot

57Allow: /

59# Anthropic (Claude). User-initiated fetches, search indexing, and the

60# Claude Code CLI fetching on a developer's behalf. All three honour

61# robots.txt. ClaudeBot (training) stays BLOCKED below, mirroring GPTBot.

62User-agent: Claude-User

63Allow: /

65User-agent: Claude-SearchBot

66Allow: /

68User-agent: claude-code

69Allow: /

72# ---------------------------------------------------------------------------

73# Operational bots we depend on — ALLOWED

74# DO NOT REMOVE. Blocking these breaks programmatic revenue and social sharing.

75# ---------------------------------------------------------------------------

77# Google advertising crawlers (ad serving / quality)

78User-agent: AdsBot-Google

79Allow: /

81User-agent: AdsBot-Google-Mobile

82Allow: /

84User-agent: Mediapartners-Google

85Allow: /

87# Social link-preview / unfurl bots

88User-agent: facebookexternalhit

89Allow: /

91User-agent: Twitterbot

92Allow: /

94User-agent: LinkedInBot

95Allow: /

97User-agent: Pinterestbot

98Allow: /

100User-agent: Slackbot

101Allow: /

102

103User-agent: Slackbot-LinkExpanding

104Allow: /

105

106User-agent: Discordbot

107Allow: /

108

109User-agent: TelegramBot

110Allow: /

111

112User-agent: WhatsApp

113Allow: /

114

115# Ad verification / brand safety / contextual vendors — ALLOWED

116# Clients use different verification partners, so all the major ones are listed.

117# Ad ops: review periodically and add any niche vendor a client requires.

118# (UA matching is case-insensitive.)

119

120# Integral Ad Science (IAS)

121User-agent: ias_crawler

122Allow: /

123

124User-agent: ias_wombles

125Allow: /

126

127User-agent: ias_admantx

128Allow: /

129

130# DoubleVerify

131User-agent: DoubleVerifyBot

132Allow: /

133

134User-agent: DVBot

135Allow: /

136

137# Comscore (contextual)

138User-agent: proximic

139Allow: /

140

141# Peer39 (contextual)

142User-agent: peer39_crawler

143Allow: /

144

145# Oracle / Grapeshot (contextual — winding down, kept for clients still on it)

146User-agent: GrapeshotCrawler

147Allow: /

148

149User-agent: grapeshot

150Allow: /

151

152# GumGum (contextual)

153User-agent: GumGum

154Allow: /

155

156# HUMAN Security (brand safety)

157User-agent: GlancelotBot

158Allow: /

159

160# The Trade Desk

161User-agent: TTD-Content

162Allow: /

163

164# PubMatic

165User-agent: PubMatic Crawler Bot

166Allow: /

167

168# Sincera (supply-chain transparency)

169User-agent: SinceraSyntheticUser

170Allow: /

171

172# Amazon advertising (distinct from Amazonbot AI-training crawler, blocked below)

173User-agent: AmazonAdBot

174Allow: /

175

176# Other brand-safety / ad-tech crawlers

177User-agent: Leikibot

178Allow: /

179

180User-agent: SlickBot

181Allow: /

182

183

184# ---------------------------------------------------------------------------

185# AI training / scraping crawlers — BLOCKED (belt-and-suspenders)

186# These are already blocked by the default-deny rule at the bottom; listed

187# explicitly to document intent. Contact us for licensing terms.

188# ---------------------------------------------------------------------------

189

190# OpenAI (training)

191User-agent: GPTBot

192Disallow: /

193

194# Anthropic (training only — Claude-User, Claude-SearchBot and claude-code

195# are allowed above)

196User-agent: ClaudeBot

197Disallow: /

198

199User-agent: anthropic-ai

200Disallow: /

201

202User-agent: Claude-Web

203Disallow: /

204

205# Google (AI training — separate from Googlebot search indexing)

206User-agent: Google-Extended

207Disallow: /

208

209User-agent: Google-CloudVertexBot

210Disallow: /

211

212# Meta

213User-agent: Meta-ExternalAgent

214Disallow: /

215

216User-agent: Meta-ExternalFetcher

217Disallow: /

218

219User-agent: FacebookBot

220Disallow: /

221

222# Apple (AI training — separate from standard Applebot)

223User-agent: Applebot-Extended

224Disallow: /

225

226# Amazon

227User-agent: Amazonbot

228Disallow: /

229

230# ByteDance / TikTok

231User-agent: Bytespider

232Disallow: /

233

234# Common Crawl

235User-agent: CCBot

236Disallow: /

237

238# Cohere

239User-agent: cohere-ai

240Disallow: /

241

242# Diffbot

243User-agent: Diffbot

244Disallow: /

245

246# You.com

247User-agent: YouBot

248Disallow: /

249

250# DuckDuckGo AI assistant (distinct from DuckDuckBot search)

251User-agent: DuckAssistBot

252Disallow: /

253

254# Timpi

255User-agent: Timpibot

256Disallow: /

257

258# Omgili / Webz.io

259User-agent: Omgilibot

260Disallow: /

261

262User-agent: Omgili

263Disallow: /

264

265# ImagesiftBot

266User-agent: ImagesiftBot

267Disallow: /

268

269# xAI / Grok

270User-agent: GrokBot

271Disallow: /

272

273# Allen Institute for AI (AI2)

274User-agent: AI2Bot

275Disallow: /

276

277# Firecrawl (dev-tool scraper)

278User-agent: FirecrawlAgent

279Disallow: /

280

281# Scrapy-based generic crawlers

282User-agent: Scrapy

283Disallow: /

284

285

286# ---------------------------------------------------------------------------

287# DEFAULT — everything else is BLOCKED

288# Any crawler not explicitly allowed above is denied. Review server logs /

289# TollBit WEEKLY and add new beneficial crawlers (a new AI search engine, or a

290# new ad-verification vendor a client brings) to the ALLOWED sections above.

291#

292# EXCEPTION: ads.txt and app-ads.txt must stay readable by ALL bots —

293# programmatic demand depends on it — so they are allowed even under default-deny.

294# ---------------------------------------------------------------------------

295User-agent: *

296Allow: /ads.txt

297Allow: /app-ads.txt

298Disallow: /

299

300

301# ---------------------------------------------------------------------------

302# Sitemaps

303# ---------------------------------------------------------------------------

304Sitemap: https://www.theregister.com/sitemapindex.xml

What do the bots actually get?

This page reads robots.txt, which states what theregister.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on theregister.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does theregister.com block GPTBot?

Yes. theregister.com’s robots.txt disallows GPTBot from the whole site (line 192: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does theregister.com block ClaudeBot?

Yes. theregister.com’s robots.txt disallows ClaudeBot from the whole site (line 197: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does theregister.com block Google-Extended?

Yes. theregister.com’s robots.txt disallows Google-Extended from the whole site (line 207: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does theregister.com block PerplexityBot?

No. theregister.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find theregister.com?

Yes. theregister.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.