Who blocks AI · #95 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does bbc.com block AI crawlers?

Yes. bbc.com’s robots.txt blocks 15 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 12 more. It also blocks PetalBot among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: www.bbc.com/robots.txt · 244 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Blocked	line 145 · `Disallow: /` under `User-agent: PetalBot`
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 169 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 174 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Blocked	line 240 · `Disallow: /` under `User-agent: Claude-SearchBot`
`DuckAssistBot`	DuckDuckGo	Blocked	line 243 · `Disallow: /` under `User-agent: DuckAssistBot`
`Amazonbot`	Amazon	Blocked	line 115 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Blocked	line 219 · `Disallow: /` under `User-agent: YouBot`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 154 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 133 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Blocked	line 164 · `Disallow: /` under `User-agent: Google-Extended`
`CCBot`	Common Crawl	Blocked	line 121 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 151 · `Disallow: /` under `User-agent: Applebot-Extended`
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Blocked	line 183 · `Disallow: /` under `User-agent: meta-externalagent`
`Bytespider`	ByteDance	Blocked	line 142 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 159 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Allowed	no rule matches the home page
`Perplexity-User`	Perplexity	Blocked	line 177 · `Disallow: /` under `User-agent: Perplexity-User`
`MistralAI-User`	Mistral	Allowed	no rule matches the home page
`Meta-ExternalFetcher`	Meta	Allowed	no rule matches the home page

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 237 · `Disallow: /` under `User-agent: AhrefsBot`
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View www.bbc.com/robots.txt · 244 lines

2# version: 2323e66fcb22a9568be0b7ab535fafcd2bf666ad

3# The BBC's Terms of Use: https://www.bbc.co.uk/terms

4# - Explain the rules for using our services

5# - Tell you what you can do with our content

7# In short: Please use our site like a human, not a robot.

8# That means:

9# - No scraping, crawling, or systematic extraction of content

10# - No use of BBC content for training or fine-tuning AI models, including large language models (LLMs)

11# - No retrieval-augmented generation (RAG), AI-powered search, agentic AI or grounding using BBC content

12# - No creating datasets from BBC content

13# - No text and data mining (TDM) under Article 4 of the EU Directive on Copyright in the Digital Single Market

14# - No using BBC content to create summaries for your own use

15# - No business use without permission (details: https://www.bbc.co.uk/usingthebbc/terms/can-i-use-bbc-content-for-my-business/)

16# - The BBC reserves all rights in its content and expressly opts out of any statutory exceptions in any jurisdiction for text and data mining, as permitted by law

18# TL;DR: Browse, read, watch, enjoy - like a human.

19#

21# HTTPS www.bbc.com

23User-agent: *

24Sitemap: https://www.bbc.com/afrique/sitemap.xml

25Sitemap: https://www.bbc.com/arabic/sitemap.xml

26Sitemap: https://www.bbc.com/bbcx/audio_archive_sitemap.xml

27Sitemap: https://www.bbc.com/bbcx/content_index_sitemap.xml

28Sitemap: https://www.bbc.com/bbcx/index_sitemap.xml

29Sitemap: https://www.bbc.com/bbcx/video_documentaries_sitemap.xml

30Sitemap: https://www.bbc.com/bengali/sitemap.xml

31Sitemap: https://www.bbc.com/burmese/sitemap.xml

32Sitemap: https://www.bbc.com/gahuza/sitemap.xml

33Sitemap: https://www.bbc.com/hausa/sitemap.xml

34Sitemap: https://www.bbc.com/hindi/sitemap.xml

35Sitemap: https://www.bbc.com/indonesia/sitemap.xml

36Sitemap: https://www.bbc.com/mundo/sitemap.xml

37Sitemap: https://www.bbc.com/pashto/sitemap.xml

38Sitemap: https://www.bbc.com/persian/sitemap.xml

39Sitemap: https://www.bbc.com/portuguese/sitemap.xml

40Sitemap: https://www.bbc.com/russian/sitemap.xml

41Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive_video.xml

42Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive.xml

43Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-archive_video.xml

44Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-archive.xml

45Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-news.xml

46Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-video.xml

47Sitemap: https://www.bbc.com/sitemaps/https-index-com-news.xml

48Sitemap: https://www.bbc.com/sitemaps/https-index-com-video.xml

49Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml

50Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml

51Sitemap: https://www.bbc.com/sitemaps/world-service-homepages.xml

52Sitemap: https://www.bbc.com/sport/sitemap.xml

53Sitemap: https://www.bbc.com/swahili/sitemap.xml

54Sitemap: https://www.bbc.com/tajik/sitemap.xml

55Sitemap: https://www.bbc.com/turkce/sitemap.xml

56Sitemap: https://www.bbc.com/ukchina/simp/sitemap.xml

57Sitemap: https://www.bbc.com/ukrainian/sitemap.xml

58Sitemap: https://www.bbc.com/urdu/sitemap.xml

59Sitemap: https://www.bbc.com/uzbek/sitemap.xml

60Sitemap: https://www.bbc.com/vietnamese/sitemap.xml

61Sitemap: https://www.bbc.com/zhongwen/simp/sitemap.xml

62Sitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml

64Disallow: /asset/

65Disallow: /backstage/bbc-login-help/

66Disallow: /backstage/bbc-login-help$

67Disallow: /bitesize/search$

68Disallow: /bitesize/search/

69Disallow: /bitesize/search?

70Disallow: /cbbc/search/

71Disallow: /cbbc/search$

72Disallow: /cbbc/search?

73Disallow: /cbeebies/search/

74Disallow: /cbeebies/search$

75Disallow: /cbeebies/search?

76Disallow: /chwilio/

77Disallow: /chwilio$

78Disallow: /chwilio?

79Disallow: /education/blocks$

80Disallow: /education/blocks/

81Disallow: /newsround

82Disallow: /search/

83Disallow: /search$

84Disallow: /search?

85Disallow: /food/favourites

86Disallow: /food/search*?*

87Disallow: /food/recipes/search*?*

88Disallow: /education/my$

89Disallow: /education/my/

90Disallow: /bitesize/my$

91Disallow: /bitesize/my/

92Disallow: /food/recipes/*/shopping-list

93Disallow: /food/menus/*/shopping-list

94Disallow: /news/0

95Disallow: /sport/alpha/

96Disallow: /ugc$

97Disallow: /ugc/

98Disallow: /ugcsupport$

99Disallow: /ugcsupport/

100Disallow: /userinfo/

101Disallow: /userinfo

102Disallow: /u5llnop$

103Disallow: /u5llnop/

104Disallow: /sounds/search$

105Disallow: /sounds/search/

106Disallow: /sounds/search?

107Disallow: /ws/includes

108Disallow: /radio/imda

109Disallow: /storyworks/preview/*

110Disallow: /rd/search$

111Disallow: /rd/search/

112Disallow: /rd/search?

113

114User-agent: Amazonbot

115Disallow: /

116

117User-agent: magpie-crawler

118Disallow: /

119

120User-agent: CCBot

121Disallow: /

122

123User-Agent: omgili

124Disallow: /

125

126User-Agent: omgilibot

127Disallow: /

128

129User-agent: Claude-Web

130Disallow: /

131

132User-agent: ClaudeBot

133Disallow: /

134

135User-agent: anthropic-ai

136Disallow: /

137

138User-agent: cohere-ai

139Disallow: /

140

141User-agent: Bytespider

142Disallow: /

143

144User-agent: PetalBot

145Disallow: /

146

147User-agent: Scrapy

148Disallow: /

149

150User-agent: Applebot-Extended

151Disallow: /

152

153User-agent: GPTBot

154Disallow: /

155Allow: /storyworks

156Allow: /storyworks/

157

158User-agent: ChatGPT-User

159Disallow: /

160Allow: /storyworks

161Allow: /storyworks/

162

163User-agent: Google-Extended

164Disallow: /

165Allow: /storyworks

166Allow: /storyworks/

167

168User-agent: OAI-SearchBot

169Disallow: /

170Allow: /storyworks

171Allow: /storyworks/

172

173User-Agent: PerplexityBot

174Disallow: /

175

176User-agent: Perplexity-User

177Disallow: /

178

179User-agent: Google-CloudVertexBot

180Disallow: /

181

182User-agent: meta-externalagent

183Disallow: /

184

185User-agent: YandexAdditional

186Disallow: /

187

188User-agent: YandexAdditionalBot

189Disallow: /

190

191User-agent: TurnitinBot

192Disallow: /

193

194User-agent: Brightbot

195Disallow: /

196

197User-agent: ApifyBot

198Disallow: /

199

200User-agent: ApifyWebsiteContentCrawler

201Disallow: /

202

203User-agent: Diffbot

204Disallow: /

205

206User-agent: Diffbot-User

207Disallow: /

208

209User-agent: ExaBot

210Disallow: /

211

212User-agent: TavilyBot

213Disallow: /

214

215User-agent: ShapBot

216Disallow: /

217

218User-agent: YouBot

219Disallow: /

220

221User-agent: FirecrawlAgent

222Disallow: /

223

224User-agent: Amzn-SearchBot

225Disallow: /

226

227User-agent: Amzn-User

228Disallow: /

229

230User-agent: ProRataInc

231Disallow: /

232

233User-agent: CloudflareBrowserRenderingCrawler

234Disallow: /

235

236User-agent: AhrefsBot

237Disallow: /

238

239User-agent: Claude-SearchBot

240Disallow: /

241

242User-agent: DuckAssistBot

243Disallow: /

244

What do the bots actually get?

This page reads robots.txt, which states what bbc.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on bbc.com Test a path in the robots.txt tester

History

19 October 2026: first checked.
2 November 2026: started blocking Claude-SearchBot; started blocking DuckAssistBot.

Questions

Does bbc.com block GPTBot?

Yes. bbc.com’s robots.txt disallows GPTBot from the whole site (line 154: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does bbc.com block ClaudeBot?

Yes. bbc.com’s robots.txt disallows ClaudeBot from the whole site (line 133: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does bbc.com block Google-Extended?

Yes. bbc.com’s robots.txt disallows Google-Extended from the whole site (line 164: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does bbc.com block PerplexityBot?

Yes. bbc.com’s robots.txt disallows PerplexityBot from the whole site (line 174: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find bbc.com?

Not through its own crawler. bbc.com’s robots.txt disallows OAI-SearchBot from the whole site (line 169: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.