Who blocks AI · #109 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does bbc.co.uk block AI crawlers?
Yes. bbc.co.uk’s robots.txt blocks 15 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 12 more. It also blocks PetalBot among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.bbc.co.uk/robots.txt · 218 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 127 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 157 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 148 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 214 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 217 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 97 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 193 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 136 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 112 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 145 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 103 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 133 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 154 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 124 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 139 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Blocked | line 151 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 211 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.bbc.co.uk/robots.txt · 218 lines
1
2# version: 2323e66fcb22a9568be0b7ab535fafcd2bf666ad
3# The BBC's Terms of Use: https://www.bbc.co.uk/terms
4# - Explain the rules for using our services
5# - Tell you what you can do with our content
6#
7# In short: Please use our site like a human, not a robot.
8# That means:
9# - No scraping, crawling, or systematic extraction of content
10# - No use of BBC content for training or fine-tuning AI models, including large language models (LLMs)
11# - No retrieval-augmented generation (RAG), AI-powered search, agentic AI or grounding using BBC content
12# - No creating datasets from BBC content
13# - No text and data mining (TDM) under Article 4 of the EU Directive on Copyright in the Digital Single Market
14# - No using BBC content to create summaries for your own use
15# - No business use without permission (details: https://www.bbc.co.uk/usingthebbc/terms/can-i-use-bbc-content-for-my-business/)
16# - The BBC reserves all rights in its content and expressly opts out of any statutory exceptions in any jurisdiction for text and data mining, as permitted by law
17
18# TL;DR: Browse, read, watch, enjoy - like a human.
19#
20
21# HTTPS www.bbc.co.uk
22
23User-agent: *
24Sitemap: https://www.bbc.co.uk/sitemap.xml
25Sitemap: https://www.bbc.co.uk/sitemaps/https-index-uk-archive.xml
26Sitemap: https://www.bbc.co.uk/sitemaps/https-index-uk-news.xml
27Sitemap: https://www.bbc.co.uk/food/sitemap.xml
28Sitemap: https://www.bbc.co.uk/bitesize/sitemap/sitemapindex.xml
29Sitemap: https://www.bbc.co.uk/teach/sitemap/sitemapindex.xml
30Sitemap: https://www.bbc.co.uk/sitemaps/https-index-uk-archive_video.xml
31Sitemap: https://www.bbc.co.uk/sitemaps/https-index-uk-video.xml
32Sitemap: https://www.bbc.co.uk/sitemaps/sitemap-uk-ws-topics.xml
33Sitemap: https://www.bbc.co.uk/sport/sitemap.xml
34Sitemap: https://www.bbc.co.uk/sitemaps/sitemap-uk-topics.xml
35Sitemap: https://www.bbc.co.uk/ideas/sitemap.xml
36Sitemap: https://www.bbc.co.uk/tiny-happy-people/sitemap/sitemapindex.xml
37
38Disallow: /asset/
39Disallow: /backstage/bbc-login-help/
40Disallow: /backstage/bbc-login-help$
41Disallow: /bitesize/search$
42Disallow: /bitesize/search/
43Disallow: /bitesize/search?
44Disallow: /cbbc/search$
45Disallow: /cbbc/search/
46Disallow: /cbbc/search?
47Disallow: /cbeebies/search$
48Disallow: /cbeebies/search/
49Disallow: /cbeebies/search?
50Disallow: /chwilio/
51Disallow: /chwilio$
52Disallow: /chwilio?
53Disallow: /iplayer/bigscreen/
54Disallow: /iplayer/cbbc/episodes/
55Disallow: /iplayer/cbbc/search
56Disallow: /iplayer/cbeebies/episodes/
57Disallow: /iplayer/cbeebies/search
58Disallow: /iplayer/search
59Disallow: /indepthtoolkit/smallprox$
60Disallow: /indepthtoolkit/smallprox/
61Disallow: /moderation/reports/
62Disallow: /modules/musicnav/language/
63Disallow: /news/0
64Disallow: /radio/aod/
65Disallow: /radio/aod$
66Disallow: /radio/imda
67Disallow: /radio/player/
68Disallow: /radio/player$
69Disallow: /search/
70Disallow: /search$
71Disallow: /search?
72Disallow: /sport/alpha/
73Disallow: /sounds/player/
74Disallow: /sounds/player$
75Disallow: /ugc$
76Disallow: /ugc/
77Disallow: /ugcsupport$
78Disallow: /ugcsupport/
79Disallow: /userinfo/
80Disallow: /userinfo
81Disallow: /food/favourites
82Disallow: /food/menus/*/shopping-list
83Disallow: /food/recipes/*/shopping-list
84Disallow: /food/search*?*
85Disallow: /sounds/search$
86Disallow: /sounds/search/
87Disallow: /sounds/search?
88Disallow: /ws/includes
89Disallow: /rd/search$
90Disallow: /rd/search/
91Disallow: /rd/search?
92Disallow: /things/search$
93Disallow: /things/search/
94Disallow: /things/search?
95
96User-agent: Amazonbot
97Disallow: /
98
99User-agent: magpie-crawler
100Disallow: /
101
102User-agent: CCBot
103Disallow: /
104
105User-Agent: omgili
106Disallow: /
107
108User-Agent: omgilibot
109Disallow: /
110
111User-agent: ClaudeBot
112Disallow: /
113
114User-agent: Claude-Web
115Disallow: /
116
117User-agent: anthropic-ai
118Disallow: /
119
120User-agent: cohere-ai
121Disallow: /
122
123User-agent: Bytespider
124Disallow: /
125
126User-agent: PetalBot
127Disallow: /
128
129User-agent: Scrapy
130Disallow: /
131
132User-agent: Applebot-Extended
133Disallow: /
134
135User-agent: GPTBot
136Disallow: /
137
138User-agent: ChatGPT-User
139Disallow: /
140
141User-agent: Google-CloudVertexBot
142Disallow: /
143
144User-agent: Google-Extended
145Disallow: /
146
147User-Agent: PerplexityBot
148Disallow: /
149
150User-agent: Perplexity-User
151Disallow: /
152
153User-agent: meta-externalagent
154Disallow: /
155
156User-agent: OAI-SearchBot
157Disallow: /
158
159User-agent: YandexAdditional
160Disallow: /
161
162User-agent: YandexAdditionalBot
163Disallow: /
164
165User-agent: TurnitinBot
166Disallow: /
167
168User-agent: Brightbot
169Disallow: /
170
171User-agent: ApifyBot
172Disallow: /
173
174User-agent: ApifyWebsiteContentCrawler
175Disallow: /
176
177User-agent: Diffbot
178Disallow: /
179
180User-agent: Diffbot-User
181Disallow: /
182
183User-agent: ExaBot
184Disallow: /
185
186User-agent: TavilyBot
187Disallow: /
188
189User-agent: ShapBot
190Disallow: /
191
192User-agent: YouBot
193Disallow: /
194
195User-agent: FirecrawlAgent
196Disallow:/
197
198User-agent: Amzn-SearchBot
199Disallow: /
200
201User-agent: Amzn-User
202Disallow: /
203
204User-agent: ProRataInc
205Disallow: /
206
207User-agent: CloudflareBrowserRenderingCrawler
208Disallow: /
209
210User-agent: AhrefsBot
211Disallow: /
212
213User-agent: Claude-SearchBot
214Disallow: /
215
216User-agent: DuckAssistBot
217Disallow: /
218
What do the bots actually get?
This page reads robots.txt, which states what bbc.co.uk asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on bbc.co.uk Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
- 2 November 2026: started blocking Claude-SearchBot; started blocking DuckAssistBot.
Questions
Does bbc.co.uk block GPTBot?
Yes. bbc.co.uk’s robots.txt disallows GPTBot from the whole site (line 136: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does bbc.co.uk block ClaudeBot?
Yes. bbc.co.uk’s robots.txt disallows ClaudeBot from the whole site (line 112: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does bbc.co.uk block Google-Extended?
Yes. bbc.co.uk’s robots.txt disallows Google-Extended from the whole site (line 145: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does bbc.co.uk block PerplexityBot?
Yes. bbc.co.uk’s robots.txt disallows PerplexityBot from the whole site (line 148: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find bbc.co.uk?
Not through its own crawler. bbc.co.uk’s robots.txt disallows OAI-SearchBot from the whole site (line 157: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.