Who blocks AI · #970 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does metro.co.uk block AI crawlers?
Yes. metro.co.uk’s robots.txt blocks all 19 AI crawlers we track, from GPTBot and ClaudeBot to PerplexityBot. It also blocks PetalBot among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- metro.co.uk/robots.txt · 179 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 145 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 145 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 145 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 145 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 145 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 145 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 145 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 145 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 145 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 145 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 145 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 145 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Blocked | line 145 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Blocked | line 145 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 145 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 145 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 145 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 145 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 145 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Blocked | line 145 · Disallow: / under User-agent: Meta-ExternalFetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 145 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View metro.co.uk/robots.txt · 179 lines
1# Robots.txt for https://metro.co.uk updated 04/11/2026
2#
3#
4# Metro content is made available for your personal and non-commercial
5# use subject to our Terms and Conditions of use:
6# https://metro.co.uk/terms/
7# Use of any device, tool, or process designed to data mine or scrape the content
8# using automated means is prohibited without prior written permission from
9# ASSOCIATED NEWSPAPERS LIMITED. Prohibited uses include but are not limited to:
10# (1) text and data mining activities under Art. 4 of the EU Directive on Copyright in
11# the Digital Single Market;
12# (2) the development of any software, machine learning, artificial intelligence (AI),
13# and/or large language models (LLMs);
14# (3) creating or providing archived or cached data sets containing our content to others; and/or
15# (4) any commercial purposes.
16# For any licensing queries please contact partnerships@dmgmedia.co.uk.
17#
18#
19User-agent: *
20Disallow: /wp-admin/
21Allow: /wp-admin/admin-ajax.php
22Disallow: /search/
23Disallow: /*.pdf
24Disallow: /post-style/*
25Disallow: /jobs-scrape/*
26Disallow: /*ia_markup=*
27Disallow: /dmg_liveblog/*
28Disallow: /wp-includes/*
29Disallow: /wp-content/*.js
30Disallow: /wp-content/*.css
31Disallow: /wp-json/*
32Disallow: /_static/*
33
34#
35User-agent: archive-it.org
36User-agent: archive.org_bot
37User-agent: ArchiveBot
38User-agent: Arquivo-web-crawler
39User-agent: Authory
40User-agent: babel
41User-agent: bl.uk_ldfc_bot
42User-agent: bl.uk_ldfc_renderbot
43User-agent: bne.es_bot
44User-agent: bnf.fr_bot
45User-agent: ecoresearch
46User-agent: ecoresearchCrawler
47User-agent: Europarchive.org
48User-agent: kb.dk_bot
49User-agent: Kulturarw3
50User-agent: LivelapBot
51User-agent: mediacloud
52User-agent: mirrorweb
53User-agent: MSIECrawler
54User-agent: nationalarchives.gov.uk
55User-agent: netarkivet.dk
56User-agent: netEstate NE Crawler
57User-agent: Nicecrawler
58User-agent: nl-israel_iaharvester2026
59User-agent: nlnbot
60User-agent: nlncrawler
61User-agent: PageFreezer
62User-agent: SafeDNS
63User-agent: SmarshBot
64User-agent: special_archiver
65User-agent: tkl.iis.u-tokyo.ac.jp
66User-agent: TurnitinBot
67User-agent: webarchiv.cz
68User-agent: WordPress.com mShots
69User-agent: XY-Archive-Compliance-Archiver
70User-agent: XY-Archive-Compliance-Crawler
71Disallow: /
72
73User-agent: AhrefsBot
74User-agent: AI2Bot
75User-agent: Ai2Bot-Dolma
76User-agent: Amazonbot
77User-agent: amazon-kendra-
78User-agent: anthropic-ai
79User-agent: Applebot-Extended
80User-agent: bedrockbot
81User-agent: Bytespider
82User-agent: CloudflareBrowserRenderingCrawler
83User-agent: CCBot
84User-agent: ChatGLM-Spider
85User-agent: ChatGPT-User
86User-agent: ClaudeBot
87User-agent: Claude-SearchBot
88User-agent: Claude-User
89User-agent: Claude-Web
90User-agent: cohere-ai
91User-agent: cohere-training-data-crawler
92User-agent: Cotoyogi
93User-agent: DeepSeekBot
94User-agent: Diffbot
95User-agent: DuckAssistBot
96User-agent: EchoboxBot
97User-agent: ExaSearchBot
98User-agent: FacebookBot
99User-agent: FriendlyCrawler
100User-agent: Gemini-Deep-Research
101User-agent: Google-CloudVertexBot
102User-agent: Google-Extended
103User-agent: GoogleOther
104User-agent: GoogleOther-Image
105User-agent: GoogleOther-Video
106User-agent: GPTBot
107User-agent: GPTBot-Search
108User-agent: Grok
109User-agent: iaskspider/2.0
110User-agent: ICC-Crawler
111User-agent: ImagesiftBot
112User-agent: img2dataset
113User-agent: ISSCyberRiskCrawler
114User-agent: Kangaroo Bot
115User-agent: KunatoCrawler
116User-agent: KeenableBot
117User-agent: LinerBot
118User-agent: Meltwater
119User-agent: Meta-ExternalAgent
120User-agent: Meta-ExternalFetcher
121User-agent: Meta-WebIndexer
122User-agent: MistralAI-User
123User-agent: OAI-Operator
124User-agent: OAI-SearchBot
125User-agent: omgili
126User-agent: omgilibot
127User-agent: PanguBot
128User-agent: PerplexityBot
129User-agent: Perplexity-User
130User-agent: PetalBot
131User-agent: QualifiedBot
132User-agent: Scrapy
133User-agent: Seekr
134User-agent: Sidetrade indexer bot
135User-agent: TaraGroup Intelligent Bot
136User-agent: TikTokSpider
137User-agent: Timpibot
138User-agent: VelenPublicWebCrawler
139User-agent: WARDBot
140User-agent: Webzio-Extended
141User-agent: wpbot
142User-agent: WRTNBot
143User-agent: xAI-SearchBot
144User-agent: YouBot
145Disallow: /
146
147User-agent: GoogleOther
148Allow: /.well-known/privacy-sandbox-attestations.json
149
150# Year Sitemap
151Sitemap: https://metro.co.uk/sitemap.xml?yyyy=2026
152Sitemap: https://metro.co.uk/sitemap.xml?yyyy=2025
153Sitemap: https://metro.co.uk/sitemap.xml?yyyy=2024
154Sitemap: https://metro.co.uk/sitemap.xml?yyyy=2023
155Sitemap: https://metro.co.uk/sitemap.xml?yyyy=2022
156Sitemap: https://metro.co.uk/sitemap.xml?yyyy=2021
157Sitemap: https://metro.co.uk/sitemap.xml?yyyy=2020
158
159# News Sitemap
160Sitemap: https://metro.co.uk/news-sitemap.xml
161
162# Sitemap archive
163Sitemap: https://metro.co.uk/sitemap.xml
164
165Sitemap: https://metro.co.uk/sitemap-channels.xml
166
167Sitemap: https://metro.co.uk/sitemap-tags.xml
168
169Sitemap: https://metro.co.uk/sitemap-authors.xml
170
171Sitemap: https://metro.co.uk/sitemap-videos.xml
172
173Sitemap: https://metro.co.uk/sitemap-tv-soaps.xml
174
175Sitemap: https://metro.co.uk/sitemap-images.xml
176# Sitemap archive
177Sitemap: https://metro.co.uk/sitemap.xml
178
179
What do the bots actually get?
This page reads robots.txt, which states what metro.co.uk asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on metro.co.uk Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does metro.co.uk block GPTBot?
Yes. metro.co.uk’s robots.txt disallows GPTBot from the whole site (line 145: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does metro.co.uk block ClaudeBot?
Yes. metro.co.uk’s robots.txt disallows ClaudeBot from the whole site (line 145: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does metro.co.uk block Google-Extended?
Yes. metro.co.uk’s robots.txt disallows Google-Extended from the whole site (line 145: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does metro.co.uk block PerplexityBot?
Yes. metro.co.uk’s robots.txt disallows PerplexityBot from the whole site (line 145: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find metro.co.uk?
Not through its own crawler. metro.co.uk’s robots.txt disallows OAI-SearchBot from the whole site (line 145: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.