Who blocks AI · #694 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does mirror.co.uk block AI crawlers?

Yes. mirror.co.uk’s robots.txt blocks 8 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, PerplexityBot and 5 more. It also blocks Bingbot among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedAllowed CCBotBlocked

robots.txt: www.mirror.co.uk/robots.txt · 95 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Blocked	line 94 · `Disallow: /` under `User-agent: bingbot`
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Allowed	no rule matches the home page
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 55 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 61 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Allowed	no rule matches the home page
`DuckAssistBot`	DuckDuckGo	Allowed	no rule matches the home page
`Amazonbot`	Amazon	Allowed	no rule matches the home page
`YouBot`	You.com	Blocked	line 70 · `Disallow: /` under `User-agent: YouBot`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 52 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 73 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Allowed	no rule matches the home page
`CCBot`	Common Crawl	Blocked	line 67 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 58 · `Disallow: /` under `User-agent: Applebot-Extended`
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Blocked	line 94 · `Disallow: /` under `User-agent: Meta-ExternalAgent`
`Bytespider`	ByteDance	Allowed	no rule matches the home page

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Allowed	no rule matches the home page
`Claude-User`	Anthropic	Allowed	no rule matches the home page
`Perplexity-User`	Perplexity	Allowed	no rule matches the home page
`MistralAI-User`	Mistral	Allowed	no rule matches the home page
`Meta-ExternalFetcher`	Meta	Allowed	no rule matches the home page

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 49 · `Disallow: /` under `User-agent: AhrefsBot`
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View www.mirror.co.uk/robots.txt · 95 lines

1Sitemap: https://www.mirror.co.uk/map_news.xml

2Sitemap: https://www.mirror.co.uk/sitemaps/sitemap_index.xml

3User-agent: *

4Disallow: /topics/*

5Disallow: /topics

6Disallow: /*token=*

7Crawl-delay: 10.0

9Disallow: /search/

10Disallow: /comm-part-test/

11Disallow: /*service=ajax

12Disallow: /centenary-fund/

13Disallow: /3am/weird-celeb-news/xxx-1341448

14Disallow: /3am/weird-celeb-news/tamara-ecclestone-watched-boyfriends-sex-1341448

15Disallow: /comm-part-test/oh-like-beside-seaside-uk-1722203

16Disallow: /resources/js/s_code.js

17Disallow: /template/

18Disallow: /tv/tv-news/jean-alexander-dies-coronation-streets-3764827

19Disallow: /3am/celebrity-news/anne-kirkbride-dead-bill-roache-5013775

20Disallow: /lifestyle/cartoons/andy-capp/andy-capp

21Disallow: /lifestyle/cartoons/the-gag-vault/gag-vault

22Disallow: /lifestyle/cartoons/perishers/perishers

23Disallow: /lifestyle/cartoons/horace/horace

24Disallow: /lifestyle/cartoons/garth/garth

25Disallow: /lifestyle/cartoons/mandy/mandy

26Disallow: /lifestyle/cartoons/kerber-black/

27Disallow: /regression-test-home/

28Disallow: /5293/

29Disallow: /exclusive-offers/

31#Agent Specific Disallowed Sections

33User-agent: daumoa

34Disallow: /

36User-agent: Sosospider

37Disallow: /

39User-agent: rogerbot

40Disallow: /

42User-agent: semetrical

43Disallow: /

45User-agent: Googlebot-News

46Disallow: /thedavedesk/

48User-agent: AhrefsBot

49Disallow: /

51User-agent: GPTBot

52Disallow: /

54User-agent: OAI-SearchBot

55Disallow: /

57User-agent: Applebot-Extended

58Disallow: /

60User-agent: PerplexityBot

61Disallow: /

63User-agent: Perplexity-ai

64Disallow: /

66User-agent: CCBot

67Disallow: /

69User-agent: YouBot

70Disallow: /

72User-agent: ClaudeBot

73Disallow: /

75User-agent: anthropic-ai

76Disallow: /

78User-agent: Claude-Web

79Disallow: /

81User-agent: grapeshot

82Crawl-delay: 0

84User-agent: bingbot

85Crawl-delay: 1

87User-agent: AmazonAdBot

88Crawl-delay: 0

90User-agent: ozone

91Crawl-delay: 0

93User-agent: Meta-ExternalAgent

94Disallow: /

What do the bots actually get?

This page reads robots.txt, which states what mirror.co.uk asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on mirror.co.uk Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does mirror.co.uk block GPTBot?

Yes. mirror.co.uk’s robots.txt disallows GPTBot from the whole site (line 52: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does mirror.co.uk block ClaudeBot?

Yes. mirror.co.uk’s robots.txt disallows ClaudeBot from the whole site (line 73: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does mirror.co.uk block Google-Extended?

No. mirror.co.uk’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does mirror.co.uk block PerplexityBot?

Yes. mirror.co.uk’s robots.txt disallows PerplexityBot from the whole site (line 61: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find mirror.co.uk?

Not through its own crawler. mirror.co.uk’s robots.txt disallows OAI-SearchBot from the whole site (line 55: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.