Who blocks AI · #691 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does thesun.co.uk block AI crawlers?
Yes. thesun.co.uk’s robots.txt blocks 15 of the 19 AI crawlers we track, including ClaudeBot, Google-Extended, PerplexityBot and 12 more. It also blocks PetalBot, YandexBot and Baiduspider among search engines.
GPTBotPartly blockedOAI-SearchBotPartly blockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.thesun.co.uk/robots.txt · 190 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 80 · Disallow: /search/ line 81 · Disallow: /horoscope/ and 99 more | |
Bingbot | Microsoft | Partly blocked | line 80 · Disallow: /search/ line 81 · Disallow: /horoscope/ and 99 more |
DuckDuckBot | DuckDuckGo | Partly blocked | line 80 · Disallow: /search/ line 81 · Disallow: /horoscope/ and 99 more |
Applebot | Apple | Partly blocked | line 80 · Disallow: /search/ line 81 · Disallow: /horoscope/ and 99 more |
PetalBot | Huawei | Blocked | line 8 · Disallow: / under User-agent: * |
YandexBot | Yandex | Blocked | line 8 · Disallow: / under User-agent: * |
Baiduspider | Baidu | Blocked | line 8 · Disallow: / under User-agent: * |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Partly blocked | line 80 · Disallow: /search/ line 81 · Disallow: /horoscope/ and 99 more |
PerplexityBot | Perplexity | Blocked | line 188 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 188 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 8 · Disallow: / under User-agent: * |
Amazonbot | Amazon | Blocked | line 8 · Disallow: / under User-agent: * |
YouBot | You.com | Blocked | line 8 · Disallow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Partly blocked | line 80 · Disallow: /search/ line 81 · Disallow: /horoscope/ and 99 more |
ClaudeBot | Anthropic | Blocked | line 8 · Disallow: / under User-agent: * |
Google-Extended | Blocked | line 188 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 188 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 8 · Disallow: / under User-agent: * |
GoogleOther | Partly blocked | line 80 · Disallow: /search/ line 81 · Disallow: /horoscope/ and 99 more | |
Meta-ExternalAgent | Meta | Blocked | line 8 · Disallow: / under User-agent: * |
Bytespider | ByteDance | Blocked | line 8 · Disallow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Partly blocked | line 80 · Disallow: /search/ line 81 · Disallow: /horoscope/ and 99 more |
Claude-User | Anthropic | Blocked | line 188 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 8 · Disallow: / under User-agent: * |
MistralAI-User | Mistral | Blocked | line 8 · Disallow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Blocked | line 8 · Disallow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 8 · Disallow: / under User-agent: * |
SemrushBot | Semrush | Partly blocked | line 80 · Disallow: /search/ line 81 · Disallow: /horoscope/ and 99 more |
MJ12bot | Majestic | Blocked | line 8 · Disallow: / under User-agent: * |
DotBot | Moz | Blocked | line 8 · Disallow: / under User-agent: * |
View www.thesun.co.uk/robots.txt · 190 lines
1# This is the robots.txt file for thesun.co.uk
2# The Sun does not permit the unlicensed use of our content for large language models.
3# News Group Newspaper's Terms of Use may be found at
4# https://www.thesun.co.uk/terms-and-conditions/#website-and-services-terms-of-use
5# Contact enquiries@newslicensing.co.uk for assistance.
6
7User-agent: *
8Disallow: /
9# Sitemap archive
10Sitemap: https://www.thesun.co.uk/sitemap.xml
11
12# Sitemaps
13Sitemap: https://www.thesun.co.uk/news-sitemap.xml
14
15# Additional sitemaps
16Sitemap: https://www.thesun.co.uk/sitemap-nypost-authors.xml
17Sitemap: https://www.thesun.co.uk/sitemap-nypost-post_tag.xml
18Sitemap: https://www.thesun.co.uk/sitemap-nypost-where.xml
19Sitemap: https://www.thesun.co.uk/sitemap-nypost-who.xml
20Sitemap: https://www.thesun.co.uk/sitemap-nypost-feeds.xml
21Sitemap: https://www.thesun.co.uk/sitemap-nypost-category.xml
22Sitemap: https://www.thesun.co.uk/sitemap-nypost-football-team-taxo.xml
23Sitemap: https://www.thesun.co.uk/sitemap-nypost-vertical-video.xml
24Sitemap: https://www.thesun.co.uk/sitemap-nypost-video.xml
25Sitemap: https://www.thesun.co.uk/sitemap-nypost-video-shows.xml
26
27
28User-agent: Googlebot
29User-agent: Googlebot-Image
30User-agent: Googlebot-Video
31User-agent: Googlebot-News
32User-agent: Google-InspectionTool
33User-agent: GoogleOther
34User-agent: APIs-Google
35User-agent: Mediapartners-Google
36User-agent: DuckDuckBot
37User-agent: Slurp
38User-agent: Twitterbot
39User-agent: Bingbot
40User-agent: adidxbot
41User-agent: MicrosoftPreview
42User-agent: Applebot
43User-agent: AdsBot-Google
44User-agent: AmazonAdBot
45User-agent: facebookexternalhit
46User-agent: Yahoo Ad Monitoring
47User-agent: parse.ly scraper
48User-agent: Screaming Frog SEO Spider
49User-agent: SemrushBot
50User-agent: OnCrawl
51User-agent: Chrome-Lighthouse
52User-agent: Slackbot-LinkExpanding
53User-agent: Slack-ImgProxy
54User-agent: Slackbot
55User-agent: Snap URL Preview Service
56User-agent: proximic
57User-agent: GumGum-Bot
58User-agent: TTD-Content
59User-agent: Pubmatic
60User-agent: ias_crawler
61User-agent: outbrain
62User-agent: Leikibot
63User-agent: weborama-fetcher
64User-agent: AHC/2.1
65User-agent: SirdataBot
66User-agent: peer39_crawler/1.0
67User-agent: EchoboxBot/1.0
68User-agent: CriteoBot/0.1
69User-agent: AdDefend Site Context Crawler/1.0
70User-agent: SmartologyBot
71User-agent: Dianomi
72User-agent: ChatGPT-User
73User-agent: GPTBot
74User-agent: OAI-SearchBot
75User-agent: snews
76
77Allow: /
78Allow: /ads.txt
79Allow: /wp-admin/admin-ajax.php
80Disallow: /search/
81Disallow: /horoscope/
82Disallow: /*?x=
83Disallow: /*?action=
84Disallow: /*&action=
85Disallow: /*?adid=
86Disallow: /*&adid=
87Disallow: /*?bitrate=
88Disallow: /*&bitrate=
89Disallow: /*?channel=
90Disallow: /*&channel=
91Disallow: /*?clipid=
92Disallow: /*&clipid=
93Disallow: /*?contentCollection=
94Disallow: /*&contentCollection=
95Disallow: /*?contentId=
96Disallow: /*&contentId=
97Disallow: /*?ei=
98Disallow: /*&ei=
99Disallow: /*?format=
100Disallow: /*&format=
101Disallow: /*?height=
102Disallow: /*&height=
103Disallow: /*?iframe=
104Disallow: /*&iframe=
105Disallow: /*?mediaId=
106Disallow: /*&mediaId=
107Disallow: /*?module=
108Disallow: /*&module=
109Disallow: /*?monthnum=
110Disallow: /*&monthnum=
111Disallow: /*?pgtype=
112Disallow: /*&pgtype=
113Disallow: /*?preview=
114Disallow: /*&preview=
115Disallow: /*?priority=
116Disallow: /*&priority=
117Disallow: /*?sa=
118Disallow: /*&sa=
119Disallow: /*?section=
120Disallow: /*§ion=
121Disallow: /*?share=
122Disallow: /*&share=
123Disallow: /*?TB_iframe=
124Disallow: /*&TB_iframe=
125Disallow: /*?TBL_Prod100Page=
126Disallow: /*&TBL_Prod100Page=
127Disallow: /*?theme_preview=
128Disallow: /*&theme_preview=
129Disallow: /*?usg=
130Disallow: /*&usg=
131Disallow: /*?ved=
132Disallow: /*&ved=
133Disallow: /*?version=
134Disallow: /*&version=
135Disallow: /*?VID=
136Disallow: /*&VID=
137Disallow: /*?vxBitrate=
138Disallow: /*&vxBitrate=
139Disallow: /*?vxChannel=
140Disallow: /*&vxChannel=
141Disallow: /*?vxClipId=
142Disallow: /*&vxClipId=
143Disallow: /*?vxSiteId=
144Disallow: /*&vxSiteId=
145Disallow: /*?width=
146Disallow: /*&width=
147Disallow: /*?year=
148Disallow: /*&year=
149Disallow: /*.xml?uuid=
150Disallow: /*?post_type=feedback&p=
151Disallow: /*?wptouch_preview_theme=
152Disallow: /*>
153Disallow: /?attachment_id=
154Disallow: /affiliate/
155Disallow: /ai_log/
156Disallow: /*?*pg99.asia*
157Disallow: /*&*pg99.asia*
158Disallow: /*?*gg155.cn*
159Disallow: /*&*gg155.cn*
160Disallow: /*?*To66.Asia*
161Disallow: /*&*To66.Asia*
162Disallow: /wp-admin/
163Disallow: /*horseracing/racecards/2019*
164Disallow: /*horseracing/racecards/2020*
165Disallow: /*horseracing/racecards/2021*
166Disallow: /*horseracing/racecards/2022*
167Disallow: /*horseracing/results/2019*
168Disallow: /*horseracing/results/2020*
169Disallow: /*horseracing/results/2021*
170Disallow: /*horseracing/results/2022*
171Disallow: /simwidgets/
172Disallow: /player/brightcove/
173Disallow: /*?s=*
174Disallow: *&s=*
175Disallow: /?p=*
176Disallow: /app/
177Disallow: /sso/login/
178Disallow: /wp-login.php
179Disallow: /amp-tealium/
180Disallow: /archives/
181
182User-agent: Google-Extended
183User-agent: Anthropic-ai
184User-agent: Claude-SearchBot
185User-agent: Claude-User
186User-agent: CCBot
187User-agent: PerplexityBot
188Disallow: /
189Allow: /sell-your-story-sun/$
190
What do the bots actually get?
This page reads robots.txt, which states what thesun.co.uk asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on thesun.co.uk Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does thesun.co.uk block GPTBot?
Partly. thesun.co.uk’s robots.txt keeps GPTBot out of some paths, such as /search/, and lets it crawl the rest. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does thesun.co.uk block ClaudeBot?
Yes. thesun.co.uk’s robots.txt disallows ClaudeBot from the whole site (line 8: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does thesun.co.uk block Google-Extended?
Yes. thesun.co.uk’s robots.txt disallows Google-Extended from the whole site (line 188: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does thesun.co.uk block PerplexityBot?
Yes. thesun.co.uk’s robots.txt disallows PerplexityBot from the whole site (line 188: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find thesun.co.uk?
Partly. thesun.co.uk’s robots.txt keeps OAI-SearchBot out of some paths, such as /search/, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.