Who blocks AI · #690 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does sky.com block AI crawlers?
No. sky.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.sky.com/robots.txt · 177 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | line 2 · Allow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.sky.com/robots.txt · 177 lines
1User-agent: OAI-SearchBot
2Allow: /
3User-agent: OAI-AdsBot
4Allow: /
5
6User-agent: *
7Disallow: /homepage/*
8Disallow: /shop/pre-reg/sky-essentials
9Disallow: /shop/pre-reg/sports-extra/success
10Disallow: /shop/pre-reg/ultrafast/success
11Disallow: /shop/pre-reg/ultrafast/agent
12Disallow: /shop/mobile/browse-mobile-vip/vip-offers
13Disallow: /shop/mobile/5g
14Disallow: /tv-guide/*
15Disallow: /portal/site/skybb/search
16Disallow: /cgi/
17Disallow: /Account_Billing/
18Disallow: /Assets/
19Disallow: /Getting_Started/
20Disallow: /How_To_Get_Sky/
21Disallow: /Product_Help/
22Disallow: /RFTS
23Disallow: /SPORTS
24Disallow: /Services/
25Disallow: /Sky_Online/
26Disallow: /anytimepc/
27Disallow: /article/
28Disallow: /corporate.
29Disallow: /designerbox.
30Disallow: /home/jvc/history.htm
31Disallow: /home_flash/
32Disallow: /javascript:''
33Disallow: /joinget
34Disallow: /journeyman
35Disallow: /motoring.
36Disallow: /movies/moviemax/index.htm
37Disallow: /movies/premier/index.htm
38Disallow: /notlikelytohappenever/
39Disallow: /pokemon
40Disallow: /pressoffice
41Disallow: /responsibilities
42Disallow: /rocket
43Disallow: /scandal
44Disallow: /sky+
45Disallow: /sky-news/
46Disallow: /simpsons2000
47Disallow: /sky1email:
48Disallow: /skybuy
49Disallow: /skydigital/info.html/
50Disallow: /skyscape/
51Disallow: /skytalk)
52Disallow: /soft/
53Disallow: /staff/
54Disallow: /students
55Disallow: /thailandgolf.
56Disallow: /unlock...been
57Disallow: /vita.html
58Disallow: /yourmoney
59Disallow: /errors
60Disallow: /skyq/buy
61Disallow: /.+
62Disallow: /*?_escaped_fragment
63Disallow: /*?adobe_mc
64Disallow: /*?agentId
65Disallow: /*?_1
66Disallow: /*?authorised
67Disallow: /*?auto
68Disallow: /*?awc
69Disallow: /*?back
70Disallow: /*?callsandtexts
71Disallow: /*?cn-type
72Disallow: /*?country
73Disallow: /*?currency
74Disallow: /*?contact
75Disallow: /*?dcmp
76Disallow: /*?dclid
77Disallow: /*?external
78Disallow: /*?ec
79Disallow: /*?epgoverlay
80Disallow: /*?explained
81Disallow: /*?goalAchievementShowContactOptions
82Disallow: /*?handset
83Disallow: /*?fbclid
84Disallow: /*?intent
85Disallow: /*?id
86Disallow: /*?integrationId
87Disallow: /*?ignore
88Disallow: /*?irct
89Disallow: /*?journey
90Disallow: /*?lightBoxContentItem
91Disallow: /*?mobileApp
92Disallow: /*?-kfc-to-offer-free,-fast-internet-across-all-uk-restaurants/
93Disallow: /*?mi_u
94Disallow: /*?product
95Disallow: /*?page
96Disallow: /*?product_typ
97Disallow: /*?OFF
98Disallow: /*?OSPV
99Disallow: /*?q
100Disallow: /*?ref
101Disallow: /*?refererIs
102Disallow: /*?returnToAuthFilter
103Disallow: /*?shortCode
104Disallow: /*?start
105Disallow: /*?sa
106Disallow: /*?said%20the%20number%20would%20be%20up%20
107Disallow: /*?topic
108Disallow: /*?ticbz
109Disallow: /*?utm_source
110Disallow: /*?target
111Disallow: /*?utm_content
112Disallow: /*new-search
113Disallow: /signin
114Disallow: /sky-go/?uuid
115Disallow: /sky-go/all?uuid
116Disallow: /sky-go/android?uuid
117Disallow: /*?successUrl=*
118Disallow: /*?utm_
119
120User-agent: AdsBot-Google
121Disallow: /shop/pre-reg/sky-essentials
122Disallow: /shop/pre-reg/sports-extra/success
123Disallow: /shop/pre-reg/ultrafast/success
124Disallow: /shop/pre-reg/ultrafast/agent
125Disallow: /shop/mobile/browse-mobile-vip/vip-offers
126Disallow: /portal/site/skybb/search
127Disallow: /cgi/
128Disallow: /Account_Billing/
129Disallow: /Assets/
130Disallow: /Getting_Started/
131Disallow: /How_To_Get_Sky/
132Disallow: /Product_Help/
133Disallow: /RFTS
134Disallow: /SPORTS
135Disallow: /Services/
136Disallow: /Sky_Online/
137Disallow: /anytimepc/
138Disallow: /article/
139Disallow: /corporate.
140Disallow: /designerbox.
141Disallow: /home/jvc/history.htm
142Disallow: /home_flash/
143Disallow: /javascript:''
144Disallow: /joinget
145Disallow: /journeyman
146Disallow: /motoring.
147Disallow: /movies/moviemax/index.htm
148Disallow: /movies/premier/index.htm
149Disallow: /notlikelytohappenever/
150Disallow: /pokemon
151Disallow: /pressoffice
152Disallow: /responsibilities
153Disallow: /rocket
154Disallow: /scandal
155Disallow: /sky+
156Disallow: /sky-news/
157Disallow: /simpsons2000
158Disallow: /sky1email:
159Disallow: /skybuy
160Disallow: /skydigital/info.html/
161Disallow: /skyscape/
162Disallow: /skytalk)
163Disallow: /soft/
164Disallow: /staff/
165Disallow: /students
166Disallow: /signin
167Disallow: /thailandgolf.
168Disallow: /unlock...been
169Disallow: /vita.html
170Disallow: /yourmoney
171Disallow: /errors
172Disallow: /errors/
173Disallow: /dtv/
174Disallow: /blackjack/
175
176Sitemap: https://www.sky.com/sitemap_index.xml
177
What do the bots actually get?
This page reads robots.txt, which states what sky.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on sky.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does sky.com block GPTBot?
No. sky.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does sky.com block ClaudeBot?
No. sky.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does sky.com block Google-Extended?
No. sky.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does sky.com block PerplexityBot?
No. sky.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find sky.com?
Yes. sky.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.