Who blocks AI · #215 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does telegraph.co.uk block AI crawlers?

Yes. telegraph.co.uk’s robots.txt blocks 17 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, PerplexityBot and 14 more. It also blocks PetalBot among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedPartly blocked CCBotBlocked

robots.txt: www.telegraph.co.uk/robots.txt · 191 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	line 47 · `Allow: /` under `User-agent: *`
`Bingbot`	Microsoft	Allowed	line 47 · `Allow: /` under `User-agent: *`
`DuckDuckBot`	DuckDuckGo	Allowed	line 47 · `Allow: /` under `User-agent: *`
`Applebot`	Apple	Allowed	line 47 · `Allow: /` under `User-agent: *`
`PetalBot`	Huawei	Blocked	line 111 · `Disallow: /` under `User-agent: PetalBot`
`YandexBot`	Yandex	Allowed	line 47 · `Allow: /` under `User-agent: *`
`Baiduspider`	Baidu	Allowed	line 47 · `Allow: /` under `User-agent: *`

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 111 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 111 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Blocked	line 111 · `Disallow: /` under `User-agent: Claude-SearchBot`
`DuckAssistBot`	DuckDuckGo	Blocked	line 111 · `Disallow: /` under `User-agent: DuckAssistBot`
`Amazonbot`	Amazon	Blocked	line 111 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Allowed	line 47 · `Allow: /` under `User-agent: *`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 111 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 111 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Partly blocked	line 6 · `Disallow: /etc.clientlibs/settings/wcm/designs/telegraph/core/clientlibs/save-article..js` line 7 · `Disallow: /etc.clientlibs/settings/wcm/designs/telegraph/core/clientlibs/page-refresh..js` and 33 more
`CCBot`	Common Crawl	Blocked	line 111 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 111 · `Disallow: /` under `User-agent: Applebot-Extended`
`GoogleOther`	Google	Blocked	line 111 · `Disallow: /` under `User-agent: GoogleOther`
`Meta-ExternalAgent`	Meta	Blocked	line 111 · `Disallow: /` under `User-agent: Meta-ExternalAgent`
`Bytespider`	ByteDance	Blocked	line 111 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 111 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Blocked	line 111 · `Disallow: /` under `User-agent: Claude-User`
`Perplexity-User`	Perplexity	Blocked	line 111 · `Disallow: /` under `User-agent: Perplexity-User`
`MistralAI-User`	Mistral	Blocked	line 111 · `Disallow: /` under `User-agent: MistralAI-User`
`Meta-ExternalFetcher`	Meta	Blocked	line 111 · `Disallow: /` under `User-agent: Meta-ExternalFetcher`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	line 47 · `Allow: /` under `User-agent: *`
`SemrushBot`	Semrush	Allowed	line 47 · `Allow: /` under `User-agent: *`
`MJ12bot`	Majestic	Allowed	line 47 · `Allow: /` under `User-agent: *`
`DotBot`	Moz	Allowed	line 47 · `Allow: /` under `User-agent: *`

View www.telegraph.co.uk/robots.txt · 191 lines

1# Production Robots.txt file

3User-agent: *

4User-agent: Google-Extended

5# Special parameters

6Disallow: /etc.clientlibs/settings/wcm/designs/telegraph/core/clientlibs/save-article.*.js

7Disallow: /etc.clientlibs/settings/wcm/designs/telegraph/core/clientlibs/page-refresh.*.js

8Disallow: /*?mobile=true

9Disallow: /*?mobile=basic

10Disallow: /*?ModPagespeed=noscript

11Disallow: /*_jcr_content*

12Disallow: /*?source=rss

13Disallow: /puzzles/puzzle/*?source=

15# Internal Search

16Disallow: /search/

17Allow: /search/$

19# Special areas

20Disallow: /news/main.jhtml

21Disallow: /p/*/embed/

22Disallow: /secure/login/*

23Disallow: /content/telegraph/

24Disallow: /customer/secure/checkout/tesco/

25Disallow: /customer/secure/reset-password/

26Disallow: /telegraph/*

27Disallow: /news-app/*

28Disallow: /amp$

29Disallow: /application/*

30Disallow: /ixale/

31Disallow: /core/Content/

32Disallow: /promotions/emails/

33Disallow: /r/

34Disallow: /sponsored/travel/msc-cruises/

35Disallow: /travel/hotel/e/*

36Disallow: /travel/hotels/hotel-finder/

37Disallow: /podcasts-more/

38Disallow: /secure/register/

39Allow: /travel/hotels/hotel-finder/$

40Disallow: /martech/js/

41Disallow: /martech/css/

42Disallow: /martech-content/

43Disallow: /bin/telegraph/recombee-config

44Disallow: /*&p=

45Disallow: /customer/subscription/*?

46Disallow: /edge-redirection/

47Allow: /

50# Bots which make unnecessary bot traffic

51User-agent: endeca

52Disallow: /archive/

53Disallow: /search/*

55User-agent: AI2Bot

56User-agent: Ai2Bot-Dolma

57User-agent: Amazonbot

58User-agent: anthropic-ai

59User-agent: Applebot-Extended

60User-agent: bedrockbot

61User-agent: Bytespider

62User-agent: CCBot

63User-agent: ChatGLM-Spider

64User-agent: ClaudeBot

65User-agent: Claude-Web

66User-agent: cohere-ai

67User-agent: Cotoyogi

68User-agent: DeepSeekBot

69User-agent: Diffbot

70User-agent: FacebookBot

71User-agent: FriendlyCrawler

72User-agent: Google-CloudVertexBot

73User-agent: GoogleOther

74User-agent: GoogleOther-Image

75User-agent: GoogleOther-Video

76User-agent: GPTBot

77User-agent: Grok

78User-agent: iaskspider/2.0

79User-agent: ICC-Crawler

80User-agent: ImagesiftBot

81User-agent: img2dataset

82User-agent: ISSCyberRiskCrawler

83User-agent: Kangaroo Bot

84User-agent: KunatoCrawler

85User-agent: Meltwater

86User-agent: Meta-ExternalAgent

87User-agent: omgili

88User-agent: omgilibot

89User-agent: PanguBot

90User-agent: PetalBot

91User-agent: QualifiedBot

92User-agent: Scrapy

93User-agent: Seekr

94User-agent: Sidetrade indexer bot

95User-agent: TikTokSpider

96User-agent: Timpibot

97User-agent: VelenPublicWebCrawler

98User-agent: WARDBot

99User-agent: Webzio-Extended

100User-agent: WRTNBot

101User-agent: Claude-SearchBot

102User-agent: DuckAssistBot

103User-agent: OAI-SearchBot

104User-agent: PerplexityBot

105User-agent: ChatGPT-User

106User-agent: Claude-User

107User-agent: Meta-ExternalFetcher

108User-agent: MistralAI-User

109User-agent: OAI-Operator

110User-agent: Perplexity-User

111Disallow: /

112Allow: /travel/explore-dubai/

113Allow: /travel/luxury-stays/

114Allow: /business/uk-business-growth/

115Allow: /ads.txt

116Allow: /bah-avis/usa-sports-road-trip-map

117Allow: /betting/

118Allow: /british-airways-holidays/cyprus-hotel-map

119Allow: /business/ai-era-job-skills/

120Allow: /business/ai-workflow-transformation/

121Allow: /business/future-hr-technology/

122Allow: /business/golf-dp-world-tour/

123Allow: /business/search-marketing/

124Allow: /business/waste-management/

125Allow: /deals/

126Allow: /environment/woodland-action/

127Allow: /fashion/joseph-ribkoff-autumn-collection/

128Allow: /financial-services/

129Allow: /food-and-drink/m-and-s-seasonal-favourites/

130Allow: /food-and-drink/m-and-s-seasonal-favourites/organic-range/

131Allow: /food-and-drink/maldon-salt-perfect-pyramid-challenge/

132Allow: /food-and-drink/spanish-dining-uk/

133Allow: /football/global-football-industry-awards/

134Allow: /health-fitness/how-to-choose-private-hospital-hca-healthcare/

135Allow: /interiors/next-home-autumn-interiors-trends/

136Allow: /junior-golf/

137Allow: /lifestyle-solutions/

138Allow: /lifestyle/apl-media/

139Allow: /branded-content/

140Allow: /money/investing/fc-investment-trust/long-term-investing/

141Allow: /branded-content/

142Allow: /rugby-union/grassroots-rugby-heroes/

143Allow: /rugby-union/levelling-the-playing-field/

144Allow: /sailgp/inside-f50-catamaran

145Allow: /seabourn/explore-the-mediterranean

146Allow: /sport/sailgp-championship/

147Allow: /travel/active-family-holidays/

148Allow: /travel/away-holidays/

149Allow: /travel/barbados-holiday-guide/

150Allow: /travel/british-airways-american-express/

151Allow: /travel/canadian-affair-holidays/rocky-mountaineer-train-route/

152Allow: /travel/croatia-holidays/

153Allow: /travel/cruises/holland-america-line-luxury-holidays/

154Allow: /travel/cruises/seabourn-luxury-cruises/

155Allow: /travel/discover-america/

156Allow: /travel/exclusiveoffers/

157Allow: /branded-content/

158Allow: /travel/florida-guide/

159Allow: /travel/fly-drive-holidays/

160Allow: /travel/grenada-guide/

161Allow: /travel/icons-of-the-sky/

162Allow: /travel/indian-ocean-escapes/

163Allow: /travel/indulge-in-ireland/

164Allow: /travel/jamaica-holiday-guide/

165Allow: /travel/japan-ninja-heritage/

166Allow: /travel/luxury-all-inclusive-resorts/affordable-caribbean-hotels/

167Allow: /travel/luxury-holidays/

168Allow: /travel/luxury-hotel-getaways/

169Allow: /travel/luxury-island-escapes/

170Allow: /travel/maldives-holiday-guide/

171Allow: /travel/ski-resorts-for-everyone/

172Allow: /travel/spark/best-of-switzerland/

173Allow: /travel/tailor-made-tours/

174Allow: /travel/tenerife-holidays/

175Allow: /travel/tips-for-visiting-spain/

176Allow: /travel/travel-counsellors/book-a-cruise-with-travel-counsellors/

177Allow: /travel/travel-solutions/

178Allow: /travel/visit-cayman-islands/

179Allow: /travel/why-choose-malta/

180Allow: /trusted-partners/

181Allow: /brand-usa

182Allow: /travel/visit-aarhus/

183Allow: /bespoke-content/

184

185# Sitemap URLs

186Sitemap: https://www.telegraph.co.uk/sitemap.xml

187Sitemap: https://www.telegraph.co.uk/custom/authors/sitemap.xml

188Sitemap: https://www.telegraph.co.uk/custom/main-pages/sitemap.xml

189Sitemap: https://www.telegraph.co.uk/custom/daily-news/sitemap.xml

190Sitemap: https://www.telegraph.co.uk/custom/author-pages/sitemap.xml

191

What do the bots actually get?

This page reads robots.txt, which states what telegraph.co.uk asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on telegraph.co.uk Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does telegraph.co.uk block GPTBot?

Yes. telegraph.co.uk’s robots.txt disallows GPTBot from the whole site (line 111: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does telegraph.co.uk block ClaudeBot?

Yes. telegraph.co.uk’s robots.txt disallows ClaudeBot from the whole site (line 111: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does telegraph.co.uk block Google-Extended?

Partly. telegraph.co.uk’s robots.txt keeps Google-Extended out of some paths, such as /etc.clientlibs/settings/wcm/designs/telegraph/core/clientlibs/save-article.*.js, and lets it crawl the rest. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does telegraph.co.uk block PerplexityBot?

Yes. telegraph.co.uk’s robots.txt disallows PerplexityBot from the whole site (line 111: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find telegraph.co.uk?

Not through its own crawler. telegraph.co.uk’s robots.txt disallows OAI-SearchBot from the whole site (line 111: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.