Who blocks AI · #138 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does bloomberg.com block AI crawlers?
Yes. bloomberg.com’s robots.txt blocks 16 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 13 more. It also blocks PetalBot among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.bloomberg.com/robots.txt · 319 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 247 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 111 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 78 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 111 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 78 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 213 · Disallow: / under User-agent: AmazonBot |
YouBot | You.com | Blocked | line 78 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 96 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 96 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 35 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 64 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 213 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 213 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 213 · Disallow: / under User-agent: ByteSpider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 111 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 111 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Blocked | line 213 · Disallow: / under User-agent: Meta-ExternalFetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.bloomberg.com/robots.txt · 319 lines
1# Bot rules:
2# 1. A bot may not injure a human being or, through inaction, allow a human being to come to harm.
3# 2. A bot must obey orders given it by human beings except where such orders would conflict with the First Law.
4# 3. A bot must protect its own existence as long as such protection does not conflict with the First or Second Law.
5# If you can read this then you should apply here https://www.bloomberg.com/careers/
6User-agent: *
7Disallow: /polska
8Allow: /account/newsletters
9Disallow: /account/*
10Disallow: /tosv*.html
11Disallow: /search
12Disallow: /company/search/
13Disallow: /professional/search/
14Disallow: /impact/search/
15Disallow: /ux/search/
16Disallow: /wnwi/search/
17Disallow: /gei/search/
18Disallow: /impact/search/
19Disallow: /netzeropathfinders/search/
20Disallow: /notices/search/
21Disallow: /distribution/search/
22Disallow: /ukinnovators/search/
23Disallow: /latam/search/
24Disallow: /faq/search/
25Disallow: /tc/search/
26Disallow: /subscriptions/group/manage/
27Disallow: /preview/lineup
28Disallow: /preview/articles
29Disallow: /explore/
30Disallow: /press-releases/
31Disallow: /artemis/
32Disallow: /sessions-publisher/
33
34User-agent: Google-Extended
35Disallow: /
36Allow: /professional
37Allow: /company
38Allow: /latam
39Allow: /faq
40Allow: /tc
41
42User-agent: Mediapartners-Google
43Disallow: /about/careers
44Disallow: /about/careers/
45Disallow: /offlinemessage/
46Disallow: /apps/fbk
47Disallow: /bb/newsarchive/
48Disallow: /apps/news
49
50User-agent: Spinn3r
51Disallow: /podcasts/
52Disallow: /feed/podcast/
53Disallow: /bb/avfile/
54
55User-agent: Googlebot-News
56Disallow: /sponsor/
57Disallow: /news/sponsors/*
58Disallow: /news/terminal/*
59
60User-agent: Twitterbot
61Allow: /en/news/thp
62
63User-agent: CCBot
64Disallow: /
65
66# Development Tools
67User-agent: Python-urllib
68User-agent: python-requests
69User-agent: Python-http-client
70Disallow: /
71
72# AI/LLM Bots
73User-agent: Claude-Web
74User-agent: anthropic-ai
75User-agent: PerplexityBot
76User-agent: YouBot
77User-agent: DuckAssistBot
78Disallow: /
79Allow: /professional
80Allow: /company
81Allow: /latam
82Allow: /faq
83Allow: /tc
84User-agent: cohere-ai
85Disallow: /
86
87# AI/LLM Bots - Homepage Access
88# Homepage and regional homepages only. The $ makes each Allow an exact match
89# so it beats "Disallow: /"; without it the rule is a prefix match and opens
90# everything under that route. /region/ is a 302 to /region and needs its own
91# rule, since robots.txt is checked before the redirect can be followed.
92
93# Training
94User-agent: GPTBot
95User-agent: ClaudeBot
96Disallow: /
97Allow: /professional
98Allow: /company
99Allow: /latam
100Allow: /faq
101Allow: /tc
102
103# Search indexing and user-triggered fetches
104# Each exact Allow has a ?bloomberg_debug= twin so we can tag our own test
105# fetches and find them in the Fastly logs, i.e.
106# /llms.txt?bloomberg_debug=digest-20261102
107User-agent: OAI-SearchBot
108User-agent: Claude-SearchBot
109User-agent: ChatGPT-User
110User-agent: Claude-User
111Disallow: /
112Allow: /professional
113Allow: /company
114Allow: /latam
115Allow: /faq
116Allow: /tc
117Allow: /$
118Allow: /?bloomberg_debug=
119Allow: /europe$
120Allow: /europe?bloomberg_debug=
121Allow: /europe/$
122Allow: /europe/?bloomberg_debug=
123Allow: /asia$
124Allow: /asia?bloomberg_debug=
125Allow: /asia/$
126Allow: /asia/?bloomberg_debug=
127Allow: /jp$
128Allow: /jp?bloomberg_debug=
129Allow: /jp/$
130Allow: /jp/?bloomberg_debug=
131Allow: /india$
132Allow: /india?bloomberg_debug=
133Allow: /india/$
134Allow: /india/?bloomberg_debug=
135Allow: /middleeast$
136Allow: /middleeast?bloomberg_debug=
137Allow: /middleeast/$
138Allow: /middleeast/?bloomberg_debug=
139Allow: /africa$
140Allow: /africa?bloomberg_debug=
141Allow: /africa/$
142Allow: /africa/?bloomberg_debug=
143Allow: /canada$
144Allow: /canada?bloomberg_debug=
145Allow: /canada/$
146Allow: /canada/?bloomberg_debug=
147Allow: /uk$
148Allow: /uk?bloomberg_debug=
149Allow: /uk/$
150Allow: /uk/?bloomberg_debug=
151Allow: /latinamerica$
152Allow: /latinamerica?bloomberg_debug=
153Allow: /latinamerica/$
154Allow: /latinamerica/?bloomberg_debug=
155Allow: /llms.txt$
156Allow: /llms.txt?bloomberg_debug=
157Allow: /llms.txt/$
158Allow: /llms.txt/?bloomberg_debug=
159Allow: /europe/llms.txt$
160Allow: /europe/llms.txt?bloomberg_debug=
161Allow: /europe/llms.txt/$
162Allow: /europe/llms.txt/?bloomberg_debug=
163Allow: /asia/llms.txt$
164Allow: /asia/llms.txt?bloomberg_debug=
165Allow: /asia/llms.txt/$
166Allow: /asia/llms.txt/?bloomberg_debug=
167Allow: /jp/llms.txt$
168Allow: /jp/llms.txt?bloomberg_debug=
169Allow: /jp/llms.txt/$
170Allow: /jp/llms.txt/?bloomberg_debug=
171Allow: /india/llms.txt$
172Allow: /india/llms.txt?bloomberg_debug=
173Allow: /india/llms.txt/$
174Allow: /india/llms.txt/?bloomberg_debug=
175Allow: /middleeast/llms.txt$
176Allow: /middleeast/llms.txt?bloomberg_debug=
177Allow: /middleeast/llms.txt/$
178Allow: /middleeast/llms.txt/?bloomberg_debug=
179Allow: /africa/llms.txt$
180Allow: /africa/llms.txt?bloomberg_debug=
181Allow: /africa/llms.txt/$
182Allow: /africa/llms.txt/?bloomberg_debug=
183Allow: /canada/llms.txt$
184Allow: /canada/llms.txt?bloomberg_debug=
185Allow: /canada/llms.txt/$
186Allow: /canada/llms.txt/?bloomberg_debug=
187Allow: /uk/llms.txt$
188Allow: /uk/llms.txt?bloomberg_debug=
189Allow: /uk/llms.txt/$
190Allow: /uk/llms.txt/?bloomberg_debug=
191Allow: /latinamerica/llms.txt$
192Allow: /latinamerica/llms.txt?bloomberg_debug=
193Allow: /latinamerica/llms.txt/$
194Allow: /latinamerica/llms.txt/?bloomberg_debug=
195
196# Google Services
197User-agent: Google-Apps
198User-agent: Google-Apps-Script
199User-agent: AppEngine-Google
200User-agent: Google-Cloud
201User-agent: Google-CloudVertexBot
202Disallow: /
203
204# Major Company Bots
205User-agent: AmazonBot
206User-agent: AmazonAdBot
207User-agent: ByteSpider
208User-agent: Meta-ExternalAgent
209User-agent: Meta-ExternalFetcher
210User-agent: Meta-ExternalAgent-Image
211User-agent: Applebot-Extended
212User-agent: Bytespider-Image
213Disallow: /
214
215# Feed & Content Aggregators
216User-agent: Feedly
217User-agent: FeedlyBot
218User-agent: FeedlyApp
219User-agent: MWFeedParser
220Disallow: /
221Allow: /professional
222Allow: /company
223Allow: /latam
224Allow: /faq
225Allow: /tc
226
227# Analytics & Marketing
228User-agent: comscore
229User-agent: HubSpot
230User-agent: Criteo
231User-agent: criteo-bot
232User-agent: Peer39_Crawler
233User-agent: rogerbot
234User-agent: linkdexbot
235User-agent: NTENT
236User-agent: AndersPinkBot
237User-agent: IndeedBot
238Disallow: /
239
240# Search Engine Bots
241User-agent: MJ12bot
242User-agent: PetalBot
243User-agent: YisouSpider
244User-agent: 360Spider
245User-agent: Qwantify
246User-agent: ToutiaoSpider
247Disallow: /
248
249# Monitoring & Security
250User-agent: Screaming Frog SEO Spider
251Disallow: /
252Allow: /professional
253Allow: /company
254Allow: /latam
255Allow: /faq
256Allow: /tc
257User-agent: PRTG
258User-agent: FreshpingBot
259User-agent: Panopta
260User-agent: DatadogSynthetics
261User-agent: Rackspace
262User-agent: censys
263User-agent: burp
264User-agent: check_http
265User-agent: DotcomMonitor
266User-agent: WatchSumo
267User-agent: WormlyBot
268User-agent: CalibreBot
269User-agent: AudistoBot
270Disallow: /
271
272# Social & Content
273User-agent: Hatena
274User-agent: Hatena-Bookmark
275User-agent: EveryoneSocialBot
276User-agent: Diffbot
277User-agent: Netvibes
278User-agent: WebCEO
279User-agent: Postano
280User-agent: RebelMouse
281User-agent: Muck-Rack
282User-agent: InstapaperViewer
283User-agent: Twurly
284User-agent: LivelapBot
285User-agent: DatagnionBot
286User-agent: Linespider
287User-agent: Discourse
288Disallow: /
289
290# Potentially Malicious
291User-agent: Medusa
292User-agent: pingback
293User-agent: WordPress
294User-agent: wp_ping
295User-agent: MauiBot
296User-agent: ltx71
297User-agent: WeSEE
298User-agent: halebot
299User-agent: BrightBot
300Disallow: /
301
302# Gaming/3D
303User-agent: UnityPlayer
304Disallow: /
305
306# Sitemaps app
307Sitemap: https://www.bloomberg.com/sitemaps/news/index.xml
308Sitemap: https://www.bloomberg.com/sitemaps/news/latest.xml
309Sitemap: https://www.bloomberg.com/sitemaps/collections/index.xml
310Sitemap: https://www.bloomberg.com/sitemaps/media/video/index.xml
311Sitemap: https://www.bloomberg.com/sitemaps/media/audio/index.xml
312Sitemap: https://www.bloomberg.com/sitemaps/people/profiles/index.xml
313Sitemap: https://www.bloomberg.com/sitemaps/companies/public-company/index.xml
314Sitemap: https://www.bloomberg.com/sitemaps/companies/private-company/index.xml
315Sitemap: https://www.bloomberg.com/sitemaps/securites/index.xml
316
317# Billionaires, owned by graphics
318Sitemap: https://www.bloomberg.com/billionaires/sitemap.xml
319
What do the bots actually get?
This page reads robots.txt, which states what bloomberg.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on bloomberg.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
- 2 November 2026: robots.txt readable again.
- 9 November 2026: robots.txt readable again.
Questions
Does bloomberg.com block GPTBot?
Yes. bloomberg.com’s robots.txt disallows GPTBot from the whole site (line 96: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does bloomberg.com block ClaudeBot?
Yes. bloomberg.com’s robots.txt disallows ClaudeBot from the whole site (line 96: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does bloomberg.com block Google-Extended?
Yes. bloomberg.com’s robots.txt disallows Google-Extended from the whole site (line 35: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does bloomberg.com block PerplexityBot?
Yes. bloomberg.com’s robots.txt disallows PerplexityBot from the whole site (line 78: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find bloomberg.com?
Not through its own crawler. bloomberg.com’s robots.txt disallows OAI-SearchBot from the whole site (line 111: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.