Who blocks AI · #777 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does binance.com block AI crawlers?
No. binance.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.binance.com/robots.txt · 203 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 18 · Disallow: */price*?hl= line 19 · Disallow: */how-to-buy*?hl= and 52 more | |
Bingbot | Microsoft | Partly blocked | line 18 · Disallow: */price*?hl= line 19 · Disallow: */how-to-buy*?hl= and 52 more |
DuckDuckBot | DuckDuckGo | Partly blocked | line 18 · Disallow: */price*?hl= line 19 · Disallow: */how-to-buy*?hl= and 52 more |
Applebot | Apple | Partly blocked | line 18 · Disallow: */price*?hl= line 19 · Disallow: */how-to-buy*?hl= and 52 more |
PetalBot | Huawei | Allowed | line 152 · Allow: / under User-agent: * |
YandexBot | Yandex | Allowed | line 152 · Allow: / under User-agent: * |
Baiduspider | Baidu | Partly blocked | line 18 · Disallow: */price*?hl= line 19 · Disallow: */how-to-buy*?hl= and 52 more |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | line 152 · Allow: / under User-agent: * |
PerplexityBot | Perplexity | Allowed | line 152 · Allow: / under User-agent: * |
Claude-SearchBot | Anthropic | Allowed | line 152 · Allow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Allowed | line 152 · Allow: / under User-agent: * |
Amazonbot | Amazon | Allowed | line 152 · Allow: / under User-agent: * |
YouBot | You.com | Allowed | line 152 · Allow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | line 152 · Allow: / under User-agent: * |
ClaudeBot | Anthropic | Allowed | line 152 · Allow: / under User-agent: * |
Google-Extended | Allowed | line 152 · Allow: / under User-agent: * | |
CCBot | Common Crawl | Allowed | line 152 · Allow: / under User-agent: * |
Applebot-Extended | Apple | Allowed | line 152 · Allow: / under User-agent: * |
GoogleOther | Allowed | line 152 · Allow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Allowed | line 152 · Allow: / under User-agent: * |
Bytespider | ByteDance | Allowed | line 152 · Allow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | line 152 · Allow: / under User-agent: * |
Claude-User | Anthropic | Allowed | line 152 · Allow: / under User-agent: * |
Perplexity-User | Perplexity | Allowed | line 152 · Allow: / under User-agent: * |
MistralAI-User | Mistral | Allowed | line 152 · Allow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Allowed | line 152 · Allow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | line 152 · Allow: / under User-agent: * |
SemrushBot | Semrush | Allowed | line 152 · Allow: / under User-agent: * |
MJ12bot | Majestic | Allowed | line 152 · Allow: / under User-agent: * |
DotBot | Moz | Allowed | line 152 · Allow: / under User-agent: * |
View www.binance.com/robots.txt · 203 lines
1# ---------------------------------------------------------------------------
2# Major search engines (Baidu / Bing / DuckDuckGo / Yahoo Slurp / Google)
3# ---------------------------------------------------------------------------
4User-agent: Baiduspider
5User-agent: Bingbot
6User-agent: DuckDuckBot
7User-agent: Slurp
8User-agent: Googlebot
9
10# Selective Allow overrides (carve-outs).
11Allow: /
12Allow: */bapi/fe/
13Allow: */sitemap_output/
14Allow: */funding-history/perpetual/
15Allow: */funding-history/quarterly/
16
17# Query-parameter blocks (?hl= honored reliably only by Google)
18Disallow: */price*?hl=
19Disallow: */how-to-buy*?hl=
20Disallow: */price-prediction*?hl=
21Disallow: */altcoins*?hl=
22
23# Legacy / undefined URL fragments
24Disallow: *undefine*
25Disallow: *UNDEFINE*
26Disallow: */collected/undefined
27
28# Activity / campaign landing (kept out of index)
29Disallow: */activity/challenge-competition/
30Disallow: */activity/collect-and-win/
31Disallow: */activity/referral-entry/
32
33# Tracking / noise query params
34Disallow: *fbclid=
35Disallow: /*?sqb=
36
37# Internal / non-indexable content surfaces
38Disallow: */advertiserDetail
39Disallow: */funding-history
40Disallow: */trading-rules
41Disallow: */comments
42Disallow: */feed/search
43Disallow: */feed/sitemap/
44Disallow: */version.web
45Disallow: */loan/data/
46Disallow: */markets/spot-FIAT/
47Disallow: */markets/spot-USDT/
48Disallow: */OTC-Trading/
49Disallow: */backtest/
50Disallow: */buy-sell-crypto/referral/
51Disallow: */buy-sell-crypto/buy-sell/result/buy/
52Disallow: */buy-sell-crypto/buy-sell/result/sell/
53Disallow: */chart-webview/
54Disallow: */chat/
55Disallow: */convert-modal-ui/
56Disallow: */customindicator/
57Disallow: */delivery/
58Disallow: */crypto/recurring/
59Disallow: */survey/
60Disallow: */academy/courses/certificate/
61
62# Compliance / privacy / account surfaces
63Disallow: */complaint-form/
64Disallow: */compliance-confirm/
65Disallow: */kyc-ui/
66Disallow: */my/
67Disallow: */deposit/result/order-v3/
68Disallow: */fiat/deposit/result/order
69
70# Internal / API / non-HTML endpoints
71Disallow: */bapi/
72Disallow: */api/
73Disallow: */fapi/
74Disallow: */fvideo/
75Disallow: */mp-cms/app/
76Disallow: */mp-cms/disclaimer/
77Disallow: */sitemap_.xml
78
79# Square (social) non-canonical surfaces
80Disallow: */en-AU/square
81Disallow: */en-GB/square
82Disallow: */square/creator-center/
83Disallow: */square/notification
84Disallow: */square/bookmark
85Disallow: */square/settings
86
87# ---------------------------------------------------------------------------
88# Yandex
89# ---------------------------------------------------------------------------
90User-agent: Yandex
91Host: www.binance.com
92
93# Selective Allow overrides (carve-outs).
94Allow: /
95Allow: */bapi/fe/
96Allow: */sitemap_output/
97Allow: */funding-history/perpetual/
98Allow: */funding-history/quarterly/
99
100# Query-parameter blocks
101Disallow: /*ref=
102Disallow: /*symbol=
103Disallow: /*version=
104Disallow: /*theme=
105Disallow: /*contract=
106Disallow: /*code=
107Disallow: /*tab=
108Disallow: /*poolId=
109Disallow: /*type=
110Disallow: /*layout=
111Disallow: /*navId=
112Disallow: /*channel=
113Disallow: /*utm_source=
114Disallow: /*source=
115Disallow: /*utm=
116Disallow: /*fbclid=
117Disallow: /*sqb=
118
119# Path-level blocks (mirrored from the major-engine group for compliance/privacy parity)
120Disallow: *undefine*
121Disallow: *UNDEFINE*
122Disallow: */collected/undefined
123Disallow: */complaint-form/
124Disallow: */compliance-confirm/
125Disallow: */kyc-ui/
126Disallow: */my/
127Disallow: */deposit/result/order-v3/
128Disallow: */fiat/deposit/result/order
129Disallow: */survey/
130Disallow: */academy/courses/certificate/
131Disallow: */bapi/
132Disallow: */api/
133Disallow: */fapi/
134Disallow: */fvideo/
135Disallow: */mp-cms/app/
136Disallow: */mp-cms/disclaimer/
137Disallow: */feed/search
138Disallow: */feed/sitemap/
139Disallow: */sitemap_.xml
140
141# Yandex-specific dynamic parameter deduplication (compact multi-param form).
142Clean-param: layout&callback&fiat&from&type&asset&packetId&shareCode
143Clean-param: r&return_to&productid
144Clean-param: _escaped_fragment_
145
146# ---------------------------------------------------------------------------
147# Default: all other crawlers (incl. AI crawlers, scrapers, etc.)
148# ---------------------------------------------------------------------------
149User-agent: *
150
151# Selective Allow overrides (carve-outs).
152Allow: /
153Allow: */bapi/fe/
154Allow: */sitemap_output/
155Allow: */funding-history/perpetual/
156Allow: */funding-history/quarterly/
157
158# Marketing/activity landing pages - explicitly allowed (documents intent)
159Allow: */activity/challenge-competition/
160Allow: */activity/collect-and-win/
161Allow: */activity/referral-entry/
162
163# Sensitive / internal paths - denied even to unlisted crawlers
164Disallow: *undefine*
165Disallow: *UNDEFINE*
166Disallow: */collected/undefined
167Disallow: */complaint-form/
168Disallow: */compliance-confirm/
169Disallow: */kyc-ui/
170Disallow: */my/
171Disallow: */deposit/result/order-v3/
172Disallow: */fiat/deposit/result/order
173Disallow: */survey/
174Disallow: */academy/courses/certificate/
175Disallow: */bapi/
176Disallow: */api/
177Disallow: */fapi/
178Disallow: */fvideo/
179Disallow: */mp-cms/app/
180Disallow: */mp-cms/disclaimer/
181Disallow: */feed/search
182Disallow: */feed/sitemap/
183Disallow: */sitemap_.xml
184
185# ---------------------------------------------------------------------------
186# Sitemaps (global - honored by all crawlers)
187# ---------------------------------------------------------------------------
188Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Trade_index.xml
189Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Blog_index.xml
190Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Nft_index.xml
191Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_SupportAndAnnouncement_index.xml
192Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Futures_index.xml
193Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Earn_index.xml
194Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Loan_index.xml
195Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Market_index.xml
196Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Default_index.xml
197Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Fiat_index.xml
198Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Crypto_index.xml
199Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Square_index.xml
200Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Academy_New_index.xml
201Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Muses_index.xml
202Sitemap: https://www.binance.com/sitemap_output/domain=www.binance.com/sitemap_Growth_index.xml
203
What do the bots actually get?
This page reads robots.txt, which states what binance.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on binance.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does binance.com block GPTBot?
No. binance.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does binance.com block ClaudeBot?
No. binance.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does binance.com block Google-Extended?
No. binance.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does binance.com block PerplexityBot?
No. binance.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find binance.com?
Yes. binance.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.