Who blocks AI · #31 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does amazon.com block AI crawlers?
Yes. amazon.com’s robots.txt blocks 17 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 14 more. It also blocks PetalBot among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.amazon.com/robots.txt · 439 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 175 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 277 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 151 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 253 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 205 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Blocked | line 241 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 144 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 163 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 154 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 148 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Blocked | line 346 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Blocked | line 166 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 169 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 274 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 247 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 256 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 193 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Blocked | line 217 · Disallow: / under User-agent: meta-externalfetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.amazon.com/robots.txt · 439 lines
1User-agent: *
2Disallow: /exec/obidos/account-access-login
3Disallow: /exec/obidos/change-style
4Disallow: /exec/obidos/flex-sign-in
5Disallow: /exec/obidos/handle-buy-box
6Disallow: /exec/obidos/tg/cm/member/
7Disallow: /gp/aw/help/id=sss
8Disallow: /gp/cart
9Disallow: /gp/flex
10Disallow: /gp/product/e-mail-friend
11Disallow: /gp/product/product-availability
12Disallow: /gp/product/rate-this-item
13Disallow: /gp/sign-in
14Disallow: /gp/reader
15Disallow: /gp/sitbv3/reader
16Disallow: /gp/richpub/syltguides/create
17Disallow: /gp/gfix
18Disallow: /gp/associations/wizard.html
19Disallow: /gp/dmusic/order
20Disallow: /gp/legacy-handle-buy-box.html
21Disallow: /gp/aws/ssop
22Disallow: /gp/yourstore
23Disallow: /gp/gift-central/organizer/add-wishlist
24Disallow: /gp/vote
25Disallow: /gp/voting/
26Disallow: /gp/music/wma-pop-up
27Disallow: /gp/customer-images
28Disallow: /gp/richpub/listmania/createpipeline
29Disallow: /gp/content-form
30Disallow: /gp/pdp/invitation/invite
31Disallow: /gp/customer-reviews/common/du
32Disallow: /gp/customer-reviews/write-a-review.html
33Disallow: /gp/associations/wizard.html
34Disallow: /gp/music/clipserve
35Disallow: /gp/customer-media/upload
36Disallow: /gp/history
37Disallow: /gp/item-dispatch
38Disallow: /gp/dmusic/order/handle-buy-box.html
39Disallow: /gp/recsradio
40Disallow: /gp/slredirect
41Disallow: /dp/shipping/
42Disallow: /dp/twister-update/
43Disallow: /dp/manual-submit/
44Disallow: /dp/e-mail-friend/
45Disallow: /dp/product-availability/
46Disallow: /dp/rate-this-item/
47Disallow: /gp/registry/wishlist/*/reserve
48Disallow: /gp/structured-ratings/actions/get-experience.html
49Disallow: /gp/twitter/
50Disallow: /ap/signin
51Disallow: /gp/registry/wishlist/
52Disallow: /wishlist/
53Allow: /wishlist/universal*
54Allow: /wishlist/vendor-button*
55Allow: /wishlist/get-button*
56Disallow: /gp/wishlist/
57Allow: /gp/wishlist/universal*
58Allow: /gp/wishlist/vendor-button*
59Allow: /gp/wishlist/ipad-install*
60Disallow: /registry/wishlist/
61Disallow: /review/common/du
62Disallow: /gp/registry/search.html
63Disallow: /product-reviews/B0069IY63Y
64Disallow: /gp/orc/rml/
65Disallow: */gcrnsts
66Disallow: /gp/gc/widget
67Disallow: /gp/dmusic/mp3/player
68Disallow: /gp/entity-alert/external
69Disallow: /gp/customer-reviews/dynamic/sims-box
70Disallow: /review/dynamic/sims-box
71Disallow: /gp/redirect.html
72Disallow: /gp/twister/ajaxv2
73Disallow: /ss/twister/ajax
74Disallow: /b?*node=7454917011
75Disallow: /b?*node=7454927011
76Disallow: /b?*node=7454939011
77Disallow: /b?*node=7454898011
78Disallow: /gp/customer-media/actions/delete/
79Disallow: /gp/customer-media/actions/edit-caption/
80Disallow: /gp/dmusic/
81Allow: /gp/dmusic/promotions/PrimeMusic
82Allow: /gp/dmusic/promotions/AmazonMusicUnlimited
83Disallow: /gp/offer-listing/
84Disallow: /b?*node=9052533011
85Disallow: /lm/R1XIHQVKXSKBNJ
86Disallow: /lm/R3HQ5WJSZK6QSO
87Disallow: /surprise/
88Disallow: /local/ajax/
89Disallow: /gp/socialmedia/giveaways
90Disallow: /gp/b2b-rd
91Disallow: /gp/aw/so.html
92Disallow: /reviews/iframe
93Disallow: /gp/switch-language
94Disallow: /ga/p/
95Disallow: /gp/profile/
96Disallow: /giveaway/host/setup/
97Disallow: /ss/customer-reviews/lighthouse/
98Disallow: /ospublishing/story/*
99Disallow: /gp/aw/ol/
100Disallow: /gp/promotion/
101Disallow: /hz/leaderboard/top-reviewers/
102Disallow: /creatorhub
103Disallow: /creatorhub/*
104Disallow: /slp/s$
105Disallow: /-/
106Allow: /-/es/
107Allow: /-/en$
108Allow: /-/zh_TW/
109Allow: /-/zh_TW$
110Allow: /-/he/
111Allow: /-/he$
112Disallow: /hz/help/contact/*/message/$
113Disallow: /gp/aw/shoppingAids/
114Disallow: /rss/people/*/reviews
115Disallow: /gp/pdp/rss/*/reviews
116Disallow: /gp/cdp/member-reviews/
117Disallow: /gp/aw/cr/
118Allow: /gp/offer-listing/B000
119Allow: /gp/offer-listing/9000
120Disallow: /gp/aag
121Allow: /gp/aag/main?*seller=ABVFEJU8LS620
122Disallow: /gp/pdp/profile/
123Disallow: /gp/help/customer/express/c2c/
124Disallow: /slp/*/b$
125Disallow: /hz/contact-us/ajax/initiate-trusted-contact/
126Disallow: /hp/video/
127Disallow: /gp/video/api
128Disallow: /gp/video/settings
129Disallow: /gp/video/library
130Disallow: /gp/video/watchlist
131Disallow: /gp/video/mystuff
132Disallow: /gp/video/profiles
133Disallow: /gp/video/search
134Disallow: /gp/video/auth
135Disallow: /gp/video/salp/i
136Disallow: /icdp/
137Disallow: /shoptheshow/*?cat=
138Disallow: /shoptheshow/*&cat=
139
140User-agent: EtaoSpider
141Disallow: /
142
143User-agent: GPTBot
144Disallow: /
145
146
147User-agent: CCBot
148Disallow: /
149
150User-Agent: PerplexityBot
151Disallow: /
152
153User-agent: Google-Extended
154Disallow: /
155
156User-agent: GoogleAgent-Mariner
157Disallow: /
158
159User-agent: GoogleAgent-Shopping
160Disallow: /
161
162User-agent: ClaudeBot
163Disallow: /
164
165User-agent: meta-externalagent
166Disallow: /
167
168User-agent: Bytespider
169Disallow: /
170
171User-agent: Scrapy
172Disallow: /
173
174User-agent: PetalBot
175Disallow: /
176
177User-agent: Devin
178Disallow: /
179
180User-agent: omgili
181Disallow: /
182
183User-agent: AI2Bot
184Disallow: /
185
186User-agent: Gemini-Deep-Research
187Disallow: /
188
189User-agent: PanguBot
190Disallow: /
191
192User-agent: MistralAI-User
193Disallow: /
194
195User-agent: Diffbot
196Disallow: /
197
198User-agent: Sidetrade indexer bot
199Disallow: /
200
201User-agent: Google-NotebookLM
202Disallow: /
203
204User-agent: DuckAssistBot
205Disallow: /
206
207User-agent: Copilot
208Disallow: /
209
210User-agent: Ai2Bot-Dolma
211Disallow: /
212
213User-agent: Google-CloudVertexBot
214Disallow: /
215
216User-agent: meta-externalfetcher
217Disallow: /
218
219User-agent: iaskspider
220Disallow: /
221
222User-agent: wpbot
223Disallow: /
224
225User-agent: aiHitBot
226Disallow: /
227
228User-agent: cohere-ai
229Disallow: /
230
231User-agent: FriendlyCrawler
232Disallow: /
233
234User-agent: img2dataset
235Disallow: /
236
237User-agent: VelenPublicWebCrawler
238Disallow: /
239
240User-agent: YouBot
241Disallow: /
242
243User-agent: Brightbot
244Disallow: /
245
246User-agent: Claude-User
247Disallow: /
248
249User-agent: ClaudeBot
250Disallow: /
251
252User-agent: Claude-SearchBot
253Disallow: /
254
255User-agent: Perplexity-User
256Disallow: /
257
258User-agent: ISSCyberRiskCrawler
259Disallow: /
260
261User-agent: CopilotNative
262Disallow: /
263
264User-agent: CopilotSapphire
265Disallow: /
266
267User-agent: imgproxy
268Disallow: /
269
270User-agent: SemrushBot-SWA
271Disallow: /
272
273User-agent: ChatGPT-User
274Disallow: /
275
276User-agent: OAI-SearchBot
277Disallow: /
278
279User-agent: Timpibot
280Disallow: /
281
282User-agent: Thinkbot
283Disallow: /
284
285User-agent: AddSearchBot
286Disallow: /
287
288User-agent: AI2Bot-DeepResearchEval
289Disallow: /
290
291User-agent: Andibot
292Disallow: /
293
294User-agent: Anomura
295Disallow: /
296
297User-agent: atlassian-bot
298Disallow: /
299
300User-agent: BuddyBot
301Disallow: /
302
303User-agent: Channel3Bot
304Disallow: /
305
306User-agent: ChatGLM-Spider
307Disallow: /
308
309User-agent: Claude-Web
310Disallow: /
311
312User-agent: Cloudflare-AutoRAG
313Disallow: /
314
315User-agent: CloudflareBrowserRenderingCrawler
316Disallow: /
317
318User-agent: cohere-training-data-crawler
319Disallow: /
320
321User-agent: Cotoyogi
322Disallow: /
323
324User-agent: Crawl4AI
325Disallow: /
326
327User-agent: Datenbank Crawler
328Disallow: /
329
330User-agent: DeepSeekBot
331Disallow: /
332
333User-agent: Echobot Bot
334Disallow: /
335
336User-agent: EchoboxBot
337Disallow: /
338
339User-agent: factset_spyderbot
340Disallow: /
341
342User-agent: Google-Firebase
343Disallow: /
344
345User-agent: GoogleOther
346Disallow: /
347
348User-agent: Grok-DeepSearch
349Disallow: /
350
351User-agent: GrokBot
352Disallow: /
353
354User-agent: iAskBot
355Disallow: /
356
357User-agent: IbouBot
358Disallow: /
359
360User-agent: ICC-Crawler
361Disallow: /
362
363User-agent: ImagesiftBot
364Disallow: /
365
366User-agent: imageSpider
367Disallow: /
368
369User-agent: Kangaroo Bot
370Disallow: /
371
372User-agent: KlaviyoAIBot
373Disallow: /
374
375User-agent: KunatoCrawler
376Disallow: /
377
378User-agent: laion-huggingface-processor
379Disallow: /
380
381User-agent: Linguee Bot
382Disallow: /
383
384User-agent: LinkupBot
385Disallow: /
386
387User-agent: Manus-User
388Disallow: /
389
390User-agent: meta-webindexer
391Disallow: /
392
393User-agent: netEstate Imprint Crawler
394Disallow: /
395
396User-agent: Panscient
397Disallow: /
398
399User-agent: panscient.com
400Disallow: /
401
402User-agent: PhindBot
403Disallow: /
404
405User-agent: Poggio-Citations
406Disallow: /
407
408User-agent: QualifiedBot
409Disallow: /
410
411User-agent: SBIntuitionsBot
412Disallow: /
413
414User-agent: ShapBot
415Disallow: /
416
417User-agent: TavilyBot
418Disallow: /
419
420User-agent: TerraCotta
421Disallow: /
422
423User-agent: TwinAgent
424Disallow: /
425
426User-agent: webzio-extended
427Disallow: /
428
429User-agent: WRTNBot
430Disallow: /
431
432User-agent: xAI-Grok
433Disallow: /
434
435User-agent: YaK
436Disallow: /
437
438User-agent: ZanistaBot
439Disallow: /
What do the bots actually get?
This page reads robots.txt, which states what amazon.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on amazon.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does amazon.com block GPTBot?
Yes. amazon.com’s robots.txt disallows GPTBot from the whole site (line 144: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does amazon.com block ClaudeBot?
Yes. amazon.com’s robots.txt disallows ClaudeBot from the whole site (line 163: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does amazon.com block Google-Extended?
Yes. amazon.com’s robots.txt disallows Google-Extended from the whole site (line 154: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does amazon.com block PerplexityBot?
Yes. amazon.com’s robots.txt disallows PerplexityBot from the whole site (line 151: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find amazon.com?
Not through its own crawler. amazon.com’s robots.txt disallows OAI-SearchBot from the whole site (line 277: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.