Who blocks AI · #492 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does walmart.com block AI crawlers?
No. walmart.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.walmart.com/robots.txt · 117 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.walmart.com/robots.txt · 117 lines
1#Sitemaps-https
2
3Sitemap: https://www.walmart.com/sitemap_category.xml
4Sitemap: https://www.walmart.com/sitemap_store_main.xml
5
6Sitemap: https://www.walmart.com/help/sitemap_gm.xml
7Sitemap: https://www.walmart.com/sitemap_browse_fst.xml
8Sitemap: https://www.walmart.com/sitemap_store_dept.xml
9
10Sitemap: https://www.walmart.com/sitemap_tp_legacy.xml
11Sitemap: https://www.walmart.com/sitemap_tp_at.xml
12Sitemap: https://www.walmart.com/sitemap_tp_br.xml
13Sitemap: https://www.walmart.com/sitemap_tp_kp.xml
14
15Sitemap: https://www.walmart.com/sitemap_brp_02.xml
16Sitemap: https://www.walmart.com/sitemap_brp_03.xml
17Sitemap: https://www.walmart.com/sitemap_brp_04.xml
18Sitemap: https://www.walmart.com/sitemap_brp_05.xml
19Sitemap: https://www.walmart.com/sitemap_brp_06.xml
20Sitemap: https://www.walmart.com/sitemap_brp_07.xml
21Sitemap: https://www.walmart.com/sitemap_brp_08.xml
22Sitemap: https://www.walmart.com/sitemap_brp_09.xml
23
24Sitemap: https://www.walmart.com/sitemap_itp_01.xml
25Sitemap: https://www.walmart.com/sitemap_itp_02.xml
26Sitemap: https://www.walmart.com/sitemap_product_03.xml
27
28Sitemap: https://www.walmart.com/sitemap_rp_01.xml
29Sitemap: https://www.walmart.com/sitemap_rp_02.xml
30Sitemap: https://www.walmart.com/sitemap_rp_03.xml
31Sitemap: https://www.walmart.com/sitemap_rp_04.xml
32Sitemap: https://www.walmart.com/sitemap_rp_05.xml
33
34Sitemap: https://www.walmart.com/sitemap_auto_dir.xml
35
36Sitemap: https://www.walmart.com/sitemap_brndshp_brand.xml
37
38Sitemap: https://www.walmart.com/sitemap_sp.xml
39
40Sitemap: https://www.walmart.com/sitemap_hi_ip.xml
41Sitemap: https://www.walmart.com/sitemap_hi_tp.xml
42Sitemap: https://www.walmart.com/sitemap_hi_bp.xml
43Sitemap: https://www.walmart.com/sitemap_hi_mp.xml
44
45
46#Disallow select URLs
47User-agent: *
48Disallow: /0/
49Disallow: /55875582/walmart-us/catalog/
50Disallow: /account/
51Disallow: /api/
52Disallow: /collection/api/logger
53Disallow: /cp/-201
54Disallow: /cp/-302
55Disallow: /cp/-306
56Disallow: /cp/-309
57Disallow: /cp/-506
58Disallow: /cp/-509
59Disallow: /cp/api/logger
60Disallow: /cp/api/wpa
61Disallow: /cservice/
62Disallow: /cservice/ya_index.gsp
63Disallow: /electrode/api/logger
64Disallow: /email_collect/
65Disallow: /giftregistry/
66Disallow: /msp
67Disallow: /nonConfig/api/wpa
68Disallow: /popup_security.jsp
69Disallow: /product/electrode/api/logger
70Disallow: /product/electrode/api/wpa
71Allow: /reviews/product/
72Allow: /reviews/seller/
73Disallow: /search
74Disallow: /search/api/wpa
75Disallow: */api/wpa
76Disallow: */midas/*
77Disallow: /search/search-ng.do
78Disallow: /solutions/
79Disallow: /store/ajax/detail-navigation
80Disallow: /store/ajax/local-store
81Disallow: /store/ajax/preferred
82Disallow: /store/ajax/search
83Disallow: /store/ajax/visited-store
84Disallow: /store/category/
85Disallow: /store/electrode/api/fetch-coupons
86Disallow: /store/electrode/api/logger
87Disallow: /store/electrode/api/p13n
88Disallow: /store/electrode/api/search
89Disallow: /store/electrode/api/stores
90Allow: /store/finder
91Disallow: /store/popular_in_grade/
92Disallow: /storeLocator/
93Disallow: /storeLocator/ca_storefinder_results.do
94Disallow: /tealeaf
95Disallow: /topic/electrode/api/logger
96Disallow: /topic/electrode/api/wpa
97Disallow: /typeahead/
98Disallow: /wmflows/
99Disallow: */undefined/*
100Disallow: /store/*/search
101Disallow: /c/kp/*/page/*
102Disallow: /feeds/*
103Disallow: /orders
104Disallow: /es/
105Disallow: /swag/*
106Disallow: /dad/trk
107Disallow: /sp/*
108
109#Crawler specific settings
110User-agent: Adsbot-Google
111
112User-agent: Mediapartners-Google
113
114# slow down Yahoo
115User-agent: Slurp
116Crawl-delay: 5
117
What do the bots actually get?
This page reads robots.txt, which states what walmart.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on walmart.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does walmart.com block GPTBot?
No. walmart.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does walmart.com block ClaudeBot?
No. walmart.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does walmart.com block Google-Extended?
No. walmart.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does walmart.com block PerplexityBot?
No. walmart.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find walmart.com?
Yes. walmart.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.