Who blocks AI · #933 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does target.com block AI crawlers?
No. target.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.target.com/robots.txt · 123 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.target.com/robots.txt · 123 lines
1Sitemap: https://www.target.com/sitemap_keywords-index.xml.gz
2Sitemap: https://www.target.com/sitemap_stores-index.xml.gz
3Sitemap: https://www.target.com/sitemap_taxonomy-categories-index.xml.gz
4Sitemap: https://www.target.com/sitemap_pdp-index.xml.gz
5Sitemap: https://www.target.com/sitemap_taxonomy-brand-index.xml.gz
6Sitemap: https://www.target.com/sitemap_facet-categories-index.xml.gz
7
8User-agent: *
9Disallow: /*/Ntk
10Disallow: /*/Ntt
11Disallow: /*/Ntx
12Disallow: /*%7Cd_
13Disallow: /*/schoollist/
14Disallow: /*BTWN
15Disallow: /[path]/
16Disallow: /7078046/
17Disallow: /7079046/
18Disallow: /AddToList
19Disallow: /AddToRegistry
20Disallow: /admin
21Disallow: /advancedGiftRegistrySearchView
22Disallow: /AjaxSearchNavigationView
23Disallow: /Allons_voter
24Disallow: /bp/c/
25Disallow: /bp/guest_mfg_brand
26Disallow: /bp/p/
27Disallow: /CallToActionModalView
28Disallow: /cgi-bin
29Disallow: /cgi-local
30Disallow: /Checkout
31Disallow: /CheckoutEditItemsDisplayView
32Disallow: /CheckoutOrderBillingView
33Disallow: /CheckoutOrderShippingView
34Disallow: /CheckoutSignInView
35Disallow: /co-
36Disallow: /common
37Disallow: /coupons.
38Disallow: /custom-reviews/
39Disallow: /data
40Disallow: /database/philboard.mdb
41Disallow: /deals/
42Disallow: /dir_on_server/
43Disallow: /EmailCartView
44Disallow: /EnlargedImageView
45Disallow: /ESPDisplayOptionsViewCmd
46Disallow: /ESPModal
47Disallow: /ExitCheckoutCmd
48Disallow: /FeaturedShowMoreOverlay
49Disallow: /FetchProdRefreshContent
50Disallow: /fiats
51Disallow: /FiatsCmd
52Disallow: /file
53Disallow: /FreeGiftDisplayView
54Disallow: /gam-
55Disallow: /GenericRegistryPortalView
56Disallow: /gc?k
57Disallow: /GiftRegistrySearchViewCmd
58Disallow: /gp/
59Disallow: /GuestAsAnonymous
60Disallow: /guestEmailNotificationView
61Disallow: /HelpContent
62Disallow: /icons/
63Disallow: /igp
64Disallow: /index.jhtml
65Disallow: /keyword=
66Disallow: /legal-contact-us/
67Disallow: /list.id=1
68Disallow: /LogonForm
69Disallow: /m/
70Disallow: /ManageOrder
71Disallow: /ManageReturns
72Disallow: /MediaDisplayView
73Disallow: /mm/
74Disallow: /moreinfo.cfm
75Disallow: /news
76Disallow: /np/
77Disallow: /OpenZoomLayer
78Disallow: /OrderItemDisplay
79Disallow: /OtherDisplayView
80Disallow: /p/premium-registry
81Disallow: /PhotoUpload
82Disallow: /pl/
83Disallow: /ProductComparisonCmd
84Disallow: /ProductDetailsTabView
85Disallow: /PromotionDetailsDisplayView
86Disallow: /PromotionDisplayView
87Disallow: /PXGWPp4wUS
88Disallow: /qi/
89Disallow: /QuickInfoView
90Disallow: /ready_sit_read/index.jhtml
91Disallow: /RegistryPortalCmd
92Disallow: /ReportAbuse
93Disallow: /reviewVote
94Disallow: /script
95Disallow: /SearchNavigationView
96Disallow: /shop/
97Disallow: /SingleShipmentOrderSummaryView
98Disallow: /SOImapPriceDisplayView
99Disallow: /SpecificationDefinitionView
100Disallow: /splitOrderItem
101Disallow: /store-locator/search-results-print
102Disallow: /supertarget/index.jhtml
103Disallow: /target_baby/
104Disallow: /target_group
105Disallow: /targetdirect_group/
106Disallow: /TargetListPortalView
107Disallow: /TargetStoreLocatorCmd
108Disallow: /tdir/p/kids-back-to-school/
109Disallow: /tsa/
110Disallow: /VariationSelectionView
111Disallow: /webapp
112Disallow: /winnt/
113Disallow: /WriteComments
114Disallow: /WriteReviews
115Disallow: /XCSA/
116Disallow: /yr
117Disallow: /s?
118Disallow: /cart
119Disallow: /account/
120Disallow: /tracking
121Disallow: /config
122Disallow: /social/
123
What do the bots actually get?
This page reads robots.txt, which states what target.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on target.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does target.com block GPTBot?
No. target.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does target.com block ClaudeBot?
No. target.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does target.com block Google-Extended?
No. target.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does target.com block PerplexityBot?
No. target.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find target.com?
Yes. target.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.