Who blocks AI · #342 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does salesforce.com block AI crawlers?
No. salesforce.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.salesforce.com/robots.txt · 338 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.salesforce.com/robots.txt · 338 lines
1# Robots.txt file for http://www.salesforce.com
2# All robots will spider the domain
3#
4Sitemap: https://www.salesforce.com/sitemap.xml
5Sitemap: https://www.salesforce.com/ap/sitemap.xml
6Sitemap: https://www.salesforce.com/au/sitemap.xml
7Sitemap: https://www.salesforce.com/br/sitemap.xml
8Sitemap: https://www.salesforce.com/ca/sitemap.xml
9Sitemap: https://www.salesforce.com/cn/sitemap.xml
10Sitemap: https://www.salesforce.com/de/sitemap.xml
11Sitemap: https://www.salesforce.com/dk/sitemap.xml
12Sitemap: https://www.salesforce.com/es/sitemap.xml
13Sitemap: https://www.salesforce.com/fi/sitemap.xml
14Sitemap: https://www.salesforce.com/fr/sitemap.xml
15Sitemap: https://www.salesforce.com/fr-ca/sitemap.xml
16Sitemap: https://www.salesforce.com/in/sitemap.xml
17Sitemap: https://www.salesforce.com/it/sitemap.xml
18Sitemap: https://www.salesforce.com/jp/sitemap.xml
19Sitemap: https://www.salesforce.com/kr/sitemap.xml
20Sitemap: https://www.salesforce.com/mx/sitemap.xml
21Sitemap: https://www.salesforce.com/nl/sitemap.xml
22Sitemap: https://www.salesforce.com/no/sitemap.xml
23Sitemap: https://www.salesforce.com/se/sitemap.xml
24Sitemap: https://www.salesforce.com/th/sitemap.xml
25Sitemap: https://www.salesforce.com/tw/sitemap.xml
26Sitemap: https://www.salesforce.com/uk/sitemap.xml
27Sitemap: https://www.salesforce.com/eu/sitemap.xml
28Sitemap: https://www.salesforce.com/sa/sitemap.xml
29Sitemap: https://www.salesforce.com/blog/sitemap_index.xml
30Sitemap: https://www.salesforce.com/au/blog/sitemap_index.xml
31Sitemap: https://www.salesforce.com/br/blog/post-sitemap.xml
32Sitemap: https://www.salesforce.com/ca/blog/post-sitemap.xml
33Sitemap: https://www.salesforce.com/de/blog/post-sitemap.xml
34Sitemap: https://www.salesforce.com/es/blog/post-sitemap.xml
35Sitemap: https://www.salesforce.com/fi/blog/sitemap.xml
36Sitemap: https://www.salesforce.com/fr/blog/post-sitemap.xml
37Sitemap: https://www.salesforce.com/it/blog/post-sitemap.xml
38Sitemap: https://www.salesforce.com/jp/blog/post-sitemap.xml
39Sitemap: https://www.salesforce.com/mx/blog/post-sitemap.xml
40Sitemap: https://www.salesforce.com/nl/blog/post-sitemap.xml
41Sitemap: https://www.salesforce.com/eu/blog/post-sitemap.xml
42Sitemap: https://www.salesforce.com/sitemap-forms.xml
43Sitemap: https://www.salesforce.com/plus-sitemap.xml
44Sitemap: https://www.salesforce.com/plus/api/sitemap.xml
45Sitemap: https://www.salesforce.com/nl/blog/sitemap_index.xml
46Sitemap: https://www.salesforce.com/sitemap-events.xml
47Sitemap: https://www.salesforce.com/au/blog/sf_topic-sitemap.xml
48Sitemap: https://www.salesforce.com/au/blog/page-sitemap.xml
49Sitemap: https://www.salesforce.com/au/blog/post-sitemap.xml
50Sitemap: https://www.salesforce.com/uk/news/sitemap_index.xml
51Sitemap: https://www.salesforce.com/news/page-sitemap.xml
52Sitemap: https://www.salesforce.com/news/post-sitemap.xml
53Sitemap: https://www.salesforce.com/news/sitemap_index.xml
54Sitemap: https://www.salesforce.com/in/blog/post-sitemap.xml
55Sitemap: https://www.salesforce.com/ap/blog/post-sitemap.xml
56
57# Keep mis-configured Microsoft SharePoint servers from hammering us
58# This is not MSN Search (msnbot), but privately owned SharePoint installations
59User-agent: MS Search 4.0 Robot
60Disallow: /
61
62User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot)
63Disallow: /
64
65User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
66Disallow: /
67
68User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot)
69Disallow: /
70
71User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 5.0 Robot)
72Disallow: /
73
74User-agent: *
75Disallow: /products/home-perf/
76Disallow: /form/conf/
77Disallow: */conf/
78Disallow: /content/dam/web/en_us/www/documents/
79Disallow: /content/dam/web/en_us/www/assets/pdf/
80Disallow: /login-messages/
81Disallow: /au/login-messages/
82Disallow: /ap/login-messages/
83Disallow: /in/login-messages/
84Disallow: /cn/login-messages/
85Disallow: /tw/login-messages/
86Disallow: /th/login-messages/
87Disallow: /uk/login-messages/
88Disallow: /de/login-messages/
89Disallow: /ca/login-messages/
90Disallow: /eu/login-messages/
91Disallow: /fr/login-messages/
92Disallow: /es/login-messages/
93Disallow: /it/login-messages/
94Disallow: /nl/login-messages/
95Disallow: /fi/login-messages/
96Disallow: /dk/login-messages/
97Disallow: /se/login-messages/
98Disallow: /fr-ca/login-messages/
99Disallow: /competitive/gated-content-for-agents/
100Allow: /plus/experience/*/series/
101Disallow: /plus/experience/*/
102Disallow: /plus/experience/dreamforce_2025_x_test
103# DPAUT-166: keep gated PDFs under /documents out of search crawlers (all locales/sites)
104Disallow: */wp-content/uploads/sites/*/documents/
105
106#Disallow DAMs
107Disallow: /content/dam/web/pt_br/www/
108Disallow: /content/dam/web/es_mx/www/
109Disallow: /content/dam/web/de_de/www/
110Disallow: /content/dam/web/fr_fr/www/
111Disallow: /content/dam/web/nl_nl/www/
112Disallow: /content/dam/web/es_es/www/
113Disallow: /content/dam/web/it_it/www/
114Disallow: /content/dam/web/sv_se/www/
115Disallow: /content/dam/web/fr_ca/www/
116Disallow: /content/dam/web/fi_fi/www/
117Disallow: /content/dam/web/da_dk/www/
118Disallow: /content/dam/web/en_is/www/
119Disallow: /content/dam/web/en_gb/www/
120Disallow: /content/dam/web/en_ie/www/
121Disallow: /content/dam/web/en_ca/www/
122Disallow: /content/dam/web/en_dk/www/
123Disallow: /content/dam/web/no_no/www/
124Disallow: /content/dam/web/de_ch/www/
125Disallow: /content/dam/web/fr_ch/www/
126Disallow: /content/dam/web/it_ch/www/
127Disallow: /content/dam/web/en_au/www/
128Disallow: /content/dam/web/en_sg/www/
129Disallow: /opencms/
130Disallow: /promos/
131Disallow: /styleguide/
132Disallow: /landing/
133Disallow: /modules/
134Disallow: /uk/promos/
135Disallow: /eu/promos/
136Disallow: /de/promos/
137Disallow: /es/promos/
138Disallow: /fr/promos/
139Disallow: /it/promos/
140Disallow: /jp/promos/
141
142# Company pages duped across locales
143Disallow: /us/developer/docs/ajaxpre
144Disallow: /us/developer/docs/ajax80
145Disallow: /us/developer/docs/ajax90
146Disallow: /us/developer/docs/ajax1*
147Disallow: /us/developer/docs/ajax2*
148Disallow: /us/developer/docs/ajax3*
149Disallow: /us/developer/docs/ajax4*
150Disallow: /us/developer/docs/apexcodepre
151Disallow: /us/developer/docs/apexcode1*
152Disallow: /us/developer/docs/apexcode2*
153Disallow: /us/developer/docs/apexcode3*
154Disallow: /us/developer/docs/apexcode4*
155Disallow: /us/developer/docs/api_asynchpre
156Disallow: /us/developer/docs/api_asynch1*
157Disallow: /us/developer/docs/api_asynch2*
158Disallow: /us/developer/docs/api_asynch3*
159Disallow: /us/developer/docs/api_asynch4*
160Disallow: /us/developer/docs/api_consolepre
161Disallow: /us/developer/docs/api_console2*
162Disallow: /us/developer/docs/api_console3*
163Disallow: /us/developer/docs/api_console4*
164Disallow: /us/developer/docs/api_metapre
165Disallow: /us/developer/docs/api_meta1*
166Disallow: /us/developer/docs/api_meta2*
167Disallow: /us/developer/docs/api_meta3*
168Disallow: /us/developer/docs/api_meta4*
169Disallow: /us/developer/docs/api_restpre
170Disallow: /us/developer/docs/api_rest2*
171Disallow: /us/developer/docs/api_rest3*
172Disallow: /us/developer/docs/api_rest4*
173Disallow: /us/developer/docs/api_streamingpre
174Disallow: /us/developer/docs/api_streaming2*
175Disallow: /us/developer/docs/api_streaming3*
176Disallow: /us/developer/docs/api_streaming4*
177Disallow: /us/developer/docs/apipre
178Disallow: /us/developer/docs/api80
179Disallow: /us/developer/docs/api90
180Disallow: /us/developer/docs/api1*
181Disallow: /us/developer/docs/api2*
182Disallow: /us/developer/docs/api3*
183Disallow: /us/developer/docs/api4*
184Disallow: /us/developer/docs/chatterapipre
185Disallow: /us/developer/docs/chatterapi2*
186Disallow: /us/developer/docs/chatterapi3*
187Disallow: /us/developer/docs/chatterapi4*
188Disallow: /us/developer/docs/dbcom_apipre
189Disallow: /us/developer/docs/dbcom_api2*
190Disallow: /us/developer/docs/dbcom_api3*
191Disallow: /us/developer/docs/dbcom_api4*
192Disallow: /us/developer/docs/dbcom_objectspre
193Disallow: /us/developer/docs/dbcom_objects2*
194Disallow: /us/developer/docs/dbcom_objects3*
195Disallow: /us/developer/docs/dbcom_objects4*
196Disallow: /us/developer/docs/dbcom_soql_soslpre
197Disallow: /us/developer/docs/dbcom_soql_sosl2*
198Disallow: /us/developer/docs/dbcom_soql_sosl3*
199Disallow: /us/developer/docs/dbcom_soql_sosl4*
200Disallow: /us/developer/docs/daaspre
201Disallow: /us/developer/docs/daas140
202Disallow: /us/developer/docs/fundamentalspre
203Disallow: /us/developer/docs/knowledge_devpre
204Disallow: /us/developer/docs/object_referencepre
205Disallow: /us/developer/docs/object_reference2*
206Disallow: /us/developer/docs/object_reference3*
207Disallow: /us/developer/docs/object_reference4*
208Disallow: /us/developer/docs/officetoolkit30
209Disallow: /us/developer/docs/pagespre
210Disallow: /us/developer/docs/pages1*
211Disallow: /us/developer/docs/pages2*
212Disallow: /us/developer/docs/pages3*
213Disallow: /us/developer/docs/pages4*
214Disallow: /us/developer/docs/sforce20
215Disallow: /us/developer/docs/sforce30
216Disallow: /us/developer/docs/sforce40
217Disallow: /us/developer/docs/sforce50
218Disallow: /us/developer/docs/sforce60
219Disallow: /us/developer/docs/sforce70
220
221
222Disallow: /search.jsp
223Disallow: /assets/pdf/company/private/
224Disallow: /jp/assets/pdf/
225Disallow: /content/dam/web/ja_jp/www/documents/
226Disallow: /content/dam/web/ja_jp/www/assets/pdf/
227Disallow: /content/blogs/au/en/*
228Disallow: /content/blogs/us/en/*
229Disallow: /content/blogs/nl/nl/*
230Disallow: /content/blogs/eu/en/*
231Disallow: /content/blogs/in/en/*
232Disallow: /content/blogs/fi/fi/*
233Disallow: /content/blogs/sg/en/*
234Disallow: /content/blogs/jp/ja/*
235Disallow: /content/blogs/mx/es/*
236Disallow: /content/blogs/es/es/*
237Disallow: /content/blogs/br/pt/*
238Disallow: /content/blogs/il/he/*
239Disallow: /content/blogs/it/it/*
240
241User-agent: AdsBot-Google
242Allow: /
243Disallow: /de/assets/
244Disallow: /fr/assets/
245Disallow: /se/assets/
246Disallow: /es/assets/
247Disallow: /it/assets/
248Disallow: /nl/assets/
249Disallow: /uk/assets/
250Disallow: /eu/assets/
251Disallow: /content/dam/web/en_us/www/documents/legal/misc/456456.pdf
252Disallow: /content/dam/web/en_us/www/documents/legal/misc/usla-excluded-services.pdf
253Disallow: /content/dam/web/en_us/www/documents/legal/supplier/salesforce-vendor-privacy-exhibit-emea_psa_subcontractors-archived-121118.pdf
254Disallow: /content/dam/web/en_us/www/documents/legal/Agreements/product-specific-terms/premier-priority-terms.pdf
255Disallow: /dig-perf-test/
256Disallow: /content/dam/web/en_us/www/documents/legal/no-index/
257Disallow: /content/dam/web/en_us/www/documents/legal/work-unit-catalog/
258Disallow: /*/wp-content/uploads/sites/*/documents/legal/no-index/*.pdf$
259Disallow: /*/wp-content/uploads/sites/*/partners/*.pdf$
260
261# STARTS
262# Block crawl of ._filter.alphaSort which is duplicate of /customer-success-stories/
263# Note the $ delimiter so that this doesnt impact other URLs based on this stem:
264Disallow: */customer-success-stories._filter.alphaSort$
265Disallow: */customer-success-stories/._filter.alphaSort$
266
267# Block all access to URLs using popularSort:
268Disallow: */customer-success-stories._filter.popularSort
269Disallow: */customer-success-stories/._filter.popularSort
270
271# Block all access to URLs using newestSort:
272Disallow: */customer-success-stories._filter.newestSort
273Disallow: */customer-success-stories/._filter.newestSort
274
275# Block crawl where 2 or more categories are used with services filter. The final . surrounded by * should match any multi-category filter URL:
276# Uncomment next line to apply to all locales
277Disallow: */customer-success-stories._filter.alphaSort.S*.*
278Disallow: */customer-success-stories/._filter.alphaSort.S*.*
279#added new Deny rules to block bots to crawl missed filter URLs.
280user-agent: *
281Disallow: */customer-success-stories/*.*._filter
282Disallow: */customer-success-stories.*._filter
283Disallow: */customer-success-stories.[a-z]*Sort.*
284Disallow: */customer-success-stories._filter.[a-z]*Sort.*
285
286# Block crawl where 2 or more categories are used with products filter. The final . surrounded by * should match any multi-category filter URL:
287Disallow: */customer-success-stories._filter.alphaSort.P*.*
288Disallow: */customer-success-stories/._filter.alphaSort.P*.*
289
290# Block crawl where 2 or more categories are used with industries filter. The final . surrounded by * should match any multi-category filter URL:
291Disallow: */customer-success-stories._filter.alphaSort.I*.*
292Disallow: */customer-success-stories/._filter.alphaSort.I*.*
293
294# Block crawl where 2 or more categories are used with business size filter. The final . surrounded by * should match any multi-category filter URL:
295Disallow: */customer-success-stories._filter.alphaSort.BS*.*
296Disallow: */customer-success-stories/._filter.alphaSort.BS*.*
297
298# Block crawl where 2 or more categories are used with business type filter. The final . surrounded by * should match any multi-category filter URL:
299Disallow: */customer-success-stories._filter.alphaSort.BT*.*
300Disallow: */customer-success-stories/._filter.alphaSort.BT*.*
301# ENDS
302
303# Rules will block when 2 or more facets are activated, but allow single facets to be crawled:
304Disallow: *services/success-plans/accelerators._filter.S*.*
305Disallow: */services/success-plans/accelerators._filter.RG*.*
306Disallow: */services/success-plans/accelerators._filter.P*.*
307Disallow: *services/success-plans/accelerators/._filter.S*.*
308Disallow: */services/success-plans/accelerators/._filter.RG*.*
309Disallow: */services/success-plans/accelerators/._filter.P*.*
310
311# First 2 rules blocks the duplicate index, $ delimiter avoids picking up valid pagination URLs:
312Disallow: */services/learn/classes/._filter.alphaSort/$
313Disallow: */services/learn/classes._filter.alphaSort/$
314
315# Next rules will fire when more than one facet is activated, or when a subpage of facet is requested, but allow individual facets to be crawled:
316Disallow: */services/learn/classes._filter.alphaSort.re*.*
317Disallow: */services/learn/classes._filter.alphaSort.P*.*/
318Disallow: */services/learn/classes/._filter.alphaSort.re*.*
319Disallow: */services/learn/classes/._filter.alphaSort.P*.*/
320
321# Blocking Acunetix
322User-agent: Acunetix Web Vulnerability Scanner
323User-agent: Acunetix Security Scanner
324Disallow: /products/commerce-cloud/partner-marketplace/.*
325Disallow: /products/einstein/ai-research.*
326Disallow: /blog/.*
327Disallow: /search/.*
328Disallow: /*.pdf$
329# Block crawl of .shtml pages
330Disallow: /*.shtml$
331Disallow: /conf/
332Disallow: /search/
333Disallow: /content/dam/
334Disallow: /*._filter.*
335Disallow: /is/
336Disallow: /form/pdf/*
337Disallow: /speed-test-parent/
338Disallow: /blog/.html.*
What do the bots actually get?
This page reads robots.txt, which states what salesforce.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on salesforce.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does salesforce.com block GPTBot?
No. salesforce.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does salesforce.com block ClaudeBot?
No. salesforce.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does salesforce.com block Google-Extended?
No. salesforce.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does salesforce.com block PerplexityBot?
No. salesforce.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find salesforce.com?
Yes. salesforce.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.