Who blocks AI · #630 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does sap.com block AI crawlers?
Yes. sap.com’s robots.txt blocks 6 of the 19 AI crawlers we track, including Meta-ExternalAgent, Applebot-Extended, Bytespider and 3 more. It also blocks PetalBot and YandexBot among search engines.
GPTBotPartly blockedOAI-SearchBotPartly blockedClaudeBotPartly blockedPerplexityBotPartly blockedGoogle-ExtendedPartly blockedCCBotPartly blocked- robots.txt
- www.sap.com/robots.txt · 156 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 51 · Disallow: /bin/sapdx/ line 52 · Disallow: /bin/sapdxc/ and 27 more | |
Bingbot | Microsoft | Partly blocked | line 51 · Disallow: /bin/sapdx/ line 52 · Disallow: /bin/sapdxc/ and 27 more |
DuckDuckBot | DuckDuckGo | Partly blocked | line 51 · Disallow: /bin/sapdx/ line 52 · Disallow: /bin/sapdxc/ and 27 more |
Applebot | Apple | Partly blocked | line 51 · Disallow: /bin/sapdx/ line 52 · Disallow: /bin/sapdxc/ and 27 more |
PetalBot | Huawei | Blocked | line 18 · Disallow: / under User-agent: * |
YandexBot | Yandex | Blocked | line 18 · Disallow: / under User-agent: * |
Baiduspider | Baidu | Partly blocked | line 51 · Disallow: /bin/sapdx/ line 52 · Disallow: /bin/sapdxc/ and 27 more |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Partly blocked | line 51 · Disallow: /bin/sapdx/ line 52 · Disallow: /bin/sapdxc/ and 27 more |
PerplexityBot | Perplexity | Partly blocked | line 51 · Disallow: /bin/sapdx/ line 52 · Disallow: /bin/sapdxc/ and 27 more |
Claude-SearchBot | Anthropic | Partly blocked | line 91 · Disallow: /fetch line 92 · Disallow: /proxy and 30 more |
DuckAssistBot | DuckDuckGo | Blocked | line 18 · Disallow: / under User-agent: * |
Amazonbot | Amazon | Partly blocked | line 91 · Disallow: /fetch line 92 · Disallow: /proxy and 30 more |
YouBot | You.com | Blocked | line 18 · Disallow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Partly blocked | line 91 · Disallow: /fetch line 92 · Disallow: /proxy and 30 more |
ClaudeBot | Anthropic | Partly blocked | line 91 · Disallow: /fetch line 92 · Disallow: /proxy and 30 more |
Google-Extended | Partly blocked | line 51 · Disallow: /bin/sapdx/ line 52 · Disallow: /bin/sapdxc/ and 27 more | |
CCBot | Common Crawl | Partly blocked | line 51 · Disallow: /bin/sapdx/ line 52 · Disallow: /bin/sapdxc/ and 27 more |
Applebot-Extended | Apple | Blocked | line 18 · Disallow: / under User-agent: * |
GoogleOther | Partly blocked | line 51 · Disallow: /bin/sapdx/ line 52 · Disallow: /bin/sapdxc/ and 27 more | |
Meta-ExternalAgent | Meta | Blocked | line 18 · Disallow: / under User-agent: * |
Bytespider | ByteDance | Blocked | line 18 · Disallow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Partly blocked | line 91 · Disallow: /fetch line 92 · Disallow: /proxy and 30 more |
Claude-User | Anthropic | Partly blocked | line 91 · Disallow: /fetch line 92 · Disallow: /proxy and 30 more |
Perplexity-User | Perplexity | Partly blocked | line 51 · Disallow: /bin/sapdx/ line 52 · Disallow: /bin/sapdxc/ and 27 more |
MistralAI-User | Mistral | Partly blocked | line 51 · Disallow: /bin/sapdx/ line 52 · Disallow: /bin/sapdxc/ and 27 more |
Meta-ExternalFetcher | Meta | Blocked | line 18 · Disallow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 18 · Disallow: / under User-agent: * |
SemrushBot | Semrush | Blocked | line 18 · Disallow: / under User-agent: * |
MJ12bot | Majestic | Blocked | line 18 · Disallow: / under User-agent: * |
DotBot | Moz | Blocked | line 18 · Disallow: / under User-agent: * |
View www.sap.com/robots.txt · 156 lines
1#
2# Welcome to SAP SE
3# ____ _ ____
4#/ ___| / \ | _ \
5#\___ \ / _ \ | |_) |
6# ___) / ___ \| __/
7#|____/_/ \_\_|
8#
9# robots.txt for https://www.sap.com
10#
11# https://www.sap.com/about.html
12#
13# Find your next job: https://www.sap.com/careers
14#
15# Last updated 2026-10-20
16
17User-agent: *
18Disallow: /
19
20# Search Engines (all share the same rule set)
21User-agent: Googlebot
22User-agent: Googlebot-Image
23User-agent: Googlebot-News
24User-agent: Googlebot-Video
25User-agent: GoogleOther
26User-agent: Google-InspectionTool
27User-agent: Storebot-Google
28User-agent: Bingbot
29User-agent: 360Spider
30User-agent: Baiduspider
31User-agent: coccocbot
32User-agent: Daum
33User-agent: DuckDuckBot
34User-agent: Ecosia_bot
35User-agent: MojeekBot
36User-agent: Yeti
37User-agent: SeznamBot
38User-agent: Sogou web spider
39User-agent: Yahoo! Slurp
40User-agent: YandexAccessibilityBot
41User-agent: YandexMobileBot
42User-agent: Yandex
43User-agent: CCBot
44User-agent: Google-Extended
45User-agent: FacebookBot
46User-agent: MistralAI-User
47User-agent: OAI-SearchBot
48User-agent: PerplexityBot
49User-agent: Perplexity-User
50User-agent: archive.org_bot
51Disallow: /bin/sapdx/
52Disallow: /bin/sapdxc/
53Disallow: /design-system/digital/
54Disallow: /dmc/
55Disallow: /heartbeat.html
56Disallow: /investors/de/stock/fixed-income/bond-documentation*
57Disallow: /investors/de/stock/fixed-income/cp-documentation*
58Disallow: /investors/en/stock/fixed-income/bond-documentation*
59Disallow: /investors/en/stock/fixed-income/cp-documentation*
60Disallow: */_jcr_content*.html
61Disallow: */abm/*
62Disallow: */cmp/dg/gdpr/*
63Disallow: */cmp/dg/gdpr-emea/*
64Disallow: */cmp/dg/gdpr-germany/*
65Disallow: */cmp/eml/*
66Disallow: */cmp/oth/crm-wx18-xrg-geb-fba/*
67Disallow: */cmp/oth/crm-wx18-xrg-geb-supfba/*
68Disallow: */cmp/oth/gdpr-opt-out/*
69Disallow: */cmp/spm/*
70Disallow: */cmp/td/sap-sustainability-data-exchange*
71Disallow: */events/sapphire/orlando/celebration-night-concert.html
72Disallow: */events/sponsorships.html
73Disallow: */legal/video-surveillance*
74Disallow: */products/erp/s4hana-private-edition/trial.html
75Disallow: */tye.html
76Disallow: */typ.html
77Disallow: */registration/*?
78Disallow: /silent.html
79Disallow: /*/silent.html
80Allow: /abm/dod.html
81Allow: /abm/dod/*
82Allow: /dmc/exp/*
83
84# AI Agents (all share the same rule set)
85User-agent: ClaudeBot
86User-agent: Claude-User
87User-agent: Claude-SearchBot
88User-agent: GPTBot
89User-agent: ChatGPT-User
90User-agent: Amazonbot
91Disallow: /fetch
92Disallow: /proxy
93Disallow: /sharedlibs/
94Disallow: /bin/sapdx/
95Disallow: /bin/sapdxc/
96Disallow: /design-system/digital/
97Disallow: /dmc/
98Disallow: /heartbeat.html
99Disallow: /investors/de/stock/fixed-income/bond-documentation*
100Disallow: /investors/de/stock/fixed-income/cp-documentation*
101Disallow: /investors/en/stock/fixed-income/bond-documentation*
102Disallow: /investors/en/stock/fixed-income/cp-documentation*
103Disallow: */_jcr_content*.html
104Disallow: */abm/*
105Disallow: */cmp/dg/gdpr/*
106Disallow: */cmp/dg/gdpr-emea/*
107Disallow: */cmp/dg/gdpr-germany/*
108Disallow: */cmp/eml/*
109Disallow: */cmp/oth/crm-wx18-xrg-geb-fba/*
110Disallow: */cmp/oth/crm-wx18-xrg-geb-supfba/*
111Disallow: */cmp/oth/gdpr-opt-out/*
112Disallow: */cmp/spm/*
113Disallow: */cmp/td/sap-sustainability-data-exchange*
114Disallow: */events/sapphire/orlando/celebration-night-concert.html
115Disallow: */events/sponsorships.html
116Disallow: */legal/video-surveillance*
117Disallow: */products/erp/s4hana-private-edition/trial.html
118Disallow: */tye.html
119Disallow: */typ.html
120Disallow: */registration/*?
121Disallow: /silent.html
122Disallow: /*/silent.html
123Allow: /abm/dod.html
124Allow: /abm/dod/*
125Allow: /dmc/exp/*
126
127# Fetcher and other
128User-agent: AdsBot-Google
129User-agent: AdsBot-Google-Mobile
130User-agent: BrightEdge
131User-agent: Chrome-Lighthouse
132User-agent: facebookexternalhit
133User-agent: Google-Read-Aloud
134User-agent: Google-Safety
135User-agent: Hatena
136User-agent: linkedinbot
137User-agent: Mediapartners-Google
138User-agent: Pinterestbot
139User-agent: SchemaBot
140User-agent: SiteAuditBot
141User-agent: SiteimproveBot
142User-agent: Slackbot-LinkExpanding
143User-agent: Telegram
144User-agent: Twitterbot
145User-agent: Prerender
146Allow: /
147
148Sitemap: https://www.sap.com/sitemap_index.xml
149Sitemap: https://www.sap.com/sitemap-index.xml
150Sitemap: https://www.sap.com/sitemap-documents-index.xml
151Sitemap: https://www.sap.com/sitemap-videos-index.xml
152Sitemap: https://www.sap.com/sitemap_siteimprove_index.xml
153Sitemap: https://www.sap.com/sitemap_us_int_only_siteimprove_index.xml
154Sitemap: https://www.sap.com/sitemap_us_int_only_siteimprove.xml
155Sitemap: https://www.sap.com/sitemap-us-int-only-siteimprove.xml
156Sitemap: https://www.sap.com/sitemap_countries_siteimprove.xml
What do the bots actually get?
This page reads robots.txt, which states what sap.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on sap.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does sap.com block GPTBot?
Partly. sap.com’s robots.txt keeps GPTBot out of some paths, such as /fetch, and lets it crawl the rest. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does sap.com block ClaudeBot?
Partly. sap.com’s robots.txt keeps ClaudeBot out of some paths, such as /fetch, and lets it crawl the rest. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does sap.com block Google-Extended?
Partly. sap.com’s robots.txt keeps Google-Extended out of some paths, such as /bin/sapdx/, and lets it crawl the rest. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does sap.com block PerplexityBot?
Partly. sap.com’s robots.txt keeps PerplexityBot out of some paths, such as /bin/sapdx/, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find sap.com?
Partly. sap.com’s robots.txt keeps OAI-SearchBot out of some paths, such as /bin/sapdx/, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.