Who blocks AI · #333 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does intel.com block AI crawlers?
No. intel.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.intel.com/robots.txt · 186 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | line 59 · Allow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | line 44 · Allow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Allowed | line 56 · Allow: / under User-agent: ClaudeBot |
Google-Extended | Allowed | line 50 · Allow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | line 47 · Allow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 183 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Blocked | line 183 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.intel.com/robots.txt · 186 lines
1# robots.txt exclusion for www.intel.com/ - US
2
3User-agent: OmtrBot/1.0
4Allow: /content/www/us/en/products/devices-systems/
5Allow: /content/dam/products/
6Allow: /libs/apps/intel/services/productseedlist
7Disallow: /cgi
8Disallow: /iaweb/
9Disallow: /cpc/vision/
10Disallow: /intel/june297/
11Disallow: /cpc/eps/
12Disallow: /design/june297/
13Disallow: /cpc/archive/
14Disallow: /cpc/dia/
15Disallow: /cpc/ecs/
16Disallow: /cpc/fcs/
17Disallow: /cpc/gif/
18Disallow: /cpc/OptContent/
19Disallow: /cpc/pix/
20Disallow: /cpc/sound/
21Disallow: /cpc/feature/
22Disallow: /emergency/
23Disallow: /content/www/*/*/buy/intelinside/
24Disallow: /content/www/*/*/buy/inteltechnologyprovider/
25Disallow: /content/www/*/*/buy/intelproducts/
26Disallow: /content/www/*/*/homepage/_jcr_content/
27Disallow: /content/www/*/*/test/
28Disallow: /content/data/disclaimers/
29Disallow: /content/www/*/*/contentlibrary.html?
30Disallow: /content/iip/
31Disallow: /content/dam/altera-www/global/*/pdfs/assets/
32Disallow: /content/dam/www/programmable/*/*/pdfs/assets/
33Disallow: /content/dam/www/public/*/forms/
34Disallow: /*.swf$
35Disallow: /*/swf/
36Disallow: /*/secure/
37Disallow: /*/wp-expand-server-memory-with-optane*.pdf$
38Disallow: /*/wp-01288-intel-fpgas-accelerate-intel-xeon-scalable*.pdf$
39Disallow: /content/dam/www/secure/
40
41# GEO: Allow all major AI crawlers — supports brand citation in AI-generated answers
42
43User-agent: GPTBot
44Allow: /
45
46User-agent: ChatGPT-User
47Allow: /
48
49User-agent: Google-Extended
50Allow: /
51
52User-agent: anthropic-ai
53Allow: /
54
55User-agent: ClaudeBot
56Allow: /
57
58User-agent: PerplexityBot
59Allow: /
60
61User-agent: *
62Allow: /llms.txt
63Allow: /llms-full.txt
64Allow: /content/dam/products/catalog/
65Allow: /content/dam/products/hero/
66Disallow: /cgi
67Disallow: /iaweb/
68Disallow: /cpc/vision/
69Disallow: /intel/june297/
70Disallow: /cpc/eps/
71Disallow: /design/june297/
72Disallow: /cpc/archive/
73Disallow: /cpc/dia/
74Disallow: /cpc/ecs/
75Disallow: /cpc/fcs/
76Disallow: /cpc/gif/
77Disallow: /cpc/OptContent/
78Disallow: /cpc/pix/
79Disallow: /cpc/sound/
80Disallow: /cpc/feature/
81Disallow: /emergency/
82Disallow: /content/www/*/*/buy/intelinside/
83Disallow: /content/www/*/*/buy/inteltechnologyprovider/
84Disallow: /content/www/*/*/buy/intelproducts/
85Disallow: /content/www/*/*/homepage/_jcr_content/
86Disallow: /content/www/*/*/test/
87Disallow: /content/data/disclaimers/
88Disallow: /content/www/*/*/contentlibrary.html?
89Disallow: /content/iip/
90Disallow: /content/dam/altera-www/global/*/pdfs/assets/
91Disallow: /content/dam/www/programmable/*/*/pdfs/assets/
92Disallow: /content/dam/www/public/*/forms/
93Disallow: /content/dam/products/
94Disallow: /libs/apps/intel/services/productseedlist
95Disallow: /now/
96Disallow: /*countrylabel=*&countrylabel=*
97Disallow: /*&_escaped_fragment_=*
98Disallow: /*.swf$
99Disallow: /*/swf/
100Disallow: /*/secure/
101Disallow: /*/wp-expand-server-memory-with-optane*.pdf$
102Disallow: /*/wp-01288-intel-fpgas-accelerate-intel-xeon-scalable*.pdf$
103Disallow: /content/dam/www/secure/
104Disallow: /content/www/br/pt/
105Disallow: /content/www/xl/es/
106Disallow: /content/www/de/de/
107Disallow: /content/www/es/es/
108Disallow: /content/www/fr/fr/
109Disallow: /content/www/jp/ja/
110Disallow: /content/www/id/id/
111Disallow: /content/www/kr/ko/
112Disallow: /content/www/th/th/
113Disallow: /content/www/tw/zh/
114Disallow: /content/www/vn/vi/
115
116User-agent: SEOkicks
117Crawl-Delay: 10
118
119User-agent: coccocbot
120Crawl-Delay: 10
121
122User-agent: AhrefsBot
123Crawl-Delay: 10
124
125User-agent: SemrushBot
126Crawl-Delay: 10
127
128User-agent: trendictionbot
129Crawl-Delay: 10
130
131User-agent: serpstatbot
132Crawl-Delay: 10
133
134User-agent: startmebot
135Crawl-Delay: 10
136
137User-agent: HatenaBlog-bot
138Crawl-Delay: 10
139
140User-agent: Cincraw
141Crawl-Delay: 10
142
143User-agent: InfoTigerBot
144Crawl-Delay: 10
145
146User-agent: snapchat
147Crawl-Delay: 10
148
149User-agent: Superfeedr
150Crawl-Delay: 10
151
152User-agent: YaK
153Crawl-Delay: 10
154
155User-agent: digitalshadowsbot
156Crawl-Delay: 10
157
158User-agent: Magus
159Crawl-Delay: 10
160
161User-agent: Quora-Bot
162Crawl-Delay: 10
163
164User-agent: INETDEX-BOT
165Crawl-Delay: 10
166
167User-agent: DataForSeoBot
168Crawl-Delay: 10
169
170User-agent: SerendeputyBot
171Crawl-Delay: 10
172
173User-agent: Deskyobot
174Crawl-Delay: 10
175
176User-agent: MJ12bot
177Crawl-Delay: 10
178
179User-agent: BrightEdge
180Crawl-Delay: 10
181
182User-agent: ICC-Crawler
183Disallow: /
184
185Sitemap: https://www.intel.com/sitemap.xml
186
What do the bots actually get?
This page reads robots.txt, which states what intel.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on intel.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does intel.com block GPTBot?
No. intel.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does intel.com block ClaudeBot?
No. intel.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does intel.com block Google-Extended?
No. intel.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does intel.com block PerplexityBot?
No. intel.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find intel.com?
Yes. intel.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.