Who blocks AI · #990 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does salon.com block AI crawlers?
No. salon.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.salon.com/robots.txt · 205 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | line 184 · Allow: / under User-agent: Applebot |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | line 154 · Allow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Allowed | line 172 · Allow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Allowed | line 163 · Allow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | line 190 · Allow: / under User-agent: Amazonbot |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | line 145 · Allow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Allowed | line 157 · Allow: / under User-agent: ClaudeBot |
Google-Extended | Allowed | line 181 · Allow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Allowed | line 178 · Allow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | line 187 · Allow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | line 193 · Allow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | line 148 · Allow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Allowed | line 160 · Allow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Allowed | line 175 · Allow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | line 196 · Allow: / under User-agent: Meta-ExternalFetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 6 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Blocked | line 9 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Blocked | line 15 · Disallow: / under User-agent: DotBot |
View www.salon.com/robots.txt · 205 lines
1User-agent: *
2Disallow: /search/
3
4# Block known SEO crawlers and data co
5User-agent: AhrefsBot
6Disallow: /
7
8User-agent: SemrushBot
9Disallow: /
10
11User-agent: MJ12bot
12Disallow: /
13
14User-agent: DotBot
15Disallow: /
16
17User-agent: Zoominfobot
18Disallow: /
19
20User-agent: BlexBot
21Disallow: /
22
23User-agent: Cliqzbot
24Disallow: /
25
26User-agent: DataForSeoBot
27Disallow: /
28
29User-agent: MauiBot
30Disallow: /
31
32User-agent: MegaIndex.ru
33Disallow: /
34
35# Image crawlers
36User-agent: Pixray-Seeker
37Disallow: /
38
39User-agent: Copyleaks
40Disallow: /
41
42User-agent: Plaghunter
43Disallow: /
44
45User-agent: ImageMax
46Disallow: /
47
48User-agent: Tineye
49Disallow: /
50
51User-agent: Copytrack
52Disallow: /
53
54User-agent: CopytrackBot
55Disallow: /
56
57# Allow major search engines
58User-agent: Googlebot
59Disallow:
60
61User-agent: Bingbot
62Disallow:
63
64User-agent: DuckDuckBot
65Disallow:
66
67User-agent: Baiduspider
68Disallow:
69
70User-agent: YandexBot
71Disallow:
72
73User-agent: Sogou
74Disallow:
75
76User-agent: PetalBot
77Disallow:
78
79User-agent: AspiegelBot
80Disallow:
81
82User-agent: SeznamBot
83Disallow:
84
85User-agent: Bytespider
86Disallow:
87
88User-agent: VoilaBot
89Disallow:
90
91# Allow social media platforms
92User-agent: Twitterbot
93Disallow:
94
95User-agent: FacebookExternalHit
96Disallow:
97
98User-agent: LinkedInBot
99Disallow:
100
101User-agent: Pinterestbot
102Disallow:
103
104User-agent: TelegramBot
105Disallow:
106
107User-agent: Discordbot
108Disallow:
109
110User-agent: WhatsApp
111Disallow:
112
113User-agent: Slackbot
114Disallow:
115
116User-agent: Redditbot
117Disallow:
118
119# Explicitly allow the following crawlers
120User-agent: Verity/1.1 (https://gumgum.com/verity; verity-support@gumgum.com)
121Allow: /
122
123User-agent: Scope3/2.0 (scope3.com)
124Allow: /
125
126User-agent: AdsBot-Google
127Allow: /
128
129User-agent: AdsBot-Google-Mobile
130Allow: /
131
132User-agent: Googlebot-Image
133Allow: /
134
135User-agent: Googlebot-Video
136Allow: /
137
138User-agent: Googlebot-News
139Allow: /
140
141User-agent: Googlebot-InspectionTool
142Allow: /
143
144User-agent: GPTBot
145Allow: /
146
147User-agent: ChatGPT-User
148Allow: /
149
150User-agent: ChatGPT-Operator
151Allow: /
152
153User-agent: OAI-SearchBot
154Allow: /
155
156User-agent: ClaudeBot
157Allow: /
158
159User-agent: Claude-User
160Allow: /
161
162User-agent: Claude-SearchBot
163Allow: /
164
165User-agent: Claude-Web
166Allow: /
167
168User-agent: anthropic-ai
169Allow: /
170
171User-agent: PerplexityBot
172Allow: /
173
174User-agent: Perplexity-User
175Allow: /
176
177User-agent: CCBot
178Allow: /
179
180User-agent: Google-Extended
181Allow: /
182
183User-agent: Applebot
184Allow: /
185
186User-agent: Applebot-Extended
187Allow: /
188
189User-agent: Amazonbot
190Allow: /
191
192User-agent: Meta-ExternalAgent
193Allow: /
194
195User-agent: Meta-ExternalFetcher
196Allow: /
197
198User-agent: Facebookbot
199Allow: /
200
201User-agent: AmazonAdBot
202Allow: /
203
204Sitemap: https://www.salon.com/sitemap_index.xml
205
What do the bots actually get?
This page reads robots.txt, which states what salon.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on salon.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does salon.com block GPTBot?
No. salon.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does salon.com block ClaudeBot?
No. salon.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does salon.com block Google-Extended?
No. salon.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does salon.com block PerplexityBot?
No. salon.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find salon.com?
Yes. salon.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.