Who blocks AI · #932 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does hootsuite.com block AI crawlers?
No. hootsuite.com’s robots.txt lets all 19 AI crawlers we track crawl the site, though it keeps OAI-SearchBot, PerplexityBot and 9 more out of some paths. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotPartly blockedOAI-SearchBotPartly blockedClaudeBotPartly blockedPerplexityBotPartly blockedGoogle-ExtendedPartly blockedCCBotAllowed- robots.txt
- www.hootsuite.com/robots.txt · 338 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 27 · Disallow: *d=pro* line 28 · Disallow: /*?*srsltid and 18 more | |
Bingbot | Microsoft | Partly blocked | line 51 · Disallow: *d=pro* line 52 · Disallow: /*?*srsltid and 18 more |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Partly blocked | line 27 · Disallow: *d=pro* line 28 · Disallow: /*?*srsltid and 18 more |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Partly blocked | line 123 · Disallow: *d=pro* line 124 · Disallow: /*?*srsltid and 18 more |
PerplexityBot | Perplexity | Partly blocked | line 195 · Disallow: *d=pro* line 196 · Disallow: /*?*srsltid and 18 more |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Partly blocked | line 291 · Disallow: *d=pro* line 292 · Disallow: /*?*srsltid and 18 more |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Partly blocked | line 75 · Disallow: *d=pro* line 76 · Disallow: /*?*srsltid and 18 more |
ClaudeBot | Anthropic | Partly blocked | line 147 · Disallow: *d=pro* line 148 · Disallow: /*?*srsltid and 18 more |
Google-Extended | Partly blocked | line 243 · Disallow: *d=pro* line 244 · Disallow: /*?*srsltid and 18 more | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Partly blocked | line 267 · Disallow: *d=pro* line 268 · Disallow: /*?*srsltid and 18 more |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Partly blocked | line 315 · Disallow: *d=pro* line 316 · Disallow: /*?*srsltid and 18 more |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Partly blocked | line 99 · Disallow: *d=pro* line 100 · Disallow: /*?*srsltid and 18 more |
Claude-User | Anthropic | Partly blocked | line 171 · Disallow: *d=pro* line 172 · Disallow: /*?*srsltid and 18 more |
Perplexity-User | Perplexity | Partly blocked | line 219 · Disallow: *d=pro* line 220 · Disallow: /*?*srsltid and 18 more |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.hootsuite.com/robots.txt · 338 lines
1User-agent: *
2Content-Signal: search=yes, ai-input=yes, ai-train=yes
3Disallow: *d=pro*
4Disallow: /*?*srsltid
5Disallow: /dashboard*
6Disallow: /signup*
7Disallow: /feed*
8Disallow: /login*
9Disallow: /hootlet/social-share*
10Disallow: /hootlet/load*
11Disallow: /social-connect/send*
12Disallow: /share-button*
13Disallow: /app/social-network/*
14Disallow: /ajax*
15Disallow: /sso-redirect*
16Disallow: /js*
17Disallow: /craft/
18Disallow: /*.pdf$
19Disallow: /billing/
20Disallow: /create-free-account*
21Disallow: /academy-free-account*
22Disallow: /content-generator-embed*
23
24User-agent: Googlebot
25Content-Signal: search=yes, ai-input=yes, ai-train=yes
26Allow: /
27Disallow: *d=pro*
28Disallow: /*?*srsltid
29Disallow: /dashboard*
30Disallow: /signup*
31Disallow: /feed*
32Disallow: /login*
33Disallow: /hootlet/social-share*
34Disallow: /hootlet/load*
35Disallow: /social-connect/send*
36Disallow: /share-button*
37Disallow: /app/social-network/*
38Disallow: /ajax*
39Disallow: /sso-redirect*
40Disallow: /js*
41Disallow: /craft/
42Disallow: /*.pdf$
43Disallow: /billing/
44Disallow: /create-free-account*
45Disallow: /academy-free-account*
46Disallow: /content-generator-embed*
47
48User-agent: Bingbot
49Content-Signal: search=yes, ai-input=yes, ai-train=yes
50Allow: /
51Disallow: *d=pro*
52Disallow: /*?*srsltid
53Disallow: /dashboard*
54Disallow: /signup*
55Disallow: /feed*
56Disallow: /login*
57Disallow: /hootlet/social-share*
58Disallow: /hootlet/load*
59Disallow: /social-connect/send*
60Disallow: /share-button*
61Disallow: /app/social-network/*
62Disallow: /ajax*
63Disallow: /sso-redirect*
64Disallow: /js*
65Disallow: /craft/
66Disallow: /*.pdf$
67Disallow: /billing/
68Disallow: /create-free-account*
69Disallow: /academy-free-account*
70Disallow: /content-generator-embed*
71
72User-agent: GPTBot
73Content-Signal: search=yes, ai-input=yes, ai-train=yes
74Allow: /
75Disallow: *d=pro*
76Disallow: /*?*srsltid
77Disallow: /dashboard*
78Disallow: /signup*
79Disallow: /feed*
80Disallow: /login*
81Disallow: /hootlet/social-share*
82Disallow: /hootlet/load*
83Disallow: /social-connect/send*
84Disallow: /share-button*
85Disallow: /app/social-network/*
86Disallow: /ajax*
87Disallow: /sso-redirect*
88Disallow: /js*
89Disallow: /craft/
90Disallow: /*.pdf$
91Disallow: /billing/
92Disallow: /create-free-account*
93Disallow: /academy-free-account*
94Disallow: /content-generator-embed*
95
96User-agent: ChatGPT-User
97Content-Signal: search=yes, ai-input=yes, ai-train=yes
98Allow: /
99Disallow: *d=pro*
100Disallow: /*?*srsltid
101Disallow: /dashboard*
102Disallow: /signup*
103Disallow: /feed*
104Disallow: /login*
105Disallow: /hootlet/social-share*
106Disallow: /hootlet/load*
107Disallow: /social-connect/send*
108Disallow: /share-button*
109Disallow: /app/social-network/*
110Disallow: /ajax*
111Disallow: /sso-redirect*
112Disallow: /js*
113Disallow: /craft/
114Disallow: /*.pdf$
115Disallow: /billing/
116Disallow: /create-free-account*
117Disallow: /academy-free-account*
118Disallow: /content-generator-embed*
119
120User-agent: OAI-SearchBot
121Content-Signal: search=yes, ai-input=yes, ai-train=yes
122Allow: /
123Disallow: *d=pro*
124Disallow: /*?*srsltid
125Disallow: /dashboard*
126Disallow: /signup*
127Disallow: /feed*
128Disallow: /login*
129Disallow: /hootlet/social-share*
130Disallow: /hootlet/load*
131Disallow: /social-connect/send*
132Disallow: /share-button*
133Disallow: /app/social-network/*
134Disallow: /ajax*
135Disallow: /sso-redirect*
136Disallow: /js*
137Disallow: /craft/
138Disallow: /*.pdf$
139Disallow: /billing/
140Disallow: /create-free-account*
141Disallow: /academy-free-account*
142Disallow: /content-generator-embed*
143
144User-agent: ClaudeBot
145Content-Signal: search=yes, ai-input=yes, ai-train=yes
146Allow: /
147Disallow: *d=pro*
148Disallow: /*?*srsltid
149Disallow: /dashboard*
150Disallow: /signup*
151Disallow: /feed*
152Disallow: /login*
153Disallow: /hootlet/social-share*
154Disallow: /hootlet/load*
155Disallow: /social-connect/send*
156Disallow: /share-button*
157Disallow: /app/social-network/*
158Disallow: /ajax*
159Disallow: /sso-redirect*
160Disallow: /js*
161Disallow: /craft/
162Disallow: /*.pdf$
163Disallow: /billing/
164Disallow: /create-free-account*
165Disallow: /academy-free-account*
166Disallow: /content-generator-embed*
167
168User-agent: Claude-User
169Content-Signal: search=yes, ai-input=yes, ai-train=yes
170Allow: /
171Disallow: *d=pro*
172Disallow: /*?*srsltid
173Disallow: /dashboard*
174Disallow: /signup*
175Disallow: /feed*
176Disallow: /login*
177Disallow: /hootlet/social-share*
178Disallow: /hootlet/load*
179Disallow: /social-connect/send*
180Disallow: /share-button*
181Disallow: /app/social-network/*
182Disallow: /ajax*
183Disallow: /sso-redirect*
184Disallow: /js*
185Disallow: /craft/
186Disallow: /*.pdf$
187Disallow: /billing/
188Disallow: /create-free-account*
189Disallow: /academy-free-account*
190Disallow: /content-generator-embed*
191
192User-agent: PerplexityBot
193Content-Signal: search=yes, ai-input=yes, ai-train=yes
194Allow: /
195Disallow: *d=pro*
196Disallow: /*?*srsltid
197Disallow: /dashboard*
198Disallow: /signup*
199Disallow: /feed*
200Disallow: /login*
201Disallow: /hootlet/social-share*
202Disallow: /hootlet/load*
203Disallow: /social-connect/send*
204Disallow: /share-button*
205Disallow: /app/social-network/*
206Disallow: /ajax*
207Disallow: /sso-redirect*
208Disallow: /js*
209Disallow: /craft/
210Disallow: /*.pdf$
211Disallow: /billing/
212Disallow: /create-free-account*
213Disallow: /academy-free-account*
214Disallow: /content-generator-embed*
215
216User-agent: Perplexity-User
217Content-Signal: search=yes, ai-input=yes, ai-train=yes
218Allow: /
219Disallow: *d=pro*
220Disallow: /*?*srsltid
221Disallow: /dashboard*
222Disallow: /signup*
223Disallow: /feed*
224Disallow: /login*
225Disallow: /hootlet/social-share*
226Disallow: /hootlet/load*
227Disallow: /social-connect/send*
228Disallow: /share-button*
229Disallow: /app/social-network/*
230Disallow: /ajax*
231Disallow: /sso-redirect*
232Disallow: /js*
233Disallow: /craft/
234Disallow: /*.pdf$
235Disallow: /billing/
236Disallow: /create-free-account*
237Disallow: /academy-free-account*
238Disallow: /content-generator-embed*
239
240User-agent: Google-Extended
241Content-Signal: search=yes, ai-input=yes, ai-train=yes
242Allow: /
243Disallow: *d=pro*
244Disallow: /*?*srsltid
245Disallow: /dashboard*
246Disallow: /signup*
247Disallow: /feed*
248Disallow: /login*
249Disallow: /hootlet/social-share*
250Disallow: /hootlet/load*
251Disallow: /social-connect/send*
252Disallow: /share-button*
253Disallow: /app/social-network/*
254Disallow: /ajax*
255Disallow: /sso-redirect*
256Disallow: /js*
257Disallow: /craft/
258Disallow: /*.pdf$
259Disallow: /billing/
260Disallow: /create-free-account*
261Disallow: /academy-free-account*
262Disallow: /content-generator-embed*
263
264User-agent: Applebot-Extended
265Content-Signal: search=yes, ai-input=yes, ai-train=yes
266Allow: /
267Disallow: *d=pro*
268Disallow: /*?*srsltid
269Disallow: /dashboard*
270Disallow: /signup*
271Disallow: /feed*
272Disallow: /login*
273Disallow: /hootlet/social-share*
274Disallow: /hootlet/load*
275Disallow: /social-connect/send*
276Disallow: /share-button*
277Disallow: /app/social-network/*
278Disallow: /ajax*
279Disallow: /sso-redirect*
280Disallow: /js*
281Disallow: /craft/
282Disallow: /*.pdf$
283Disallow: /billing/
284Disallow: /create-free-account*
285Disallow: /academy-free-account*
286Disallow: /content-generator-embed*
287
288User-agent: Amazonbot
289Content-Signal: search=yes, ai-input=yes, ai-train=yes
290Allow: /
291Disallow: *d=pro*
292Disallow: /*?*srsltid
293Disallow: /dashboard*
294Disallow: /signup*
295Disallow: /feed*
296Disallow: /login*
297Disallow: /hootlet/social-share*
298Disallow: /hootlet/load*
299Disallow: /social-connect/send*
300Disallow: /share-button*
301Disallow: /app/social-network/*
302Disallow: /ajax*
303Disallow: /sso-redirect*
304Disallow: /js*
305Disallow: /craft/
306Disallow: /*.pdf$
307Disallow: /billing/
308Disallow: /create-free-account*
309Disallow: /academy-free-account*
310Disallow: /content-generator-embed*
311
312User-agent: Meta-ExternalAgent
313Content-Signal: search=yes, ai-input=yes, ai-train=yes
314Allow: /
315Disallow: *d=pro*
316Disallow: /*?*srsltid
317Disallow: /dashboard*
318Disallow: /signup*
319Disallow: /feed*
320Disallow: /login*
321Disallow: /hootlet/social-share*
322Disallow: /hootlet/load*
323Disallow: /social-connect/send*
324Disallow: /share-button*
325Disallow: /app/social-network/*
326Disallow: /ajax*
327Disallow: /sso-redirect*
328Disallow: /js*
329Disallow: /craft/
330Disallow: /*.pdf$
331Disallow: /billing/
332Disallow: /create-free-account*
333Disallow: /academy-free-account*
334Disallow: /content-generator-embed*
335Sitemap: https://www.hootsuite.com/sitemap.xml
336Host: https://www.hootsuite.com
337Agentmap: https://www.hootsuite.com/.well-known/ard.json
338
What do the bots actually get?
This page reads robots.txt, which states what hootsuite.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on hootsuite.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does hootsuite.com block GPTBot?
Partly. hootsuite.com’s robots.txt keeps GPTBot out of some paths, such as *d=pro*, and lets it crawl the rest. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does hootsuite.com block ClaudeBot?
Partly. hootsuite.com’s robots.txt keeps ClaudeBot out of some paths, such as *d=pro*, and lets it crawl the rest. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does hootsuite.com block Google-Extended?
Partly. hootsuite.com’s robots.txt keeps Google-Extended out of some paths, such as *d=pro*, and lets it crawl the rest. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does hootsuite.com block PerplexityBot?
Partly. hootsuite.com’s robots.txt keeps PerplexityBot out of some paths, such as *d=pro*, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find hootsuite.com?
Partly. hootsuite.com’s robots.txt keeps OAI-SearchBot out of some paths, such as *d=pro*, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.