Who blocks AI · #195 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does hubspot.com block AI crawlers?
No. hubspot.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.hubspot.com/robots.txt · 359 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.hubspot.com/robots.txt · 359 lines
1User-agent: *
2Disallow: /wt-assets/static-files/mktg-analytics
3Disallow: /_hcms/iplookup
4Disallow: /_hcms/perf
5Disallow: /meetings
6Disallow: /raven/static-3.4
7Disallow: /react-utils/static-2.40
8Disallow: /jquery-libs/static-1.47
9Disallow: /styled-components/static-1.3
10Disallow: /react-select-plus/static-1.18
11Disallow: /ZeroClipboard/static-1.6
12Disallow: /raven-hubspot/static-1.43
13Disallow: /PatternValidationJS/static-1.55
14Disallow: /Reagan/static-6.27
15Disallow: /jasmine/static-3.48
16Disallow: /react-select-plus/static-1.21
17Disallow: /atom/static-1.28
18Disallow: /HublyticsTrackerJS/static-3.23
19Disallow: /HublyticsTrackerJS/static-3.24
20Disallow: /HubStyleTokens/static-2.81
21Disallow: /react-redux/static-5.3
22Disallow: /react-select-plus/static-1.25
23Disallow: /icons/static-2.133
24Disallow: /hub-http/static-1.158
25Disallow: /react-redux/static-5.4
26Disallow: /MeetingsLib/static-1.1167
27Disallow: /salesImages/static-1.378
28Disallow: /jasmine-runner/static-1.14
29Disallow: /react/static-6.15
30Disallow: /jasmine/static-3.49
31Disallow: /reselect/static-2.5
32Disallow: /transmute/static-1.85
33Disallow: /MeetingsBase/static-1.1538
34Disallow: /timezone-utils/static-2.29
35Disallow: /I18n/static-7.192
36Disallow: /enviro/static-3.70
37Disallow: /Baldric/static-1.71
38Disallow: /tracking-pixels/static-1.8
39Disallow: /redux-thunk/static-2.1
40Disallow: /urlinator/static-1.55
41Disallow: /atom/static-1.31
42Disallow: /react-utils/static-2.42
43Disallow: /UIComponents/static-2.882
44Disallow: /MeetingsPublic/static-1.2503
45Disallow: /PortalIdParser/static-1.29
46Disallow: /pickadate/static-1.6
47Disallow: /StyleGuideUI/static-3.98
48Disallow: /MeetingsLib/static-2.226
49Disallow: /usage-tracker-core/static-1.65
50Disallow: /I18n/static-7.204
51Disallow: /PatternValidationJS/static-1.60
52Disallow: /timezone-utils/static-2.30
53Disallow: /salesImages/static-1.370
54Disallow: /immutable/static-1.11
55Disallow: /raven-hubspot/static-1.51
56Disallow: /I18n/static-7.237
57Disallow: /HubStyleTokens/static-2.86
58Disallow: /MeetingsLib/static-2.140
59Disallow: /jasmine-runner/static-1.15
60Disallow: /q/static-4.11
61Disallow: /Baldric/static-1.73
62Disallow: /StyleGuideUI/static-3.100
63Disallow: /MeetingsPublic/static-1.2825
64Disallow: /HubStyle/static-2.309
65Disallow: /MeetingsBase/static-1.1835
66Disallow: /classnames/static-2.3
67Disallow: /jasmine/static-3.50
68Disallow: /atom/static-1.30
69Disallow: /PatternValidationJS/static-1.56
70Disallow: /Reagan/static-6.31
71Disallow: /MeetingsBase/static-1.1723
72Disallow: /HubStyleTokens/static-2.87
73Disallow: /redux/static-3.6
74Disallow: /PortalIdParser/static-1.27
75Disallow: /q/static-4.12
76Disallow: /react-redux/static-5.5
77Disallow: /raven-hubspot/static-1.46
78Disallow: /react-dom/static-6.4
79Disallow: /salesImages/static-1.367
80Disallow: /transmute/static-2.8
81Disallow: /ui-images/static-2.106
82Disallow: /common_assets/static-2.198
83Disallow: /urlinator/static-1.54
84Disallow: /HubStyleTokens/static-2.93
85Disallow: /tracking-pixels/static-1.10
86Disallow: /usage-tracker/static-1.76
87Disallow: /HubStyleTokens/static-2.88
88Disallow: /UIComponents/static-2.1132
89Disallow: /usage-tracker-core/static-1.72
90Disallow: /timezone-utils/static-2.36
91Disallow: /Reagan/static-7.4
92Disallow: /ui-addon-avatars/static-1.1307
93Disallow: /HubStyle/static-2.366
94Disallow: /salesImages/static-1.383
95Disallow: /react-utils/static-2.45
96Disallow: /UIComponents/static-2.1109
97Disallow: /jasmine/static-3.51
98Disallow: /MeetingsBase/static-1.1796
99Disallow: /MeetingsPublic/static-1.2796
100Disallow: /raven-hubspot/static-1.48
101Disallow: /usage-tracker/static-1.64
102Disallow: /MeetingsLib/static-2.85
103Disallow: /HublyticsTrackerJS/static-3.27
104Disallow: /react-utils/static-2.41
105Disallow: /icons/static-2.128
106Disallow: /styled-components/static-1.4
107Disallow: /HubStyle/static-2.348
108Disallow: /I18n/static-7.210
109Disallow: /I18n/static-7.270
110Disallow: /transmute/static-2.6
111Disallow: /Baldric/static-1.80
112Disallow: /icons/static-2.122
113Disallow: /I18n/static-7.233
114Disallow: /ui-images/static-2.105
115Disallow: /MeetingsPublic/static-1.3073
116Disallow: /StyleGuideUI/static-3.109
117Disallow: /Reagan/static-7.12
118Disallow: /MeetingsBase/static-1.1714
119Disallow: /MeetingsPublic/static-1.2718
120Disallow: /urlinator/static-1.59
121Disallow: /MeetingsBase/static-1.2082
122Disallow: /HubStyle/static-2.394
123Disallow: /salesImages/static-1.382
124Disallow: /HubStyle/static-2.402
125Disallow: /ui-addon-avatars/static-1.1494
126Disallow: /UIComponents/static-2.1282
127Disallow: /ui-addon-avatars/static-1.1461
128Disallow: /MeetingsPublic/static-1.3018
129Disallow: /common_assets/static-2.197
130Disallow: /icons/static-2.130
131Disallow: /Reagan/static-7.9
132Disallow: /q/static-3.28
133Disallow: /ui-images/static-2.92
134Disallow: /hub-http/static-1.137
135Disallow: /I18n/static-7.198
136Disallow: /redux/static-3.7
137Disallow: /MeetingsBase/static-1.1591
138Disallow: /HubStyle/static-2.344
139Disallow: /HubStyle/static-2.315
140Disallow: /HeadJS/static-2.84
141Disallow: /ui-addon-avatars/static-1.1104
142Disallow: /UIComponents/static-2.933
143Disallow: /ui-addon-avatars/static-1.1051
144Disallow: /PatternValidationJS/static-1.74
145Disallow: /UIComponents/static-2.1046
146Disallow: /MeetingsPublic/static-1.2732
147Disallow: /ui-addon-avatars/static-1.1121
148Disallow: /react-dom/static-6.5
149Disallow: /HubStyle/static-2.596
150Disallow: /MeetingsLib/static-2.875
151Disallow: /MeetingsPublic/static-1.3500
152Disallow: /react/static-6.16
153Disallow: /react-input-autosize/static-2.7
154Disallow: /react/static-6.14
155Disallow: /HubStyle/static-2.313
156Disallow: /hublabs-core/static-4.9
157Disallow: /MeetingsBase/static-1.1682
158Disallow: /ui-addon-avatars/static-1.1107
159Disallow: /HubStyle/static-2.316
160Disallow: /atom/static-1.29
161Disallow: /I18n/static-7.228
162Disallow: /react-select-plus/static-1.20
163Disallow: /icons/static-2.123
164Disallow: /MeetingsPublic/static-1.2544
165Disallow: /icons/static-2.143
166Disallow: /MeetingsLib/static-1.1206
167Disallow: /UIComponents/static-2.918
168Disallow: /ui-images/static-2.116
169Disallow: /atom/static-1.33
170Disallow: /urlinator/static-1.57
171Disallow: /usage-tracker-core/static-1.79
172Disallow: /UIComponents/static-2.1315
173Disallow: /raven/static-3.6
174Disallow: /HubStyleTokens/static-2.96
175Disallow: /salesImages/static-1.374
176Disallow: /HubStyle/static-2.359
177Disallow: /raven-hubspot/static-1.47
178Disallow: /HubStyleTokens/static-2.83
179Disallow: /salesImages/static-1.368
180Disallow: /usage-tracker/static-1.67
181Disallow: /usage-tracker-core/static-1.68
182Disallow: /Reagan/static-6.28
183Disallow: /usage-tracker-core/static-1.76
184Disallow: /jquery-libs/static-1.48
185Disallow: /UIComponents/static-2.1011
186Disallow: /MeetingsBase/static-1.1609
187Disallow: /UIComponents/static-2.949
188Disallow: /MeetingsLib/static-1.1242
189Disallow: /MeetingsPublic/static-1.2581
190Disallow: /timezone-utils/static-2.67
191Disallow: /HubStyle/static-2.342
192Disallow: /HeadJS/static-2.92
193Disallow: /hublabs-experiments/static-1.248
194Disallow: /q/static-5.12
195Disallow: /MeetingsLib/static-2.180
196Disallow: /q/static-5.23
197Disallow: /react-utils/static-2.49
198Disallow: /Reagan/static-7.27
199Disallow: /I18n/static-7.298
200Disallow: /ui-addon-avatars/static-1.1222
201Disallow: /react-select-plus/static-1.26
202Disallow: /ui-images/static-2.101
203Disallow: /icons/static-2.124
204Disallow: /I18n/static-7.293
205Disallow: /usage-tracker/static-1.110
206Disallow: /PortalIdParser/static-1.28
207Disallow: /hublabs-experiments/static-1.262
208Disallow: /usage-tracker-core/static-1.89
209Disallow: /HubStyleTokens/static-2.310
210Disallow: /HubStyle/static-2.622
211Disallow: /MeetingsPublic/static-1.3655
212Disallow: /MeetingsBase/static-1.2042
213Disallow: /usage-tracker/static-1.108
214Disallow: /hub-http/static-1.157
215Disallow: /usage-tracker/static-1.68
216Disallow: /usage-tracker-core/static-1.69
217Disallow: /MeetingsLib/static-1.1227
218Disallow: /MeetingsLib/static-2.413
219Disallow: /usage-tracker/static-1.71
220Disallow: /MeetingsLib/static-2.464
221Disallow: /MeetingsLib/static-2.201
222Disallow: /icons/static-2.131
223Disallow: /MeetingsLib/static-1.1224
224Disallow: /hub-http/static-1.161
225Disallow: /enviro/static-3.72
226Disallow: /MeetingsPublic/static-1.2562
227Disallow: /q/static-5.22
228Disallow: /transmute/static-2.9
229Disallow: /hub-http/static-1.156
230Disallow: /HeadJS/static-2.83
231Disallow: /Reagan/static-7.16
232Disallow: /HubStyle/static-2.323
233Disallow: /Reagan/static-6.29
234Disallow: /styled-components/static-1.6
235Disallow: /ui-addon-avatars/static-1.1251
236Disallow: /MeetingsPublic/static-1.2773
237Disallow: /MeetingsPublic/static-1.2674
238Disallow: /ui-images/static-2.100
239Disallow: /ui-addon-avatars/static-1.1183
240Disallow: /icons/static-2.144
241Disallow: /raven-hubspot/static-1.60
242Disallow: /UIComponents/static-2.1578
243Disallow: /MeetingsLib/static-2.556
244Disallow: /ui-addon-avatars/static-1.1542
245Disallow: /HubStyle/static-2.365
246Disallow: /ui-addon-avatars/static-1.1089
247Disallow: /MeetingsBase/static-1.2604
248Disallow: /timezone-utils/static-2.66
249Disallow: /I18n/static-7.312
250Disallow: /I18n/static-7.278
251Disallow: /HubStyleTokens/static-2.167
252Disallow: /HubStyleTokens/static-2.150
253Disallow: /MeetingsPublic/static-1.3264
254Disallow: /MeetingsBase/static-1.2253
255Disallow: /I18n/static-7.269
256Disallow: /PatternValidationJS/static-1.103
257Disallow: /hub-http/static-1.178
258Disallow: /Reagan/static-7.39
259Disallow: /HeadJS/static-2.115
260Disallow: /I18n/static-7.304
261Disallow: /StyleGuideUI/static-3.136
262Disallow: /EarlyRequesterJS/static-1.38
263Disallow: /ui-addon-avatars/static-1.1867
264Disallow: /HubStyle/static-2.544
265Disallow: /q/static-5.28
266Disallow: /PatternValidationJS/static-1.102
267Disallow: /usage-tracker-core/static-1.103
268Disallow: /MeetingsPublic/static-1.4178
269Disallow: /MeetingsBase/static-1.2041
270Disallow: /usage-tracker-core/static-1.101
271Disallow: /hublabs-core/static-4.44
272Disallow: /timezone-utils/static-2.111
273Disallow: /Baldric/static-1.94
274Disallow: /MeetingsPublic/static-1.3012
275Disallow: /usage-tracker-container/static-1.158
276Disallow: /usage-tracker-container/static-1.162
277Disallow: /FormsCommonComponents/static-1.879
278Disallow: /usage-tracker-container/static-1.163
279Disallow: /FormsCommonComponents/static-1.864
280Disallow: /FormsCommonComponents/static-1.886
281Disallow: /PatternValidationJS/static-1.100
282Disallow: /usage-tracker-public/static-1.49
283Disallow: /ui-addon-avatars/static-1.1889
284Disallow: /HubStyleTokens/static-2.453
285Disallow: /MeetingsBase/static-1.3127
286Disallow: /MeetingsPublic/static-1.4227
287Disallow: /MeetingsBase/static-1.3176
288Disallow: /HubStyleTokens/static-2.431
289Disallow: /timezone-utils/static-2.115
290Disallow: /UIComponents/static-2.1699
291Disallow: /MeetingsLib/static-2.1553
292Disallow: /MeetingsLib/static-2.1602
293Disallow: /MeetingsBase/static-1.3196
294Disallow: /react-utils/static-2.55
295Disallow: /Reagan/static-7.35
296Disallow: /HubStyle/static-2.757
297Disallow: /I18n/static-7.315
298Disallow: /raven-hubspot/static-1.59
299Disallow: /MeetingsBase/static-1.1946
300Disallow: /usage-tracker-core/static-1.75
301Disallow: /MeetingsPublic/static-1.4247
302Disallow: /MeetingsLib/static-2.1158
303Disallow: /HubStyle/static-2.646
304Disallow: /jasmine-runner/static-1.17
305Disallow: /timezone-utils/static-2.114
306Disallow: /ui-addon-avatars/static-1.1882
307Disallow: /q/static-5.29
308Disallow: /atom/static-1.40
309Disallow: /MeetingsLib/static-2.1622
310Disallow: /HubStyle/static-2.765
311Disallow: /UIComponents/static-2.1713
312Disallow: /salesImages/static-1.366
313Disallow: /MeetingsBase/static-1.2032
314Disallow: /I18n/static-7.314
315Disallow: /UIComponents/static-2.1795
316Disallow: /MeetingsBase/static-1.3474
317Disallow: /MeetingsLib/static-2.1900
318Disallow: /MeetingsPublic/static-1.4525
319Disallow: /HubStyleTokens/static-2.527
320Disallow: /HubStyle/static-2.839
321Disallow: /FormsCommonComponents/static-1.956
322Disallow: /I18n/static-7.329
323Disallow: /react-utils/static-2.58
324Disallow: /jasmine-runner/static-1.18
325Disallow: /raven-hubspot/static-1.73
326Disallow: /ui-images/static-2.193
327Disallow: /urlinator/static-1.62
328Disallow: /tally-loader-react/static-1.65
329Disallow: /usage-tracker/static-1.107
330Disallow: /salesImages/static-1.389
331Disallow: /hs/hsstatic/content_shared
332Disallow: /hs/hsstatic/jquery-libs/st
333Allow: /hs/hsstatic/jquery-libs/static-1.1/jquery/
334Disallow: /_hcms/iplookup
335Disallow: /_hcms/perf
336Disallow: *?portalId=
337Disallow: *?inpageEditorUI=
338Disallow: *?cssPath=
339Disallow: *?_health_check_=
340Disallow: *?_preview=
341Disallow: *?benderPackage=
342Disallow: *?preview_key=
343Disallow: *?staticVersion=
344Disallow: *?cacheBurst=
345Disallow: *?tc_deviceCategory=
346Disallow: *?isPreview=
347Disallow: *?preview_theme=
348Disallow: *?cos=
349Disallow: /this-is-a-test
350Disallow: /_hcms/livechat/widget
351Disallow: /business-templates/*?__hstc
352Disallow: /resources/thank-you/*
353Disallow: /resources/template/thank-you/*
354Disallow: /_hcms/preview/
355Disallow: /hs/manage-preferences/
356Disallow: /hs/preferences-center/
357Disallow: /*?*hs_preview=*
358Disallow: /*?*hsCacheBuster=*
359
What do the bots actually get?
This page reads robots.txt, which states what hubspot.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on hubspot.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does hubspot.com block GPTBot?
No. hubspot.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does hubspot.com block ClaudeBot?
No. hubspot.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does hubspot.com block Google-Extended?
No. hubspot.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does hubspot.com block PerplexityBot?
No. hubspot.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find hubspot.com?
Yes. hubspot.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.