Who blocks AI · #1035 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does houzz.com block AI crawlers?
No. houzz.com’s robots.txt lets all 19 AI crawlers we track crawl the site, though it keeps OAI-SearchBot, PerplexityBot and 5 more out of some paths. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotPartly blockedOAI-SearchBotPartly blockedClaudeBotPartly blockedPerplexityBotPartly blockedGoogle-ExtendedPartly blockedCCBotAllowed- robots.txt
- www.houzz.com/robots.txt · 427 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | line 201 · Allow: / under User-agent: * | |
Bingbot | Microsoft | Allowed | line 201 · Allow: / under User-agent: * |
DuckDuckBot | DuckDuckGo | Allowed | line 201 · Allow: / under User-agent: * |
Applebot | Apple | Allowed | line 201 · Allow: / under User-agent: * |
PetalBot | Huawei | Allowed | line 201 · Allow: / under User-agent: * |
YandexBot | Yandex | Allowed | line 201 · Allow: / under User-agent: * |
Baiduspider | Baidu | Allowed | line 201 · Allow: / under User-agent: * |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Partly blocked | line 212 · Disallow: /ad/* line 213 · Disallow: /discussions/*sort=* and 142 more |
PerplexityBot | Perplexity | Partly blocked | line 212 · Disallow: /ad/* line 213 · Disallow: /discussions/*sort=* and 142 more |
Claude-SearchBot | Anthropic | Allowed | line 201 · Allow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Allowed | line 201 · Allow: / under User-agent: * |
Amazonbot | Amazon | Allowed | line 201 · Allow: / under User-agent: * |
YouBot | You.com | Allowed | line 201 · Allow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Partly blocked | line 212 · Disallow: /ad/* line 213 · Disallow: /discussions/*sort=* and 142 more |
ClaudeBot | Anthropic | Partly blocked | line 212 · Disallow: /ad/* line 213 · Disallow: /discussions/*sort=* and 142 more |
Google-Extended | Partly blocked | line 212 · Disallow: /ad/* line 213 · Disallow: /discussions/*sort=* and 142 more | |
CCBot | Common Crawl | Allowed | line 201 · Allow: / under User-agent: * |
Applebot-Extended | Apple | Allowed | line 201 · Allow: / under User-agent: * |
GoogleOther | Allowed | line 201 · Allow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Allowed | line 201 · Allow: / under User-agent: * |
Bytespider | ByteDance | Allowed | line 201 · Allow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Partly blocked | line 212 · Disallow: /ad/* line 213 · Disallow: /discussions/*sort=* and 142 more |
Claude-User | Anthropic | Partly blocked | line 212 · Disallow: /ad/* line 213 · Disallow: /discussions/*sort=* and 142 more |
Perplexity-User | Perplexity | Allowed | line 201 · Allow: / under User-agent: * |
MistralAI-User | Mistral | Allowed | line 201 · Allow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Allowed | line 201 · Allow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | line 201 · Allow: / under User-agent: * |
SemrushBot | Semrush | Allowed | line 201 · Allow: / under User-agent: * |
MJ12bot | Majestic | Allowed | line 201 · Allow: / under User-agent: * |
DotBot | Moz | Allowed | line 201 · Allow: / under User-agent: * |
View www.houzz.com/robots.txt · 427 lines
1Sitemap: https://www.houzz.com/sitemap/v3/browse-photo/browse-photo_index_00.xml
2Sitemap: https://www.houzz.com/sitemap/v3/view-pro/view-pro_index_00.xml
3Sitemap: https://www.houzz.com/sitemap/v3/pro-directory/pro-directory_index_00.xml
4
5User-agent: *
6
7Disallow: /ad/*
8Disallow: /discussions/*sort=*
9Disallow: /discussions/*view=*
10Disallow: /proMatch/*
11Disallow: /writeReview/cmd=*
12Disallow: /writeReview2/cmd=*
13Disallow: /writeReview2/ew
14Disallow: /sketches/
15Disallow: */badges/*
16Disallow: /imageClipperUpload*
17Disallow: /edit/*
18Disallow: /*/tag=*
19Disallow: /pro2/*
20Disallow: /*spf=*
21Disallow: /trk/*
22Disallow: /hsc/*
23Disallow: *jsGalleryWidget
24Disallow: *reviewWidget
25Disallow: /feed/
26Disallow: /activities/
27Disallow: /proMatch/
28Disallow: */user/
29Disallow: /pro-site/
30Disallow: /serviceQuestionnaireAjax
31Disallow: /siteSwitch
32Disallow: /projects/
33Disallow: /proProjects/
34Disallow: /browseReviews
35Disallow: /error/
36Disallow: /reviewSeller
37Disallow: /houzz-historyzz
38Disallow: /j/pro-site
39Disallow: /http
40Disallow: /thumbs/
41
42# Marketing
43Disallow: /builders/
44Disallow: /designers/
45Disallow: /pro-meeting/
46Disallow: /estimates/*/client
47Disallow: /manage/leads/
48
49# Internal
50Disallow: /grow/
51Disallow: /settings/
52Disallow: /manage/
53Disallow: /pro-meeting/
54Disallow: /lead-form/
55Disallow: /editProfile/
56Disallow: /estimates/
57Disallow: /visual-tools/
58Disallow: /builders/
59Disallow: /designers/
60
61# CMS search pages
62Disallow: /pro-learn/s/
63Disallow: /pro-help/q/
64
65# Scholarship pages
66Disallow: /scholarships
67Disallow: *scholarship
68
69# Block buttonWidget and imageClipperUpload
70Disallow: *buttonWidget
71Disallow: *imageClipperUpload
72Disallow: *proVideoWidget
73
74# Facets
75Disallow: /photos/*a_86-
76Disallow: *phbr*_template_
77Allow: /photos/*phbr0
78Allow: /photos/*phbr1-
79Disallow: /photos/*phbr1-*a_86-
80Disallow: /photos/hznb/*phbr*
81Disallow: /photos/hznb/
82Disallow: /professionals/hznb/
83Disallow: /products/hznb/
84Disallow: /magazine/hznb/
85Disallow: /discussions/hznb/
86Disallow: *phbr
87Allow: /products/*prbr0
88Allow: /products/*prbr1
89Disallow: /products/hznb/*prbr
90Disallow: *prbr
91Disallow: *phbr~r_
92Disallow: *probr*_template_
93Allow: /professionals/*probr-1~/project-type--*/$
94Disallow: /professionals/hznb/*probr-1~/project-type--*/$
95Disallow: /professionals/*/*project-type-*-probr1-*_template_
96Allow: /professionals/*/*project-type-*-probr1-*
97Disallow: /professionals/hznb/*/*project-type-*-probr1-*
98Disallow: /professionals/hznb/*probr0*
99Allow: /professionals/*probr0*
100Disallow: /professionals/*probr0*_template_
101Disallow: *probr*
102
103# Query/search pages
104Disallow: */query/*
105Disallow: /professionals/*/s/
106Disallow: /professionals/s/
107Disallow: /s/*
108Disallow: */searchDirectory
109Disallow: /professionals/query*
110Disallow: *query=
111Disallow: *nqrwns
112Allow: /products/query/
113Allow: /photos/query/
114Disallow: /products/query/*nqrwns
115Disallow: *?oq=
116
117# Email
118Disallow: *OptOutEmail*
119Disallow: *optInEmail*
120
121# Old pages
122Disallow: /js/widget.js*
123Disallow: /js/ajaxconn*
124Disallow: *jsErrAjax
125Disallow: *uiLoggerAjax
126
127# Sort filters
128Disallow: /*/live-stream/
129Disallow: /*/popular-today/
130Disallow: /*/popular/
131Disallow: /*/avg-review/
132Disallow: /professionals/*/sortReviews
133
134# View filters
135Disallow: */ls=
136
137# Pros
138Disallow: /organizationMembers/
139
140# Reviews
141Disallow: /viewReview/
142
143# Shared bot exclusions
144Disallow: /getMoreSimilarGalleriesAjax
145# Block hznb URLs for default crawlers
146Disallow: *hznb*
147
148# Legacy
149Disallow: /sw$
150Disallow: /uk$
151Disallow: /au$
152Disallow: /de$
153Disallow: /fr$
154Disallow: /ru$
155Disallow: /gws$
156Disallow: /ie$
157Disallow: /nz$
158Disallow: /sg$
159Disallow: /in$
160Disallow: /se$
161Disallow: /jp$
162Disallow: /es$
163Disallow: /it$
164Disallow: /dk$
165Disallow: /sw/
166Disallow: /uk/
167Disallow: /au/
168Disallow: /de/
169Disallow: /fr/
170Disallow: /ru/
171Disallow: /ie/
172Disallow: /nz/
173Disallow: /sg/
174Disallow: /in/
175Disallow: /se/
176Disallow: /jp/
177Disallow: /es/
178Disallow: /it/
179Disallow: /dk/
180Disallow: /*/s/*
181Disallow: /*/vendorActivityManager/*
182
183# Cobrands
184Disallow: *?cbp
185
186# Ideabooks
187Disallow: */list/*
188Disallow: */plans/*
189Allow: /products/holiday-sales-*sgbr-sg*
190Disallow: *sgbr-sg
191
192# Old pages
193Allow: /pro-learn/blog/*error*
194Disallow: /get-hired/
195Disallow: /fbhome
196Disallow: *error*
197
198# Other
199Disallow: *_template
200Disallow: *?null
201Allow: /
202
203# AI bots
204User-agent: GPTBot
205User-agent: OAI-SearchBot
206User-agent: ChatGPT-User
207User-agent: ClaudeBot
208User-agent: PerplexityBot
209User-agent: Claude-User
210User-agent: Google-Extended
211
212Disallow: /ad/*
213Disallow: /discussions/*sort=*
214Disallow: /discussions/*view=*
215Disallow: /proMatch/*
216Disallow: /writeReview/cmd=*
217Disallow: /writeReview2/cmd=*
218Disallow: /writeReview2/ew
219Disallow: /sketches/
220Disallow: */badges/*
221Disallow: /imageClipperUpload*
222Disallow: /edit/*
223Disallow: /*/tag=*
224Disallow: /pro2/*
225Disallow: /*spf=*
226Disallow: /trk/*
227Disallow: /hsc/*
228Disallow: *jsGalleryWidget
229Disallow: *reviewWidget
230Disallow: /feed/
231Disallow: /activities/
232Disallow: /proMatch/
233Disallow: */user/
234Disallow: /pro-site/
235Disallow: /serviceQuestionnaireAjax
236Disallow: /siteSwitch
237Disallow: /projects/
238Disallow: /proProjects/
239Disallow: /browseReviews
240Disallow: /error/
241Disallow: /reviewSeller
242Disallow: /houzz-historyzz
243Disallow: /j/pro-site
244Disallow: /http
245Disallow: /thumbs/
246
247# Marketing
248Disallow: /builders/
249Disallow: /designers/
250Disallow: /pro-meeting/
251Disallow: /estimates/*/client
252Disallow: /manage/leads/
253
254# Internal
255Disallow: /grow/
256Disallow: /settings/
257Disallow: /manage/
258Disallow: /pro-meeting/
259Disallow: /lead-form/
260Disallow: /editProfile/
261Disallow: /estimates/
262Disallow: /visual-tools/
263Disallow: /builders/
264Disallow: /designers/
265
266# CMS search pages
267Disallow: /pro-learn/s/
268Disallow: /pro-help/q/
269
270# Scholarship pages
271Disallow: /scholarships
272Disallow: *scholarship
273
274# Block buttonWidget and imageClipperUpload
275Disallow: *buttonWidget
276Disallow: *imageClipperUpload
277Disallow: *proVideoWidget
278
279# Facets
280Disallow: /photos/*a_86-
281Disallow: *phbr*_template_
282Allow: /photos/*phbr0
283Allow: /photos/*phbr1-
284Disallow: /photos/*phbr1-*a_86-
285Disallow: /photos/hznb/*phbr*
286Disallow: /photos/hznb/
287Disallow: /professionals/hznb/
288Disallow: /products/hznb/
289Disallow: /magazine/hznb/
290Disallow: /discussions/hznb/
291Disallow: *phbr
292Allow: /products/*prbr0
293Allow: /products/*prbr1
294Disallow: /products/hznb/*prbr
295Disallow: *prbr
296Disallow: *phbr~r_
297Disallow: *probr*_template_
298Allow: /professionals/*probr-1~/project-type--*/$
299Disallow: /professionals/hznb/*probr-1~/project-type--*/$
300Disallow: /professionals/*/*project-type-*-probr1-*_template_
301Allow: /professionals/*/*project-type-*-probr1-*
302Disallow: /professionals/hznb/*/*project-type-*-probr1-*
303Disallow: /professionals/hznb/*probr0*
304Allow: /professionals/*probr0*
305Disallow: /professionals/*probr0*_template_
306Disallow: *probr*
307
308# Query/search pages
309Disallow: */query/*
310Disallow: /professionals/*/s/
311Disallow: /professionals/s/
312Disallow: /s/*
313Disallow: */searchDirectory
314Disallow: /professionals/query*
315Disallow: *query=
316Disallow: *nqrwns
317Allow: /products/query/
318Allow: /photos/query/
319Disallow: /products/query/*nqrwns
320Disallow: *?oq=
321
322# Email
323Disallow: *OptOutEmail*
324Disallow: *optInEmail*
325
326# Old pages
327Disallow: /js/widget.js*
328Disallow: /js/ajaxconn*
329Disallow: *jsErrAjax
330Disallow: *uiLoggerAjax
331
332# Sort filters
333Disallow: /*/live-stream/
334Disallow: /*/popular-today/
335Disallow: /*/popular/
336Disallow: /*/avg-review/
337Disallow: /professionals/*/sortReviews
338
339# View filters
340Disallow: */ls=
341
342# Pros
343Disallow: /organizationMembers/
344
345# Reviews
346Disallow: /viewReview/
347
348# Shared bot exclusions
349Disallow: /getMoreSimilarGalleriesAjax
350# AI project access
351Allow: /hznb/projects/
352Disallow: /hznb/projects/*?from=
353
354
355# Legacy
356Disallow: /sw$
357Disallow: /uk$
358Disallow: /au$
359Disallow: /de$
360Disallow: /fr$
361Disallow: /ru$
362Disallow: /gws$
363Disallow: /ie$
364Disallow: /nz$
365Disallow: /sg$
366Disallow: /in$
367Disallow: /se$
368Disallow: /jp$
369Disallow: /es$
370Disallow: /it$
371Disallow: /dk$
372Disallow: /sw/
373Disallow: /uk/
374Disallow: /au/
375Disallow: /de/
376Disallow: /fr/
377Disallow: /ru/
378Disallow: /ie/
379Disallow: /nz/
380Disallow: /sg/
381Disallow: /in/
382Disallow: /se/
383Disallow: /jp/
384Disallow: /es/
385Disallow: /it/
386Disallow: /dk/
387Disallow: /*/s/*
388Disallow: /*/vendorActivityManager/*
389
390# Cobrands
391Disallow: *?cbp
392
393# Ideabooks
394Disallow: */list/*
395Disallow: */plans/*
396Allow: /products/holiday-sales-*sgbr-sg*
397Disallow: *sgbr-sg
398
399# Old pages
400Allow: /pro-learn/blog/*error*
401Disallow: /get-hired/
402Disallow: /fbhome
403Disallow: *error*
404
405# Other
406Disallow: *_template
407Disallow: *?null
408Allow: /
409
410# Twitter
411User-agent: Twitterbot
412Allow: *sgbr-sg
413Allow: *phvw-vp
414Allow: /magazine/
415Allow: /ideabooks/
416Allow: /pro-learn/
417Disallow: /discussions/*/p/_template_
418Allow: /discussions/
419Disallow: /
420
421# AdsBot
422User-agent: AdsBot-Google
423User-agent: AdsBot-Google-Mobile
424Disallow: /js/widget.js*
425Disallow: /js/ajaxconn*
426Disallow: *jsErrAjax
427Disallow: *uiLoggerAjax
What do the bots actually get?
This page reads robots.txt, which states what houzz.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on houzz.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked, robots.txt couldn’t be read.
- 9 November 2026: robots.txt readable again.
Questions
Does houzz.com block GPTBot?
Partly. houzz.com’s robots.txt keeps GPTBot out of some paths, such as /ad/*, and lets it crawl the rest. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does houzz.com block ClaudeBot?
Partly. houzz.com’s robots.txt keeps ClaudeBot out of some paths, such as /ad/*, and lets it crawl the rest. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does houzz.com block Google-Extended?
Partly. houzz.com’s robots.txt keeps Google-Extended out of some paths, such as /ad/*, and lets it crawl the rest. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does houzz.com block PerplexityBot?
Partly. houzz.com’s robots.txt keeps PerplexityBot out of some paths, such as /ad/*, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find houzz.com?
Partly. houzz.com’s robots.txt keeps OAI-SearchBot out of some paths, such as /ad/*, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.