Who blocks AI · #533 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does airbnb.com block AI crawlers?
Yes. airbnb.com’s robots.txt blocks 3 of the 19 AI crawlers we track, including GPTBot, ClaudeBot and Applebot-Extended. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotPartly blockedClaudeBotBlockedPerplexityBotPartly blockedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.airbnb.com/robots.txt · 780 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 28 · Disallow: /.well-known/assetlinks.json line 29 · Disallow: /*/skeleton and 59 more | |
Bingbot | Microsoft | Partly blocked | line 93 · Disallow: /.well-known/assetlinks.json line 94 · Disallow: /*/skeleton and 59 more |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Partly blocked | line 28 · Disallow: /.well-known/assetlinks.json line 29 · Disallow: /*/skeleton and 59 more |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Partly blocked | line 158 · Disallow: /.well-known/assetlinks.json line 159 · Disallow: /*/skeleton and 59 more |
Baiduspider | Baidu | Partly blocked | line 223 · Disallow: /.well-known/assetlinks.json line 224 · Disallow: /*/skeleton and 59 more |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Partly blocked | line 288 · Disallow: /.well-known/assetlinks.json line 289 · Disallow: /*/skeleton and 59 more |
PerplexityBot | Perplexity | Partly blocked | line 483 · Disallow: /.well-known/assetlinks.json line 484 · Disallow: /*/skeleton and 61 more |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 691 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 695 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Blocked | line 699 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Partly blocked | line 418 · Disallow: /.well-known/assetlinks.json line 419 · Disallow: /*/skeleton and 59 more |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Partly blocked | line 353 · Disallow: /.well-known/assetlinks.json line 354 · Disallow: /*/skeleton and 59 more |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.airbnb.com/robots.txt · 780 lines
1#
2# ///////
3# // //
4# // //
5# // // /// /// ///
6# // // /// ///
7# // /// // //// /// /// /// //// /// //// /// //// /// ////
8# // /// /// // ////////// /// ////////// /////////// ////////// ///////////
9# // // // // /// /// /// /// /// /// /// /// /// ///
10# // // // // /// /// /// /// /// /// /// /// /// ///
11# // // // // /// /// /// /// /// /// /// /// /// ///
12# // // // // ////////// /// /// ////////// /// /// //////////
13# // ///// //
14# // ///// //
15# // /// /// //
16# ////// //////
17#
18#
19# Hello bot, engineer, or very lost layperson! Welcome to your stay on the Airbnb site, we're happy to have you as a guest.
20# If you're a human who likes solving interesting challenges with other humans, check out our careers page: https://careers.airbnb.com/positions/?_departments=engineering
21# If you're a bot who likes crawling webpages, please mind the house rules and avoid accessing any disallowed subfolders to earn a 5-star review from us.
22# Either way, thanks for stopping by! There's no need to collect your garbage at the end of your stay - we use Javascript for that.
23# See you on the next crawl!
24
25User-agent: Googlebot
26Allow: /calendar/ical/
27Allow: /.well-known/amphtml/apikey.pub
28Disallow: /.well-known/assetlinks.json
29Disallow: /*/skeleton
30Disallow: /*/sw_skeleton
31Disallow: /500
32Disallow: /account
33Disallow: /alumni
34Disallow: /api/v1/trebuchet
35Disallow: /associates/click
36Disallow: /book/
37Disallow: /calendar/
38Disallow: /contact_host
39Disallow: /disaster/lookup
40Disallow: /email/unsubscribe
41Disallow: /embeddable
42Disallow: /experiences/*?*scheduled_id
43Disallow: /experiences/*?*modal
44Disallow: /experiences/*/book
45Disallow: /external_link?
46Disallow: /fix-it
47Disallow: /fixit
48Disallow: /forgot_password
49Disallow: /google_place_photo
50Disallow: /api/v2/google_place_photos
51Disallow: /groups
52Disallow: /guidebooks
53Disallow: /help/feedback
54Disallow: /help/search
55Disallow: /help/search
56Disallow: /home/dashboard
57Disallow: /inbox
58Disallow: /login_with_redirect
59Disallow: /logout
60Disallow: /manage-listing
61Disallow: /messaging/ajax_already_messaged/
62Disallow: /my_listings
63Disallow: /oauth_connect
64Disallow: /payments/book
65Disallow: /reservation
66Disallow: /rooms/*/amenities
67Disallow: /rooms/*/enhanced-cleaning
68Disallow: /rooms/*/house-rules
69Disallow: /rooms/*/location
70Disallow: /rooms/*/photos
71Disallow: /rooms/*/reviews
72Disallow: /rooms/*/safety
73Disallow: /rooms/*?viralityEntryPoint
74Disallow: /rooms/*/cancellation-policy
75Disallow: /rooms/*/description
76Disallow: /s/guidebooks
77Disallow: /signed_out_modal.json
78Disallow: /signup_modal
79Disallow: /stories
80Disallow: /trips/upcoming
81Disallow: /trips/v1/
82Disallow: /update-your-browser
83Disallow: /users/*/listings
84Disallow: /users/show
85Disallow: /users/profile
86Disallow: /api/v3/LandingPageStaysQuery
87Disallow: /s/*/*
88Disallow: /things-to-do/places
89
90User-agent: Bingbot
91Allow: /calendar/ical/
92Allow: /.well-known/amphtml/apikey.pub
93Disallow: /.well-known/assetlinks.json
94Disallow: /*/skeleton
95Disallow: /*/sw_skeleton
96Disallow: /500
97Disallow: /account
98Disallow: /alumni
99Disallow: /api/v1/trebuchet
100Disallow: /associates/click
101Disallow: /book/
102Disallow: /calendar/
103Disallow: /contact_host
104Disallow: /disaster/lookup
105Disallow: /email/unsubscribe
106Disallow: /embeddable
107Disallow: /experiences/*?*scheduled_id
108Disallow: /experiences/*?*modal
109Disallow: /experiences/*/book
110Disallow: /external_link?
111Disallow: /fix-it
112Disallow: /fixit
113Disallow: /forgot_password
114Disallow: /google_place_photo
115Disallow: /api/v2/google_place_photos
116Disallow: /groups
117Disallow: /guidebooks
118Disallow: /help/feedback
119Disallow: /help/search
120Disallow: /help/search
121Disallow: /home/dashboard
122Disallow: /inbox
123Disallow: /login_with_redirect
124Disallow: /logout
125Disallow: /manage-listing
126Disallow: /messaging/ajax_already_messaged/
127Disallow: /my_listings
128Disallow: /oauth_connect
129Disallow: /payments/book
130Disallow: /reservation
131Disallow: /rooms/*/amenities
132Disallow: /rooms/*/enhanced-cleaning
133Disallow: /rooms/*/house-rules
134Disallow: /rooms/*/location
135Disallow: /rooms/*/photos
136Disallow: /rooms/*/reviews
137Disallow: /rooms/*/safety
138Disallow: /rooms/*?viralityEntryPoint
139Disallow: /rooms/*/cancellation-policy
140Disallow: /rooms/*/description
141Disallow: /s/guidebooks
142Disallow: /signed_out_modal.json
143Disallow: /signup_modal
144Disallow: /stories
145Disallow: /trips/upcoming
146Disallow: /trips/v1/
147Disallow: /update-your-browser
148Disallow: /users/*/listings
149Disallow: /users/show
150Disallow: /users/profile
151Disallow: /api/v3/LandingPageStaysQuery
152Disallow: /s/*/*
153Disallow: /things-to-do/places
154
155User-agent: YandexBot
156Allow: /calendar/ical/
157Allow: /.well-known/amphtml/apikey.pub
158Disallow: /.well-known/assetlinks.json
159Disallow: /*/skeleton
160Disallow: /*/sw_skeleton
161Disallow: /500
162Disallow: /account
163Disallow: /alumni
164Disallow: /api/v1/trebuchet
165Disallow: /associates/click
166Disallow: /book/
167Disallow: /calendar/
168Disallow: /contact_host
169Disallow: /disaster/lookup
170Disallow: /email/unsubscribe
171Disallow: /embeddable
172Disallow: /experiences/*?*scheduled_id
173Disallow: /experiences/*?*modal
174Disallow: /experiences/*/book
175Disallow: /external_link?
176Disallow: /fix-it
177Disallow: /fixit
178Disallow: /forgot_password
179Disallow: /google_place_photo
180Disallow: /api/v2/google_place_photos
181Disallow: /groups
182Disallow: /guidebooks
183Disallow: /help/feedback
184Disallow: /help/search
185Disallow: /help/search
186Disallow: /home/dashboard
187Disallow: /inbox
188Disallow: /login_with_redirect
189Disallow: /logout
190Disallow: /manage-listing
191Disallow: /messaging/ajax_already_messaged/
192Disallow: /my_listings
193Disallow: /oauth_connect
194Disallow: /payments/book
195Disallow: /reservation
196Disallow: /rooms/*/amenities
197Disallow: /rooms/*/enhanced-cleaning
198Disallow: /rooms/*/house-rules
199Disallow: /rooms/*/location
200Disallow: /rooms/*/photos
201Disallow: /rooms/*/reviews
202Disallow: /rooms/*/safety
203Disallow: /rooms/*?viralityEntryPoint
204Disallow: /rooms/*/cancellation-policy
205Disallow: /rooms/*/description
206Disallow: /s/guidebooks
207Disallow: /signed_out_modal.json
208Disallow: /signup_modal
209Disallow: /stories
210Disallow: /trips/upcoming
211Disallow: /trips/v1/
212Disallow: /update-your-browser
213Disallow: /users/*/listings
214Disallow: /users/show
215Disallow: /users/profile
216Disallow: /api/v3/LandingPageStaysQuery
217Disallow: /s/*/*
218Disallow: /things-to-do/places
219
220User-agent: Baiduspider
221Allow: /calendar/ical/
222Allow: /.well-known/amphtml/apikey.pub
223Disallow: /.well-known/assetlinks.json
224Disallow: /*/skeleton
225Disallow: /*/sw_skeleton
226Disallow: /500
227Disallow: /account
228Disallow: /alumni
229Disallow: /api/v1/trebuchet
230Disallow: /associates/click
231Disallow: /book/
232Disallow: /calendar/
233Disallow: /contact_host
234Disallow: /disaster/lookup
235Disallow: /email/unsubscribe
236Disallow: /embeddable
237Disallow: /experiences/*?*scheduled_id
238Disallow: /experiences/*?*modal
239Disallow: /experiences/*/book
240Disallow: /external_link?
241Disallow: /fix-it
242Disallow: /fixit
243Disallow: /forgot_password
244Disallow: /google_place_photo
245Disallow: /api/v2/google_place_photos
246Disallow: /groups
247Disallow: /guidebooks
248Disallow: /help/feedback
249Disallow: /help/search
250Disallow: /help/search
251Disallow: /home/dashboard
252Disallow: /inbox
253Disallow: /login_with_redirect
254Disallow: /logout
255Disallow: /manage-listing
256Disallow: /messaging/ajax_already_messaged/
257Disallow: /my_listings
258Disallow: /oauth_connect
259Disallow: /payments/book
260Disallow: /reservation
261Disallow: /rooms/*/amenities
262Disallow: /rooms/*/enhanced-cleaning
263Disallow: /rooms/*/house-rules
264Disallow: /rooms/*/location
265Disallow: /rooms/*/photos
266Disallow: /rooms/*/reviews
267Disallow: /rooms/*/safety
268Disallow: /rooms/*?viralityEntryPoint
269Disallow: /rooms/*/cancellation-policy
270Disallow: /rooms/*/description
271Disallow: /s/guidebooks
272Disallow: /signed_out_modal.json
273Disallow: /signup_modal
274Disallow: /stories
275Disallow: /trips/upcoming
276Disallow: /trips/v1/
277Disallow: /update-your-browser
278Disallow: /users/*/listings
279Disallow: /users/show
280Disallow: /users/profile
281Disallow: /api/v3/LandingPageStaysQuery
282Disallow: /s/*/*
283Disallow: /things-to-do/places
284
285User-agent: OAI-Searchbot
286Allow: /calendar/ical/
287Allow: /.well-known/amphtml/apikey.pub
288Disallow: /.well-known/assetlinks.json
289Disallow: /*/skeleton
290Disallow: /*/sw_skeleton
291Disallow: /500
292Disallow: /account
293Disallow: /alumni
294Disallow: /api/v1/trebuchet
295Disallow: /associates/click
296Disallow: /book/
297Disallow: /calendar/
298Disallow: /contact_host
299Disallow: /disaster/lookup
300Disallow: /email/unsubscribe
301Disallow: /embeddable
302Disallow: /experiences/*?*scheduled_id
303Disallow: /experiences/*?*modal
304Disallow: /experiences/*/book
305Disallow: /external_link?
306Disallow: /fix-it
307Disallow: /fixit
308Disallow: /forgot_password
309Disallow: /google_place_photo
310Disallow: /api/v2/google_place_photos
311Disallow: /groups
312Disallow: /guidebooks
313Disallow: /help/feedback
314Disallow: /help/search
315Disallow: /help/search
316Disallow: /home/dashboard
317Disallow: /inbox
318Disallow: /login_with_redirect
319Disallow: /logout
320Disallow: /manage-listing
321Disallow: /messaging/ajax_already_messaged/
322Disallow: /my_listings
323Disallow: /oauth_connect
324Disallow: /payments/book
325Disallow: /reservation
326Disallow: /rooms/*/amenities
327Disallow: /rooms/*/enhanced-cleaning
328Disallow: /rooms/*/house-rules
329Disallow: /rooms/*/location
330Disallow: /rooms/*/photos
331Disallow: /rooms/*/reviews
332Disallow: /rooms/*/safety
333Disallow: /rooms/*?viralityEntryPoint
334Disallow: /rooms/*/cancellation-policy
335Disallow: /rooms/*/description
336Disallow: /s/guidebooks
337Disallow: /signed_out_modal.json
338Disallow: /signup_modal
339Disallow: /stories
340Disallow: /trips/upcoming
341Disallow: /trips/v1/
342Disallow: /update-your-browser
343Disallow: /users/*/listings
344Disallow: /users/show
345Disallow: /users/profile
346Disallow: /api/v3/LandingPageStaysQuery
347Disallow: /s/*/*
348Disallow: /things-to-do/places
349
350User-agent: ChatGPT-User
351Allow: /calendar/ical/
352Allow: /.well-known/amphtml/apikey.pub
353Disallow: /.well-known/assetlinks.json
354Disallow: /*/skeleton
355Disallow: /*/sw_skeleton
356Disallow: /500
357Disallow: /account
358Disallow: /alumni
359Disallow: /api/v1/trebuchet
360Disallow: /associates/click
361Disallow: /book/
362Disallow: /calendar/
363Disallow: /contact_host
364Disallow: /disaster/lookup
365Disallow: /email/unsubscribe
366Disallow: /embeddable
367Disallow: /experiences/*?*scheduled_id
368Disallow: /experiences/*?*modal
369Disallow: /experiences/*/book
370Disallow: /external_link?
371Disallow: /fix-it
372Disallow: /fixit
373Disallow: /forgot_password
374Disallow: /google_place_photo
375Disallow: /api/v2/google_place_photos
376Disallow: /groups
377Disallow: /guidebooks
378Disallow: /help/feedback
379Disallow: /help/search
380Disallow: /help/search
381Disallow: /home/dashboard
382Disallow: /inbox
383Disallow: /login_with_redirect
384Disallow: /logout
385Disallow: /manage-listing
386Disallow: /messaging/ajax_already_messaged/
387Disallow: /my_listings
388Disallow: /oauth_connect
389Disallow: /payments/book
390Disallow: /reservation
391Disallow: /rooms/*/amenities
392Disallow: /rooms/*/enhanced-cleaning
393Disallow: /rooms/*/house-rules
394Disallow: /rooms/*/location
395Disallow: /rooms/*/photos
396Disallow: /rooms/*/reviews
397Disallow: /rooms/*/safety
398Disallow: /rooms/*?viralityEntryPoint
399Disallow: /rooms/*/cancellation-policy
400Disallow: /rooms/*/description
401Disallow: /s/guidebooks
402Disallow: /signed_out_modal.json
403Disallow: /signup_modal
404Disallow: /stories
405Disallow: /trips/upcoming
406Disallow: /trips/v1/
407Disallow: /update-your-browser
408Disallow: /users/*/listings
409Disallow: /users/show
410Disallow: /users/profile
411Disallow: /api/v3/LandingPageStaysQuery
412Disallow: /s/*/*
413Disallow: /things-to-do/places
414
415User-agent: meta-externalagent
416Allow: /calendar/ical/
417Allow: /.well-known/amphtml/apikey.pub
418Disallow: /.well-known/assetlinks.json
419Disallow: /*/skeleton
420Disallow: /*/sw_skeleton
421Disallow: /500
422Disallow: /account
423Disallow: /alumni
424Disallow: /api/v1/trebuchet
425Disallow: /associates/click
426Disallow: /book/
427Disallow: /calendar/
428Disallow: /contact_host
429Disallow: /disaster/lookup
430Disallow: /email/unsubscribe
431Disallow: /embeddable
432Disallow: /experiences/*?*scheduled_id
433Disallow: /experiences/*?*modal
434Disallow: /experiences/*/book
435Disallow: /external_link?
436Disallow: /fix-it
437Disallow: /fixit
438Disallow: /forgot_password
439Disallow: /google_place_photo
440Disallow: /api/v2/google_place_photos
441Disallow: /groups
442Disallow: /guidebooks
443Disallow: /help/feedback
444Disallow: /help/search
445Disallow: /help/search
446Disallow: /home/dashboard
447Disallow: /inbox
448Disallow: /login_with_redirect
449Disallow: /logout
450Disallow: /manage-listing
451Disallow: /messaging/ajax_already_messaged/
452Disallow: /my_listings
453Disallow: /oauth_connect
454Disallow: /payments/book
455Disallow: /reservation
456Disallow: /rooms/*/amenities
457Disallow: /rooms/*/enhanced-cleaning
458Disallow: /rooms/*/house-rules
459Disallow: /rooms/*/location
460Disallow: /rooms/*/photos
461Disallow: /rooms/*/reviews
462Disallow: /rooms/*/safety
463Disallow: /rooms/*?viralityEntryPoint
464Disallow: /rooms/*/cancellation-policy
465Disallow: /rooms/*/description
466Disallow: /s/guidebooks
467Disallow: /signed_out_modal.json
468Disallow: /signup_modal
469Disallow: /stories
470Disallow: /trips/upcoming
471Disallow: /trips/v1/
472Disallow: /update-your-browser
473Disallow: /users/*/listings
474Disallow: /users/show
475Disallow: /users/profile
476Disallow: /api/v3/LandingPageStaysQuery
477Disallow: /s/*/*
478Disallow: /things-to-do/places
479
480User-agent: PerplexityBot
481Allow: /calendar/ical/
482Allow: /.well-known/amphtml/apikey.pub
483Disallow: /.well-known/assetlinks.json
484Disallow: /*/skeleton
485Disallow: /*/sw_skeleton
486Disallow: /500
487Disallow: /account
488Disallow: /alumni
489Disallow: /api/v1/trebuchet
490Disallow: /associates/click
491Disallow: /book/
492Disallow: /calendar/
493Disallow: /contact_host
494Disallow: /disaster/lookup
495Disallow: /email/unsubscribe
496Disallow: /embeddable
497Disallow: /experiences/*?*scheduled_id
498Disallow: /experiences/*?*modal
499Disallow: /experiences/*/book
500Disallow: /external_link?
501Disallow: /fix-it
502Disallow: /fixit
503Disallow: /forgot_password
504Disallow: /google_place_photo
505Disallow: /api/v2/google_place_photos
506Disallow: /groups
507Disallow: /guidebooks
508Disallow: /help/feedback
509Disallow: /help/search
510Disallow: /help/search
511Disallow: /home/dashboard
512Disallow: /inbox
513Disallow: /login_with_redirect
514Disallow: /logout
515Disallow: /manage-listing
516Disallow: /messaging/ajax_already_messaged/
517Disallow: /my_listings
518Disallow: /oauth_connect
519Disallow: /payments/book
520Disallow: /reservation
521Disallow: /rooms/*/amenities
522Disallow: /rooms/*/enhanced-cleaning
523Disallow: /rooms/*/house-rules
524Disallow: /rooms/*/location
525Disallow: /rooms/*/photos
526Disallow: /rooms/*/reviews
527Disallow: /rooms/*/safety
528Disallow: /rooms/*?viralityEntryPoint
529Disallow: /rooms/*/cancellation-policy
530Disallow: /rooms/*/description
531Disallow: /s/guidebooks
532Disallow: /signed_out_modal.json
533Disallow: /signup_modal
534Disallow: /stories
535Disallow: /trips/upcoming
536Disallow: /trips/v1/
537Disallow: /update-your-browser
538Disallow: /users/*/listings
539Disallow: /users/show
540Disallow: /users/profile
541Disallow: /api/v3/LandingPageStaysQuery
542Disallow: /s/*/*
543Disallow: /things-to-do/places
544Disallow: /rooms/
545Disallow: /experiences/
546
547User-agent: anthropic-ai
548Allow: /calendar/ical/
549Allow: /.well-known/amphtml/apikey.pub
550Disallow: /.well-known/assetlinks.json
551Disallow: /*/skeleton
552Disallow: /*/sw_skeleton
553Disallow: /500
554Disallow: /account
555Disallow: /alumni
556Disallow: /api/v1/trebuchet
557Disallow: /associates/click
558Disallow: /book/
559Disallow: /calendar/
560Disallow: /contact_host
561Disallow: /disaster/lookup
562Disallow: /email/unsubscribe
563Disallow: /embeddable
564Disallow: /experiences/*?*scheduled_id
565Disallow: /experiences/*?*modal
566Disallow: /experiences/*/book
567Disallow: /external_link?
568Disallow: /fix-it
569Disallow: /fixit
570Disallow: /forgot_password
571Disallow: /google_place_photo
572Disallow: /api/v2/google_place_photos
573Disallow: /groups
574Disallow: /guidebooks
575Disallow: /help/feedback
576Disallow: /help/search
577Disallow: /help/search
578Disallow: /home/dashboard
579Disallow: /inbox
580Disallow: /login_with_redirect
581Disallow: /logout
582Disallow: /manage-listing
583Disallow: /messaging/ajax_already_messaged/
584Disallow: /my_listings
585Disallow: /oauth_connect
586Disallow: /payments/book
587Disallow: /reservation
588Disallow: /rooms/*/amenities
589Disallow: /rooms/*/enhanced-cleaning
590Disallow: /rooms/*/house-rules
591Disallow: /rooms/*/location
592Disallow: /rooms/*/photos
593Disallow: /rooms/*/reviews
594Disallow: /rooms/*/safety
595Disallow: /rooms/*?viralityEntryPoint
596Disallow: /rooms/*/cancellation-policy
597Disallow: /rooms/*/description
598Disallow: /s/guidebooks
599Disallow: /signed_out_modal.json
600Disallow: /signup_modal
601Disallow: /stories
602Disallow: /trips/upcoming
603Disallow: /trips/v1/
604Disallow: /update-your-browser
605Disallow: /users/*/listings
606Disallow: /users/show
607Disallow: /users/profile
608Disallow: /api/v3/LandingPageStaysQuery
609Disallow: /s/*/*
610Disallow: /things-to-do/places
611Disallow: /rooms/
612Disallow: /experiences/
613
614User-agent: cohere-ai
615Allow: /calendar/ical/
616Allow: /.well-known/amphtml/apikey.pub
617Disallow: /.well-known/assetlinks.json
618Disallow: /*/skeleton
619Disallow: /*/sw_skeleton
620Disallow: /500
621Disallow: /account
622Disallow: /alumni
623Disallow: /api/v1/trebuchet
624Disallow: /associates/click
625Disallow: /book/
626Disallow: /calendar/
627Disallow: /contact_host
628Disallow: /disaster/lookup
629Disallow: /email/unsubscribe
630Disallow: /embeddable
631Disallow: /experiences/*?*scheduled_id
632Disallow: /experiences/*?*modal
633Disallow: /experiences/*/book
634Disallow: /external_link?
635Disallow: /fix-it
636Disallow: /fixit
637Disallow: /forgot_password
638Disallow: /google_place_photo
639Disallow: /api/v2/google_place_photos
640Disallow: /groups
641Disallow: /guidebooks
642Disallow: /help/feedback
643Disallow: /help/search
644Disallow: /help/search
645Disallow: /home/dashboard
646Disallow: /inbox
647Disallow: /login_with_redirect
648Disallow: /logout
649Disallow: /manage-listing
650Disallow: /messaging/ajax_already_messaged/
651Disallow: /my_listings
652Disallow: /oauth_connect
653Disallow: /payments/book
654Disallow: /reservation
655Disallow: /rooms/*/amenities
656Disallow: /rooms/*/enhanced-cleaning
657Disallow: /rooms/*/house-rules
658Disallow: /rooms/*/location
659Disallow: /rooms/*/photos
660Disallow: /rooms/*/reviews
661Disallow: /rooms/*/safety
662Disallow: /rooms/*?viralityEntryPoint
663Disallow: /rooms/*/cancellation-policy
664Disallow: /rooms/*/description
665Disallow: /s/guidebooks
666Disallow: /signed_out_modal.json
667Disallow: /signup_modal
668Disallow: /stories
669Disallow: /trips/upcoming
670Disallow: /trips/v1/
671Disallow: /update-your-browser
672Disallow: /users/*/listings
673Disallow: /users/show
674Disallow: /users/profile
675Disallow: /api/v3/LandingPageStaysQuery
676Disallow: /s/*/*
677Disallow: /things-to-do/places
678Disallow: /rooms/
679Disallow: /experiences/
680Disallow: /help/
681Disallow: /*/things-to-do
682Disallow: /*/stays/
683
684User-agent: ImagesiftBot
685Allow: /calendar/ical/
686Allow: /.well-known/amphtml/apikey.pub
687Disallow: /
688
689
690User-agent: GPTBot
691Disallow: /
692
693
694User-agent: ClaudeBot
695Disallow: /
696
697
698User-agent: Applebot-Extended
699Disallow: /
700
701
702User-agent: Webzio-Extended
703Disallow: /
704
705
706User-agent: AI2Bot
707Disallow: /
708
709
710User-agent: MistralAI-Training
711Disallow: /
712
713User-agent: *
714Allow: /calendar/ical/
715Allow: /.well-known/amphtml/apikey.pub
716Disallow: /.well-known/assetlinks.json
717Disallow: /*/skeleton
718Disallow: /*/sw_skeleton
719Disallow: /500
720Disallow: /account
721Disallow: /alumni
722Disallow: /api/v1/trebuchet
723Disallow: /associates/click
724Disallow: /book/
725Disallow: /calendar/
726Disallow: /contact_host
727Disallow: /disaster/lookup
728Disallow: /email/unsubscribe
729Disallow: /embeddable
730Disallow: /experiences/*?*scheduled_id
731Disallow: /experiences/*?*modal
732Disallow: /experiences/*/book
733Disallow: /external_link?
734Disallow: /fix-it
735Disallow: /fixit
736Disallow: /forgot_password
737Disallow: /google_place_photo
738Disallow: /api/v2/google_place_photos
739Disallow: /groups
740Disallow: /guidebooks
741Disallow: /help/feedback
742Disallow: /help/search
743Disallow: /help/search
744Disallow: /home/dashboard
745Disallow: /inbox
746Disallow: /login_with_redirect
747Disallow: /logout
748Disallow: /manage-listing
749Disallow: /messaging/ajax_already_messaged/
750Disallow: /my_listings
751Disallow: /oauth_connect
752Disallow: /payments/book
753Disallow: /reservation
754Disallow: /rooms/*/amenities
755Disallow: /rooms/*/enhanced-cleaning
756Disallow: /rooms/*/house-rules
757Disallow: /rooms/*/location
758Disallow: /rooms/*/photos
759Disallow: /rooms/*/reviews
760Disallow: /rooms/*/safety
761Disallow: /rooms/*?viralityEntryPoint
762Disallow: /rooms/*/cancellation-policy
763Disallow: /rooms/*/description
764Disallow: /s/guidebooks
765Disallow: /signed_out_modal.json
766Disallow: /signup_modal
767Disallow: /stories
768Disallow: /trips/upcoming
769Disallow: /trips/v1/
770Disallow: /update-your-browser
771Disallow: /users/*/listings
772Disallow: /users/show
773Disallow: /users/profile
774Disallow: /api/v3/LandingPageStaysQuery
775Disallow: /s/*/*
776Disallow: /things-to-do/places
777
778Sitemap: https://www.airbnb.com/sitemap-master-index.xml.gz
779
780
What do the bots actually get?
This page reads robots.txt, which states what airbnb.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on airbnb.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does airbnb.com block GPTBot?
Yes. airbnb.com’s robots.txt disallows GPTBot from the whole site (line 691: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does airbnb.com block ClaudeBot?
Yes. airbnb.com’s robots.txt disallows ClaudeBot from the whole site (line 695: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does airbnb.com block Google-Extended?
No. airbnb.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does airbnb.com block PerplexityBot?
Partly. airbnb.com’s robots.txt keeps PerplexityBot out of some paths, such as /.well-known/assetlinks.json, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find airbnb.com?
Partly. airbnb.com’s robots.txt keeps OAI-SearchBot out of some paths, such as /.well-known/assetlinks.json, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.