Who blocks AI · #269 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does yelp.com block AI crawlers?
Yes. yelp.com’s robots.txt blocks all 19 AI crawlers we track, from GPTBot and ClaudeBot to PerplexityBot. It also blocks DuckDuckBot, PetalBot and 2 more among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.yelp.com/robots.txt · 366 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 98 · Disallow: *US_CENSUS_NAME* line 99 · Disallow: *US_FEMALE_NAME* and 226 more | |
Bingbot | Microsoft | Partly blocked | line 98 · Disallow: *US_CENSUS_NAME* line 99 · Disallow: *US_FEMALE_NAME* and 127 more |
DuckDuckBot | DuckDuckGo | Blocked | line 365 · Disallow: / under User-agent: * |
Applebot | Apple | Partly blocked | line 98 · Disallow: *US_CENSUS_NAME* line 99 · Disallow: *US_FEMALE_NAME* and 226 more |
PetalBot | Huawei | Blocked | line 365 · Disallow: / under User-agent: * |
YandexBot | Yandex | Blocked | line 365 · Disallow: / under User-agent: * |
Baiduspider | Baidu | Blocked | line 365 · Disallow: / under User-agent: * |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 341 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 351 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 351 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 365 · Disallow: / under User-agent: * |
Amazonbot | Amazon | Blocked | line 365 · Disallow: / under User-agent: * |
YouBot | You.com | Blocked | line 365 · Disallow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 362 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 362 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 362 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 362 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 365 · Disallow: / under User-agent: * |
GoogleOther | Blocked | line 365 · Disallow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Blocked | line 362 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 365 · Disallow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 351 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 351 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 351 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 365 · Disallow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Blocked | line 362 · Disallow: / under User-agent: Meta-ExternalFetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 365 · Disallow: / under User-agent: * |
SemrushBot | Semrush | Blocked | line 365 · Disallow: / under User-agent: * |
MJ12bot | Majestic | Blocked | line 365 · Disallow: / under User-agent: * |
DotBot | Moz | Blocked | line 365 · Disallow: / under User-agent: * |
View www.yelp.com/robots.txt · 366 lines
1# By accessing Yelp's website (© 2026) you agree to Yelp's Terms of Service, available at
2# https://www.yelp.com/static?country=US&p=tos
3#
4# If you would like to inquire about crawling Yelp, please contact us at
5# https://www.yelp.com/contact
6#
7# As always, Asimov's Three Laws are in effect:
8# 1. A robot may not injure a human being or, through inaction, allow a human
9# being to come to harm.
10# 2. A robot must obey orders given it by human beings except where such
11# orders would conflict with the First Law.
12# 3. A robot must protect its own existence as long as such protection does
13# not conflict with the First or Second Law.
14#
15# AI / LLM Crawlers and Agents:
16# Use of any robot, spider, service search/retrieval application, or other automated device, process or means to access, retrieve,
17# copy, scrape, or index any portion of the service or any content is prohibited, except as expressly permitted by Yelp.
18# If you would like to request permission to crawl or use Yelp content, contact us here: https://www.yelp.com/contact
19
20User-Agent: Google-InspectionTool
21User-Agent: Googlebot
22User-Agent: Googlebot-Image
23User-Agent: Googlebot-Mobile
24User-Agent: Googlebot-Video
25Allow: /biz/taste-of-lurgan-lost-springs
26Allow: /biz_photos/*-albuquerque*
27Allow: /biz_photos/*-anaheim*
28Allow: /biz_photos/*-atlanta*
29Allow: /biz_photos/*-austin*
30Allow: /biz_photos/*-baltimore*
31Allow: /biz_photos/*-boston*
32Allow: /biz_photos/*-brooklyn*
33Allow: /biz_photos/*-charlotte*
34Allow: /biz_photos/*-chicago*
35Allow: /biz_photos/*-cincinnati*
36Allow: /biz_photos/*-columbus*
37Allow: /biz_photos/*-dallas*
38Allow: /biz_photos/*-denver*
39Allow: /biz_photos/*-fort-worth*
40Allow: /biz_photos/*-fresno*
41Allow: /biz_photos/*-honolulu*
42Allow: /biz_photos/*-houston*
43Allow: /biz_photos/*-indianapolis*
44Allow: /biz_photos/*-irvine*
45Allow: /biz_photos/*-jacksonville*
46Allow: /biz_photos/*-las-vegas*
47Allow: /biz_photos/*-long-beach*
48Allow: /biz_photos/*-los-angeles*
49Allow: /biz_photos/*-louisville*
50Allow: /biz_photos/*-manhattan*
51Allow: /biz_photos/*-miami*
52Allow: /biz_photos/*-minneapolis*
53Allow: /biz_photos/*-nashville*
54Allow: /biz_photos/*-new-orleans*
55Allow: /biz_photos/*-new-york*
56Allow: /biz_photos/*-oakland*
57Allow: /biz_photos/*-oklahoma-city*
58Allow: /biz_photos/*-orlando*
59Allow: /biz_photos/*-philadelphia*
60Allow: /biz_photos/*-phoenix*
61Allow: /biz_photos/*-pittsburgh*
62Allow: /biz_photos/*-portland*
63Allow: /biz_photos/*-raleigh*
64Allow: /biz_photos/*-richmond*
65Allow: /biz_photos/*-sacramento*
66Allow: /biz_photos/*-san-antonio*
67Allow: /biz_photos/*-san-diego*
68Allow: /biz_photos/*-san-francisco*
69Allow: /biz_photos/*-san-jose*
70Allow: /biz_photos/*-scottsdale*
71Allow: /biz_photos/*-seattle*
72Allow: /biz_photos/*-tampa*
73Allow: /biz_photos/*-tampa-bay*
74Allow: /biz_photos/*-tucson*
75Allow: /biz_photos/*-washington*
76Allow: /biz_photos/*?select_video=
77Allow: /biz_photos/video_url
78
79User-Agent: LinkedInBot
80User-Agent: Twitterbot
81User-Agent: facebookexternalhit
82Allow: /article/
83
84User-Agent: AdsBot-Google
85User-Agent: BingPreview
86User-Agent: Google-InspectionTool
87User-Agent: Googlebot
88User-Agent: Googlebot-Image
89User-Agent: Googlebot-Mobile
90User-Agent: Googlebot-Video
91User-Agent: LinkedInBot
92User-Agent: Mediapartners-Google
93User-Agent: STC-bot
94User-Agent: Twitterbot
95User-Agent: Yahoo! Slurp
96User-Agent: bingbot
97User-Agent: facebookexternalhit
98Disallow: *US_CENSUS_NAME*
99Disallow: *US_FEMALE_NAME*
100Disallow: *US_MALE_NAME*
101Disallow: /1014943
102Disallow: /3584794
103Disallow: /5787254
104Disallow: /ad_acknowledgment
105Disallow: /ad_spice
106Disallow: /ad_syndication_user_tracking
107Disallow: /ad_visibility
108Disallow: /adredir?
109Disallow: /adtrack
110Disallow: /advertise?
111Disallow: /answers/*/delete
112Disallow: /answers/*/edit
113Disallow: /answers/*/vote
114Disallow: /assistant
115Disallow: /biz/*?*&translate=1*
116Disallow: /biz/*?*&translate=True*
117Disallow: /biz/*?translate=1*
118Disallow: /biz/*?translate=True*
119Disallow: /biz/*destination=*
120Disallow: /biz/*entry_point=*
121Disallow: /biz/impression/*
122Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-0
123Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-1
124Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-2
125Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-3
126Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-4
127Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-5
128Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-6
129Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-7
130Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-8
131Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-9
132Disallow: /biz/taste-of-lurgan-lost-springs
133Disallow: /biz_attribute
134Disallow: /biz_link
135Disallow: /biz_photos
136Disallow: /biz_photos/*/log
137Disallow: /biz_photos/*/log_views
138Disallow: /biz_photos/feedback/
139Disallow: /biz_redir
140Disallow: /biz_share
141Disallow: /biz_update
142Disallow: /bn/
143Disallow: /bookmark?
144Disallow: /browse/reviews/recent
145Disallow: /client_errors
146Disallow: /collections/following
147Disallow: /collections/user
148Disallow: /cookie_sync
149Disallow: /csp_block
150Disallow: /csp_report_only
151Disallow: /elite?
152Disallow: /email_actions
153Disallow: /events/*/browse
154Disallow: /events/*/feed
155Disallow: /events/addphoto/
156Disallow: /events/cancel/
157Disallow: /events/cancel_rsvp/
158Disallow: /events/change_guest/
159Disallow: /events/create
160Disallow: /events/create/
161Disallow: /events/deletephoto/
162Disallow: /events/details/
163Disallow: /events/elite/
164Disallow: /events/email_notifications/
165Disallow: /events/event_talk_subscribe
166Disallow: /events/event_talk_unsubscribe
167Disallow: /events/export/
168Disallow: /events/get_media_slice/
169Disallow: /events/interested/
170Disallow: /events/load_more_messages/
171Disallow: /events/photos/
172Disallow: /events/record_rsvp/
173Disallow: /events/share/
174Disallow: /events/similar_snippet
175Disallow: /events/subscriber_snippet
176Disallow: /events/talk/
177Disallow: /events/uncancel/
178Disallow: /events/update
179Disallow: /events/update/
180Disallow: /events/update_primary_photo
181Disallow: /events/update_subscription
182Disallow: /events/updatecaption
183Disallow: /facebook_connect
184Disallow: /flag_content/submit
185Disallow: /flag_content?
186Disallow: /gamtarget
187Disallow: /location_suggest
188Disallow: /mail?
189Disallow: /menu/*/feedback_form
190Disallow: /message_the_business
191Disallow: /mss/review_feedback/
192Disallow: /mtb_composer
193Disallow: /not_recommended_reviews
194Disallow: /opportunity
195Disallow: /photo/
196Disallow: /possible_biz_owner
197Disallow: /proredir?
198Disallow: /px.gif
199Disallow: /questions/*/create_answer/
200Disallow: /questions/*/delete
201Disallow: /questions/*/edit
202Disallow: /questions/*/subscribe
203Disallow: /redir?
204Disallow: /reservations/*/notifyme
205Disallow: /review_feed_auto_fetch
206Disallow: /search_suggest
207Disallow: /send_to_friend
208Disallow: /sit_rep
209Disallow: /spice
210Disallow: /start_order
211Disallow: /syndicate
212Disallow: /syndication_cookie_sync
213Disallow: /talk/new_topic
214Disallow: /thanx?
215Disallow: /transaction_platform/start_order
216Disallow: /user_details
217Disallow: /user_details_answers_given
218Disallow: /user_details_bookmarks
219Disallow: /user_details_friends
220Disallow: /user_details_reviews_self
221Disallow: /user_details_thanx
222Disallow: /user_photos
223Disallow: /weekly/signup
224Disallow: /writeareview/
225Disallow: /yuv
226
227User-Agent: Google-InspectionTool
228User-Agent: Googlebot
229User-Agent: Googlebot-Image
230User-Agent: Googlebot-Mobile
231User-Agent: Googlebot-Video
232Disallow: /biz_photos/*-albuquerque*/log
233Disallow: /biz_photos/*-albuquerque*/log_views
234Disallow: /biz_photos/*-anaheim*/log
235Disallow: /biz_photos/*-anaheim*/log_views
236Disallow: /biz_photos/*-atlanta*/log
237Disallow: /biz_photos/*-atlanta*/log_views
238Disallow: /biz_photos/*-austin*/log
239Disallow: /biz_photos/*-austin*/log_views
240Disallow: /biz_photos/*-baltimore*/log
241Disallow: /biz_photos/*-baltimore*/log_views
242Disallow: /biz_photos/*-boston*/log
243Disallow: /biz_photos/*-boston*/log_views
244Disallow: /biz_photos/*-brooklyn*/log
245Disallow: /biz_photos/*-brooklyn*/log_views
246Disallow: /biz_photos/*-charlotte*/log
247Disallow: /biz_photos/*-charlotte*/log_views
248Disallow: /biz_photos/*-chicago*/log
249Disallow: /biz_photos/*-chicago*/log_views
250Disallow: /biz_photos/*-cincinnati*/log
251Disallow: /biz_photos/*-cincinnati*/log_views
252Disallow: /biz_photos/*-columbus*/log
253Disallow: /biz_photos/*-columbus*/log_views
254Disallow: /biz_photos/*-dallas*/log
255Disallow: /biz_photos/*-dallas*/log_views
256Disallow: /biz_photos/*-denver*/log
257Disallow: /biz_photos/*-denver*/log_views
258Disallow: /biz_photos/*-fort-worth*/log
259Disallow: /biz_photos/*-fort-worth*/log_views
260Disallow: /biz_photos/*-fresno*/log
261Disallow: /biz_photos/*-fresno*/log_views
262Disallow: /biz_photos/*-honolulu*/log
263Disallow: /biz_photos/*-honolulu*/log_views
264Disallow: /biz_photos/*-houston*/log
265Disallow: /biz_photos/*-houston*/log_views
266Disallow: /biz_photos/*-indianapolis*/log
267Disallow: /biz_photos/*-indianapolis*/log_views
268Disallow: /biz_photos/*-irvine*/log
269Disallow: /biz_photos/*-irvine*/log_views
270Disallow: /biz_photos/*-jacksonville*/log
271Disallow: /biz_photos/*-jacksonville*/log_views
272Disallow: /biz_photos/*-las-vegas*/log
273Disallow: /biz_photos/*-las-vegas*/log_views
274Disallow: /biz_photos/*-long-beach*/log
275Disallow: /biz_photos/*-long-beach*/log_views
276Disallow: /biz_photos/*-los-angeles*/log
277Disallow: /biz_photos/*-los-angeles*/log_views
278Disallow: /biz_photos/*-louisville*/log
279Disallow: /biz_photos/*-louisville*/log_views
280Disallow: /biz_photos/*-manhattan*/log
281Disallow: /biz_photos/*-manhattan*/log_views
282Disallow: /biz_photos/*-miami*/log
283Disallow: /biz_photos/*-miami*/log_views
284Disallow: /biz_photos/*-minneapolis*/log
285Disallow: /biz_photos/*-minneapolis*/log_views
286Disallow: /biz_photos/*-nashville*/log
287Disallow: /biz_photos/*-nashville*/log_views
288Disallow: /biz_photos/*-new-orleans*/log
289Disallow: /biz_photos/*-new-orleans*/log_views
290Disallow: /biz_photos/*-new-york*/log
291Disallow: /biz_photos/*-new-york*/log_views
292Disallow: /biz_photos/*-oakland*/log
293Disallow: /biz_photos/*-oakland*/log_views
294Disallow: /biz_photos/*-oklahoma-city*/log
295Disallow: /biz_photos/*-oklahoma-city*/log_views
296Disallow: /biz_photos/*-orlando*/log
297Disallow: /biz_photos/*-orlando*/log_views
298Disallow: /biz_photos/*-philadelphia*/log
299Disallow: /biz_photos/*-philadelphia*/log_views
300Disallow: /biz_photos/*-phoenix*/log
301Disallow: /biz_photos/*-phoenix*/log_views
302Disallow: /biz_photos/*-pittsburgh*/log
303Disallow: /biz_photos/*-pittsburgh*/log_views
304Disallow: /biz_photos/*-portland*/log
305Disallow: /biz_photos/*-portland*/log_views
306Disallow: /biz_photos/*-raleigh*/log
307Disallow: /biz_photos/*-raleigh*/log_views
308Disallow: /biz_photos/*-richmond*/log
309Disallow: /biz_photos/*-richmond*/log_views
310Disallow: /biz_photos/*-sacramento*/log
311Disallow: /biz_photos/*-sacramento*/log_views
312Disallow: /biz_photos/*-san-antonio*/log
313Disallow: /biz_photos/*-san-antonio*/log_views
314Disallow: /biz_photos/*-san-diego*/log
315Disallow: /biz_photos/*-san-diego*/log_views
316Disallow: /biz_photos/*-san-francisco*/log
317Disallow: /biz_photos/*-san-francisco*/log_views
318Disallow: /biz_photos/*-san-jose*/log
319Disallow: /biz_photos/*-san-jose*/log_views
320Disallow: /biz_photos/*-scottsdale*/log
321Disallow: /biz_photos/*-scottsdale*/log_views
322Disallow: /biz_photos/*-seattle*/log
323Disallow: /biz_photos/*-seattle*/log_views
324Disallow: /biz_photos/*-tampa*/log
325Disallow: /biz_photos/*-tampa*/log_views
326Disallow: /biz_photos/*-tampa-bay*/log
327Disallow: /biz_photos/*-tampa-bay*/log_views
328Disallow: /biz_photos/*-tucson*/log
329Disallow: /biz_photos/*-tucson*/log_views
330Disallow: /biz_photos/*-washington*/log
331Disallow: /biz_photos/*-washington*/log_views
332
333User-Agent: Yahoo! Slurp
334User-Agent: bingbot
335Disallow: /search*start=
336
337User-Agent: OAI-SearchBot
338Allow: /spk/
339Allow: /article$
340Allow: /article/
341Disallow: /
342
343User-Agent: ChatGPT-User
344User-Agent: Claude-User
345User-Agent: PerplexityUser
346User-Agent: Perplexity-User
347User-Agent: Claude-SearchBot
348User-Agent: PerplexityBot
349Allow: /article$
350Allow: /article/
351Disallow: /
352
353User-Agent: GPTBot
354User-Agent: ClaudeBot
355User-Agent: CCBot
356User-Agent: Google-Extended
357User-Agent: anthropic-ai
358User-Agent: Claude-Web
359User-Agent: Meta-ExternalAgent
360User-Agent: Meta-ExternalFetcher
361User-Agent: RuxitSynthetic
362Disallow: /
363
364User-Agent: *
365Disallow: /
366
What do the bots actually get?
This page reads robots.txt, which states what yelp.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on yelp.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does yelp.com block GPTBot?
Yes. yelp.com’s robots.txt disallows GPTBot from the whole site (line 362: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does yelp.com block ClaudeBot?
Yes. yelp.com’s robots.txt disallows ClaudeBot from the whole site (line 362: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does yelp.com block Google-Extended?
Yes. yelp.com’s robots.txt disallows Google-Extended from the whole site (line 362: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does yelp.com block PerplexityBot?
Yes. yelp.com’s robots.txt disallows PerplexityBot from the whole site (line 351: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find yelp.com?
Not through its own crawler. yelp.com’s robots.txt disallows OAI-SearchBot from the whole site (line 341: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.