Who blocks AI · #153 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does trustpilot.com block AI crawlers?
Yes. trustpilot.com’s robots.txt blocks 8 of the 19 AI crawlers we track, including Applebot-Extended, Bytespider, DuckAssistBot and 5 more. It also blocks PetalBot and YandexBot among search engines.
GPTBotPartly blockedOAI-SearchBotPartly blockedClaudeBotPartly blockedPerplexityBotPartly blockedGoogle-ExtendedPartly blockedCCBotPartly blocked- robots.txt
- www.trustpilot.com/robots.txt · 1,067 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 2 · Disallow: /error line 3 · Disallow: /evaluate/ and 31 more | |
Bingbot | Microsoft | Partly blocked | line 182 · Disallow: /error line 183 · Disallow: /evaluate/ and 31 more |
DuckDuckBot | DuckDuckGo | Partly blocked | line 542 · Disallow: /error line 543 · Disallow: /evaluate/ and 31 more |
Applebot | Apple | Partly blocked | line 2 · Disallow: /error line 3 · Disallow: /evaluate/ and 31 more |
PetalBot | Huawei | Blocked | line 1061 · Disallow: / under User-agent: * |
YandexBot | Yandex | Blocked | line 1061 · Disallow: / under User-agent: * |
Baiduspider | Baidu | Partly blocked | line 434 · Disallow: /error line 435 · Disallow: /evaluate/ and 31 more |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Partly blocked | line 614 · Disallow: /error line 615 · Disallow: /evaluate/ and 31 more |
PerplexityBot | Perplexity | Partly blocked | line 650 · Disallow: /error line 651 · Disallow: /evaluate/ and 31 more |
Claude-SearchBot | Anthropic | Partly blocked | line 722 · Disallow: /error line 723 · Disallow: /evaluate/ and 31 more |
DuckAssistBot | DuckDuckGo | Blocked | line 1061 · Disallow: / under User-agent: * |
Amazonbot | Amazon | Blocked | line 1061 · Disallow: / under User-agent: * |
YouBot | You.com | Blocked | line 1058 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Partly blocked | line 758 · Disallow: /error line 759 · Disallow: /evaluate/ and 31 more |
ClaudeBot | Anthropic | Partly blocked | line 830 · Disallow: /error line 831 · Disallow: /evaluate/ and 31 more |
Google-Extended | Partly blocked | line 794 · Disallow: /error line 795 · Disallow: /evaluate/ and 31 more | |
CCBot | Common Crawl | Partly blocked | line 866 · Disallow: /error line 867 · Disallow: /evaluate/ and 31 more |
Applebot-Extended | Apple | Blocked | line 1025 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Blocked | line 1061 · Disallow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Partly blocked | line 974 · Disallow: /error line 975 · Disallow: /evaluate/ and 31 more |
Bytespider | ByteDance | Blocked | line 1040 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Partly blocked | line 578 · Disallow: /error line 579 · Disallow: /evaluate/ and 31 more |
Claude-User | Anthropic | Partly blocked | line 686 · Disallow: /error line 687 · Disallow: /evaluate/ and 31 more |
Perplexity-User | Perplexity | Blocked | line 1061 · Disallow: / under User-agent: * |
MistralAI-User | Mistral | Blocked | line 1061 · Disallow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Partly blocked | line 938 · Disallow: /error line 939 · Disallow: /evaluate/ and 31 more |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 1061 · Disallow: / under User-agent: * |
SemrushBot | Semrush | Blocked | line 1061 · Disallow: / under User-agent: * |
MJ12bot | Majestic | Blocked | line 1061 · Disallow: / under User-agent: * |
DotBot | Moz | Blocked | line 1061 · Disallow: / under User-agent: * |
View www.trustpilot.com/robots.txt · 1,067 lines
1User-agent: Googlebot
2Disallow: /error
3Disallow: /evaluate/
4Disallow: /evaluate-link
5Disallow: /evaluate-unique-link
6Disallow: /pingdom
7Disallow: /reviews/
8Disallow: /review/*/transparency
9Disallow: /product-reviews/
10Disallow: /*?*editmode=
11Disallow: /api/*
12Disallow: /*?*languages=
13Disallow: /*?*stars=
14Disallow: /*?*sort=
15Disallow: /*?*verified=
16Disallow: /*?*topics=
17Disallow: /*?*replies=
18Disallow: /*?*date=
19Disallow: /*?*trustscore=
20Disallow: /*?*claimed=
21Disallow: /*?*country=
22Disallow: /*?*location=
23Disallow: /*?*subcategories=
24Disallow: /search
25Disallow: /sessions
26Disallow: /submitted/review
27Disallow: /users/
28Allow: /users/connect
29Disallow: /users/connect?redirect
30Disallow: /users/connect?signup
31Disallow: /users/connect?intent
32Disallow: /users/activate
33Disallow: /users/embed-login-to-edit
34Disallow: /users/create
35Disallow: /unsubscribe/invitation
36
37User-agent: Mediapartners-Google
38Disallow: /error
39Disallow: /evaluate/
40Disallow: /evaluate-link
41Disallow: /evaluate-unique-link
42Disallow: /pingdom
43Disallow: /reviews/
44Disallow: /review/*/transparency
45Disallow: /product-reviews/
46Disallow: /*?*editmode=
47Disallow: /api/*
48Disallow: /*?*languages=
49Disallow: /*?*stars=
50Disallow: /*?*sort=
51Disallow: /*?*verified=
52Disallow: /*?*topics=
53Disallow: /*?*replies=
54Disallow: /*?*date=
55Disallow: /*?*trustscore=
56Disallow: /*?*claimed=
57Disallow: /*?*country=
58Disallow: /*?*location=
59Disallow: /*?*subcategories=
60Disallow: /search
61Disallow: /sessions
62Disallow: /submitted/review
63Disallow: /users/
64Allow: /users/connect
65Disallow: /users/connect?redirect
66Disallow: /users/connect?signup
67Disallow: /users/connect?intent
68Disallow: /users/activate
69Disallow: /users/embed-login-to-edit
70Disallow: /users/create
71Disallow: /unsubscribe/invitation
72
73User-agent: Googlebot-Mobile
74Disallow: /error
75Disallow: /evaluate/
76Disallow: /evaluate-link
77Disallow: /evaluate-unique-link
78Disallow: /pingdom
79Disallow: /reviews/
80Disallow: /review/*/transparency
81Disallow: /product-reviews/
82Disallow: /*?*editmode=
83Disallow: /api/*
84Disallow: /*?*languages=
85Disallow: /*?*stars=
86Disallow: /*?*sort=
87Disallow: /*?*verified=
88Disallow: /*?*topics=
89Disallow: /*?*replies=
90Disallow: /*?*date=
91Disallow: /*?*trustscore=
92Disallow: /*?*claimed=
93Disallow: /*?*country=
94Disallow: /*?*location=
95Disallow: /*?*subcategories=
96Disallow: /search
97Disallow: /sessions
98Disallow: /submitted/review
99Disallow: /users/
100Allow: /users/connect
101Disallow: /users/connect?redirect
102Disallow: /users/connect?signup
103Disallow: /users/connect?intent
104Disallow: /users/activate
105Disallow: /users/embed-login-to-edit
106Disallow: /users/create
107Disallow: /unsubscribe/invitation
108
109User-agent: AdsBot-Google
110Disallow: /error
111Disallow: /evaluate/
112Disallow: /evaluate-link
113Disallow: /evaluate-unique-link
114Disallow: /pingdom
115Disallow: /reviews/
116Disallow: /review/*/transparency
117Disallow: /product-reviews/
118Disallow: /*?*editmode=
119Disallow: /api/*
120Disallow: /*?*languages=
121Disallow: /*?*stars=
122Disallow: /*?*sort=
123Disallow: /*?*verified=
124Disallow: /*?*topics=
125Disallow: /*?*replies=
126Disallow: /*?*date=
127Disallow: /*?*trustscore=
128Disallow: /*?*claimed=
129Disallow: /*?*country=
130Disallow: /*?*location=
131Disallow: /*?*subcategories=
132Disallow: /search
133Disallow: /sessions
134Disallow: /submitted/review
135Disallow: /users/
136Allow: /users/connect
137Disallow: /users/connect?redirect
138Disallow: /users/connect?signup
139Disallow: /users/connect?intent
140Disallow: /users/activate
141Disallow: /users/embed-login-to-edit
142Disallow: /users/create
143Disallow: /unsubscribe/invitation
144
145User-agent: AdsBot-Google-Mobile
146Disallow: /error
147Disallow: /evaluate/
148Disallow: /evaluate-link
149Disallow: /evaluate-unique-link
150Disallow: /pingdom
151Disallow: /reviews/
152Disallow: /review/*/transparency
153Disallow: /product-reviews/
154Disallow: /*?*editmode=
155Disallow: /api/*
156Disallow: /*?*languages=
157Disallow: /*?*stars=
158Disallow: /*?*sort=
159Disallow: /*?*verified=
160Disallow: /*?*topics=
161Disallow: /*?*replies=
162Disallow: /*?*date=
163Disallow: /*?*trustscore=
164Disallow: /*?*claimed=
165Disallow: /*?*country=
166Disallow: /*?*location=
167Disallow: /*?*subcategories=
168Disallow: /search
169Disallow: /sessions
170Disallow: /submitted/review
171Disallow: /users/
172Allow: /users/connect
173Disallow: /users/connect?redirect
174Disallow: /users/connect?signup
175Disallow: /users/connect?intent
176Disallow: /users/activate
177Disallow: /users/embed-login-to-edit
178Disallow: /users/create
179Disallow: /unsubscribe/invitation
180
181User-agent: bingbot
182Disallow: /error
183Disallow: /evaluate/
184Disallow: /evaluate-link
185Disallow: /evaluate-unique-link
186Disallow: /pingdom
187Disallow: /reviews/
188Disallow: /review/*/transparency
189Disallow: /product-reviews/
190Disallow: /*?*editmode=
191Disallow: /api/*
192Disallow: /*?*languages=
193Disallow: /*?*stars=
194Disallow: /*?*sort=
195Disallow: /*?*verified=
196Disallow: /*?*topics=
197Disallow: /*?*replies=
198Disallow: /*?*date=
199Disallow: /*?*trustscore=
200Disallow: /*?*claimed=
201Disallow: /*?*country=
202Disallow: /*?*location=
203Disallow: /*?*subcategories=
204Disallow: /search
205Disallow: /sessions
206Disallow: /submitted/review
207Disallow: /users/
208Allow: /users/connect
209Disallow: /users/connect?redirect
210Disallow: /users/connect?signup
211Disallow: /users/connect?intent
212Disallow: /users/activate
213Disallow: /users/embed-login-to-edit
214Disallow: /users/create
215Disallow: /unsubscribe/invitation
216
217User-agent: msnbot
218Disallow: /error
219Disallow: /evaluate/
220Disallow: /evaluate-link
221Disallow: /evaluate-unique-link
222Disallow: /pingdom
223Disallow: /reviews/
224Disallow: /review/*/transparency
225Disallow: /product-reviews/
226Disallow: /*?*editmode=
227Disallow: /api/*
228Disallow: /*?*languages=
229Disallow: /*?*stars=
230Disallow: /*?*sort=
231Disallow: /*?*verified=
232Disallow: /*?*topics=
233Disallow: /*?*replies=
234Disallow: /*?*date=
235Disallow: /*?*trustscore=
236Disallow: /*?*claimed=
237Disallow: /*?*country=
238Disallow: /*?*location=
239Disallow: /*?*subcategories=
240Disallow: /search
241Disallow: /sessions
242Disallow: /submitted/review
243Disallow: /users/
244Allow: /users/connect
245Disallow: /users/connect?redirect
246Disallow: /users/connect?signup
247Disallow: /users/connect?intent
248Disallow: /users/activate
249Disallow: /users/embed-login-to-edit
250Disallow: /users/create
251Disallow: /unsubscribe/invitation
252
253User-agent: ia_archiver
254Disallow: /error
255Disallow: /evaluate/
256Disallow: /evaluate-link
257Disallow: /evaluate-unique-link
258Disallow: /pingdom
259Disallow: /reviews/
260Disallow: /review/*/transparency
261Disallow: /product-reviews/
262Disallow: /*?*editmode=
263Disallow: /api/*
264Disallow: /*?*languages=
265Disallow: /*?*stars=
266Disallow: /*?*sort=
267Disallow: /*?*verified=
268Disallow: /*?*topics=
269Disallow: /*?*replies=
270Disallow: /*?*date=
271Disallow: /*?*trustscore=
272Disallow: /*?*claimed=
273Disallow: /*?*country=
274Disallow: /*?*location=
275Disallow: /*?*subcategories=
276Disallow: /search
277Disallow: /sessions
278Disallow: /submitted/review
279Disallow: /users/
280Allow: /users/connect
281Disallow: /users/connect?redirect
282Disallow: /users/connect?signup
283Disallow: /users/connect?intent
284Disallow: /users/activate
285Disallow: /users/embed-login-to-edit
286Disallow: /users/create
287Disallow: /unsubscribe/invitation
288
289User-agent: Alexabot
290Disallow: /error
291Disallow: /evaluate/
292Disallow: /evaluate-link
293Disallow: /evaluate-unique-link
294Disallow: /pingdom
295Disallow: /reviews/
296Disallow: /review/*/transparency
297Disallow: /product-reviews/
298Disallow: /*?*editmode=
299Disallow: /api/*
300Disallow: /*?*languages=
301Disallow: /*?*stars=
302Disallow: /*?*sort=
303Disallow: /*?*verified=
304Disallow: /*?*topics=
305Disallow: /*?*replies=
306Disallow: /*?*date=
307Disallow: /*?*trustscore=
308Disallow: /*?*claimed=
309Disallow: /*?*country=
310Disallow: /*?*location=
311Disallow: /*?*subcategories=
312Disallow: /search
313Disallow: /sessions
314Disallow: /submitted/review
315Disallow: /users/
316Allow: /users/connect
317Disallow: /users/connect?redirect
318Disallow: /users/connect?signup
319Disallow: /users/connect?intent
320Disallow: /users/activate
321Disallow: /users/embed-login-to-edit
322Disallow: /users/create
323Disallow: /unsubscribe/invitation
324
325User-agent: ScoutJet
326Disallow: /error
327Disallow: /evaluate/
328Disallow: /evaluate-link
329Disallow: /evaluate-unique-link
330Disallow: /pingdom
331Disallow: /reviews/
332Disallow: /review/*/transparency
333Disallow: /product-reviews/
334Disallow: /*?*editmode=
335Disallow: /api/*
336Disallow: /*?*languages=
337Disallow: /*?*stars=
338Disallow: /*?*sort=
339Disallow: /*?*verified=
340Disallow: /*?*topics=
341Disallow: /*?*replies=
342Disallow: /*?*date=
343Disallow: /*?*trustscore=
344Disallow: /*?*claimed=
345Disallow: /*?*country=
346Disallow: /*?*location=
347Disallow: /*?*subcategories=
348Disallow: /search
349Disallow: /sessions
350Disallow: /submitted/review
351Disallow: /users/
352Allow: /users/connect
353Disallow: /users/connect?redirect
354Disallow: /users/connect?signup
355Disallow: /users/connect?intent
356Disallow: /users/activate
357Disallow: /users/embed-login-to-edit
358Disallow: /users/create
359Disallow: /unsubscribe/invitation
360
361User-agent: Slurp
362Disallow: /error
363Disallow: /evaluate/
364Disallow: /evaluate-link
365Disallow: /evaluate-unique-link
366Disallow: /pingdom
367Disallow: /reviews/
368Disallow: /review/*/transparency
369Disallow: /product-reviews/
370Disallow: /*?*editmode=
371Disallow: /api/*
372Disallow: /*?*languages=
373Disallow: /*?*stars=
374Disallow: /*?*sort=
375Disallow: /*?*verified=
376Disallow: /*?*topics=
377Disallow: /*?*replies=
378Disallow: /*?*date=
379Disallow: /*?*trustscore=
380Disallow: /*?*claimed=
381Disallow: /*?*country=
382Disallow: /*?*location=
383Disallow: /*?*subcategories=
384Disallow: /search
385Disallow: /sessions
386Disallow: /submitted/review
387Disallow: /users/
388Allow: /users/connect
389Disallow: /users/connect?redirect
390Disallow: /users/connect?signup
391Disallow: /users/connect?intent
392Disallow: /users/activate
393Disallow: /users/embed-login-to-edit
394Disallow: /users/create
395Disallow: /unsubscribe/invitation
396
397User-agent: Twitterbot
398Disallow: /error
399Disallow: /evaluate/
400Disallow: /evaluate-link
401Disallow: /evaluate-unique-link
402Disallow: /pingdom
403Disallow: /reviews/
404Disallow: /review/*/transparency
405Disallow: /product-reviews/
406Disallow: /*?*editmode=
407Disallow: /api/*
408Disallow: /*?*languages=
409Disallow: /*?*stars=
410Disallow: /*?*sort=
411Disallow: /*?*verified=
412Disallow: /*?*topics=
413Disallow: /*?*replies=
414Disallow: /*?*date=
415Disallow: /*?*trustscore=
416Disallow: /*?*claimed=
417Disallow: /*?*country=
418Disallow: /*?*location=
419Disallow: /*?*subcategories=
420Disallow: /search
421Disallow: /sessions
422Disallow: /submitted/review
423Disallow: /users/
424Allow: /users/connect
425Disallow: /users/connect?redirect
426Disallow: /users/connect?signup
427Disallow: /users/connect?intent
428Disallow: /users/activate
429Disallow: /users/embed-login-to-edit
430Disallow: /users/create
431Disallow: /unsubscribe/invitation
432
433User-agent: Baiduspider
434Disallow: /error
435Disallow: /evaluate/
436Disallow: /evaluate-link
437Disallow: /evaluate-unique-link
438Disallow: /pingdom
439Disallow: /reviews/
440Disallow: /review/*/transparency
441Disallow: /product-reviews/
442Disallow: /*?*editmode=
443Disallow: /api/*
444Disallow: /*?*languages=
445Disallow: /*?*stars=
446Disallow: /*?*sort=
447Disallow: /*?*verified=
448Disallow: /*?*topics=
449Disallow: /*?*replies=
450Disallow: /*?*date=
451Disallow: /*?*trustscore=
452Disallow: /*?*claimed=
453Disallow: /*?*country=
454Disallow: /*?*location=
455Disallow: /*?*subcategories=
456Disallow: /search
457Disallow: /sessions
458Disallow: /submitted/review
459Disallow: /users/
460Allow: /users/connect
461Disallow: /users/connect?redirect
462Disallow: /users/connect?signup
463Disallow: /users/connect?intent
464Disallow: /users/activate
465Disallow: /users/embed-login-to-edit
466Disallow: /users/create
467Disallow: /unsubscribe/invitation
468
469User-agent: FacebookBot
470Disallow: /error
471Disallow: /evaluate/
472Disallow: /evaluate-link
473Disallow: /evaluate-unique-link
474Disallow: /pingdom
475Allow: /reviews/
476Disallow: /review/*/transparency
477Disallow: /product-reviews/
478Disallow: /*?*editmode=
479Disallow: /api/*
480Disallow: /*?*languages=
481Disallow: /*?*stars=
482Disallow: /*?*sort=
483Disallow: /*?*verified=
484Disallow: /*?*topics=
485Disallow: /*?*replies=
486Disallow: /*?*date=
487Disallow: /*?*trustscore=
488Disallow: /*?*claimed=
489Disallow: /*?*country=
490Disallow: /*?*location=
491Disallow: /*?*subcategories=
492Disallow: /search
493Disallow: /sessions
494Disallow: /submitted/review
495Disallow: /users/
496Allow: /users/connect
497Disallow: /users/connect?redirect
498Disallow: /users/connect?signup
499Disallow: /users/connect?intent
500Disallow: /users/activate
501Disallow: /users/embed-login-to-edit
502Disallow: /users/create
503Disallow: /unsubscribe/invitation
504
505User-agent: facebookexternalhit
506Disallow: /error
507Disallow: /evaluate/
508Disallow: /evaluate-link
509Disallow: /evaluate-unique-link
510Disallow: /pingdom
511Allow: /reviews/
512Disallow: /review/*/transparency
513Disallow: /product-reviews/
514Disallow: /*?*editmode=
515Disallow: /api/*
516Disallow: /*?*languages=
517Disallow: /*?*stars=
518Disallow: /*?*sort=
519Disallow: /*?*verified=
520Disallow: /*?*topics=
521Disallow: /*?*replies=
522Disallow: /*?*date=
523Disallow: /*?*trustscore=
524Disallow: /*?*claimed=
525Disallow: /*?*country=
526Disallow: /*?*location=
527Disallow: /*?*subcategories=
528Disallow: /search
529Disallow: /sessions
530Disallow: /submitted/review
531Disallow: /users/
532Allow: /users/connect
533Disallow: /users/connect?redirect
534Disallow: /users/connect?signup
535Disallow: /users/connect?intent
536Disallow: /users/activate
537Disallow: /users/embed-login-to-edit
538Disallow: /users/create
539Disallow: /unsubscribe/invitation
540
541User-agent: DuckDuckBot
542Disallow: /error
543Disallow: /evaluate/
544Disallow: /evaluate-link
545Disallow: /evaluate-unique-link
546Disallow: /pingdom
547Disallow: /reviews/
548Disallow: /review/*/transparency
549Disallow: /product-reviews/
550Disallow: /*?*editmode=
551Disallow: /api/*
552Disallow: /*?*languages=
553Disallow: /*?*stars=
554Disallow: /*?*sort=
555Disallow: /*?*verified=
556Disallow: /*?*topics=
557Disallow: /*?*replies=
558Disallow: /*?*date=
559Disallow: /*?*trustscore=
560Disallow: /*?*claimed=
561Disallow: /*?*country=
562Disallow: /*?*location=
563Disallow: /*?*subcategories=
564Disallow: /search
565Disallow: /sessions
566Disallow: /submitted/review
567Disallow: /users/
568Allow: /users/connect
569Disallow: /users/connect?redirect
570Disallow: /users/connect?signup
571Disallow: /users/connect?intent
572Disallow: /users/activate
573Disallow: /users/embed-login-to-edit
574Disallow: /users/create
575Disallow: /unsubscribe/invitation
576
577User-agent: ChatGPT-User
578Disallow: /error
579Disallow: /evaluate/
580Disallow: /evaluate-link
581Disallow: /evaluate-unique-link
582Disallow: /pingdom
583Disallow: /reviews/
584Disallow: /review/*/transparency
585Disallow: /product-reviews/
586Disallow: /*?*editmode=
587Disallow: /api/*
588Disallow: /*?*languages=
589Disallow: /*?*stars=
590Disallow: /*?*sort=
591Disallow: /*?*verified=
592Disallow: /*?*topics=
593Disallow: /*?*replies=
594Disallow: /*?*date=
595Disallow: /*?*trustscore=
596Disallow: /*?*claimed=
597Disallow: /*?*country=
598Disallow: /*?*location=
599Disallow: /*?*subcategories=
600Disallow: /search
601Disallow: /sessions
602Disallow: /submitted/review
603Disallow: /users/
604Allow: /users/connect
605Disallow: /users/connect?redirect
606Disallow: /users/connect?signup
607Disallow: /users/connect?intent
608Disallow: /users/activate
609Disallow: /users/embed-login-to-edit
610Disallow: /users/create
611Disallow: /unsubscribe/invitation
612
613User-agent: OAI-SearchBot
614Disallow: /error
615Disallow: /evaluate/
616Disallow: /evaluate-link
617Disallow: /evaluate-unique-link
618Disallow: /pingdom
619Disallow: /reviews/
620Disallow: /review/*/transparency
621Disallow: /product-reviews/
622Disallow: /*?*editmode=
623Disallow: /api/*
624Disallow: /*?*languages=
625Disallow: /*?*stars=
626Disallow: /*?*sort=
627Disallow: /*?*verified=
628Disallow: /*?*topics=
629Disallow: /*?*replies=
630Disallow: /*?*date=
631Disallow: /*?*trustscore=
632Disallow: /*?*claimed=
633Disallow: /*?*country=
634Disallow: /*?*location=
635Disallow: /*?*subcategories=
636Disallow: /search
637Disallow: /sessions
638Disallow: /submitted/review
639Disallow: /users/
640Allow: /users/connect
641Disallow: /users/connect?redirect
642Disallow: /users/connect?signup
643Disallow: /users/connect?intent
644Disallow: /users/activate
645Disallow: /users/embed-login-to-edit
646Disallow: /users/create
647Disallow: /unsubscribe/invitation
648
649User-agent: PerplexityBot
650Disallow: /error
651Disallow: /evaluate/
652Disallow: /evaluate-link
653Disallow: /evaluate-unique-link
654Disallow: /pingdom
655Disallow: /reviews/
656Disallow: /review/*/transparency
657Disallow: /product-reviews/
658Disallow: /*?*editmode=
659Disallow: /api/*
660Disallow: /*?*languages=
661Disallow: /*?*stars=
662Disallow: /*?*sort=
663Disallow: /*?*verified=
664Disallow: /*?*topics=
665Disallow: /*?*replies=
666Disallow: /*?*date=
667Disallow: /*?*trustscore=
668Disallow: /*?*claimed=
669Disallow: /*?*country=
670Disallow: /*?*location=
671Disallow: /*?*subcategories=
672Disallow: /search
673Disallow: /sessions
674Disallow: /submitted/review
675Disallow: /users/
676Allow: /users/connect
677Disallow: /users/connect?redirect
678Disallow: /users/connect?signup
679Disallow: /users/connect?intent
680Disallow: /users/activate
681Disallow: /users/embed-login-to-edit
682Disallow: /users/create
683Disallow: /unsubscribe/invitation
684
685User-agent: Claude-User
686Disallow: /error
687Disallow: /evaluate/
688Disallow: /evaluate-link
689Disallow: /evaluate-unique-link
690Disallow: /pingdom
691Disallow: /reviews/
692Disallow: /review/*/transparency
693Disallow: /product-reviews/
694Disallow: /*?*editmode=
695Disallow: /api/*
696Disallow: /*?*languages=
697Disallow: /*?*stars=
698Disallow: /*?*sort=
699Disallow: /*?*verified=
700Disallow: /*?*topics=
701Disallow: /*?*replies=
702Disallow: /*?*date=
703Disallow: /*?*trustscore=
704Disallow: /*?*claimed=
705Disallow: /*?*country=
706Disallow: /*?*location=
707Disallow: /*?*subcategories=
708Disallow: /search
709Disallow: /sessions
710Disallow: /submitted/review
711Disallow: /users/
712Allow: /users/connect
713Disallow: /users/connect?redirect
714Disallow: /users/connect?signup
715Disallow: /users/connect?intent
716Disallow: /users/activate
717Disallow: /users/embed-login-to-edit
718Disallow: /users/create
719Disallow: /unsubscribe/invitation
720
721User-agent: Claude-SearchBot
722Disallow: /error
723Disallow: /evaluate/
724Disallow: /evaluate-link
725Disallow: /evaluate-unique-link
726Disallow: /pingdom
727Disallow: /reviews/
728Disallow: /review/*/transparency
729Disallow: /product-reviews/
730Disallow: /*?*editmode=
731Disallow: /api/*
732Disallow: /*?*languages=
733Disallow: /*?*stars=
734Disallow: /*?*sort=
735Disallow: /*?*verified=
736Disallow: /*?*topics=
737Disallow: /*?*replies=
738Disallow: /*?*date=
739Disallow: /*?*trustscore=
740Disallow: /*?*claimed=
741Disallow: /*?*country=
742Disallow: /*?*location=
743Disallow: /*?*subcategories=
744Disallow: /search
745Disallow: /sessions
746Disallow: /submitted/review
747Disallow: /users/
748Allow: /users/connect
749Disallow: /users/connect?redirect
750Disallow: /users/connect?signup
751Disallow: /users/connect?intent
752Disallow: /users/activate
753Disallow: /users/embed-login-to-edit
754Disallow: /users/create
755Disallow: /unsubscribe/invitation
756
757User-agent: GPTBot
758Disallow: /error
759Disallow: /evaluate/
760Disallow: /evaluate-link
761Disallow: /evaluate-unique-link
762Disallow: /pingdom
763Disallow: /reviews/
764Disallow: /review/*/transparency
765Disallow: /product-reviews/
766Disallow: /*?*editmode=
767Disallow: /api/*
768Disallow: /*?*languages=
769Disallow: /*?*stars=
770Disallow: /*?*sort=
771Disallow: /*?*verified=
772Disallow: /*?*topics=
773Disallow: /*?*replies=
774Disallow: /*?*date=
775Disallow: /*?*trustscore=
776Disallow: /*?*claimed=
777Disallow: /*?*country=
778Disallow: /*?*location=
779Disallow: /*?*subcategories=
780Disallow: /search
781Disallow: /sessions
782Disallow: /submitted/review
783Disallow: /users/
784Allow: /users/connect
785Disallow: /users/connect?redirect
786Disallow: /users/connect?signup
787Disallow: /users/connect?intent
788Disallow: /users/activate
789Disallow: /users/embed-login-to-edit
790Disallow: /users/create
791Disallow: /unsubscribe/invitation
792
793User-agent: Google-Extended
794Disallow: /error
795Disallow: /evaluate/
796Disallow: /evaluate-link
797Disallow: /evaluate-unique-link
798Disallow: /pingdom
799Disallow: /reviews/
800Disallow: /review/*/transparency
801Disallow: /product-reviews/
802Disallow: /*?*editmode=
803Disallow: /api/*
804Disallow: /*?*languages=
805Disallow: /*?*stars=
806Disallow: /*?*sort=
807Disallow: /*?*verified=
808Disallow: /*?*topics=
809Disallow: /*?*replies=
810Disallow: /*?*date=
811Disallow: /*?*trustscore=
812Disallow: /*?*claimed=
813Disallow: /*?*country=
814Disallow: /*?*location=
815Disallow: /*?*subcategories=
816Disallow: /search
817Disallow: /sessions
818Disallow: /submitted/review
819Disallow: /users/
820Allow: /users/connect
821Disallow: /users/connect?redirect
822Disallow: /users/connect?signup
823Disallow: /users/connect?intent
824Disallow: /users/activate
825Disallow: /users/embed-login-to-edit
826Disallow: /users/create
827Disallow: /unsubscribe/invitation
828
829User-agent: ClaudeBot
830Disallow: /error
831Disallow: /evaluate/
832Disallow: /evaluate-link
833Disallow: /evaluate-unique-link
834Disallow: /pingdom
835Disallow: /reviews/
836Disallow: /review/*/transparency
837Disallow: /product-reviews/
838Disallow: /*?*editmode=
839Disallow: /api/*
840Disallow: /*?*languages=
841Disallow: /*?*stars=
842Disallow: /*?*sort=
843Disallow: /*?*verified=
844Disallow: /*?*topics=
845Disallow: /*?*replies=
846Disallow: /*?*date=
847Disallow: /*?*trustscore=
848Disallow: /*?*claimed=
849Disallow: /*?*country=
850Disallow: /*?*location=
851Disallow: /*?*subcategories=
852Disallow: /search
853Disallow: /sessions
854Disallow: /submitted/review
855Disallow: /users/
856Allow: /users/connect
857Disallow: /users/connect?redirect
858Disallow: /users/connect?signup
859Disallow: /users/connect?intent
860Disallow: /users/activate
861Disallow: /users/embed-login-to-edit
862Disallow: /users/create
863Disallow: /unsubscribe/invitation
864
865User-agent: CCBot
866Disallow: /error
867Disallow: /evaluate/
868Disallow: /evaluate-link
869Disallow: /evaluate-unique-link
870Disallow: /pingdom
871Disallow: /reviews/
872Disallow: /review/*/transparency
873Disallow: /product-reviews/
874Disallow: /*?*editmode=
875Disallow: /api/*
876Disallow: /*?*languages=
877Disallow: /*?*stars=
878Disallow: /*?*sort=
879Disallow: /*?*verified=
880Disallow: /*?*topics=
881Disallow: /*?*replies=
882Disallow: /*?*date=
883Disallow: /*?*trustscore=
884Disallow: /*?*claimed=
885Disallow: /*?*country=
886Disallow: /*?*location=
887Disallow: /*?*subcategories=
888Disallow: /search
889Disallow: /sessions
890Disallow: /submitted/review
891Disallow: /users/
892Allow: /users/connect
893Disallow: /users/connect?redirect
894Disallow: /users/connect?signup
895Disallow: /users/connect?intent
896Disallow: /users/activate
897Disallow: /users/embed-login-to-edit
898Disallow: /users/create
899Disallow: /unsubscribe/invitation
900
901User-agent: meta-webindexer
902Disallow: /error
903Disallow: /evaluate/
904Disallow: /evaluate-link
905Disallow: /evaluate-unique-link
906Disallow: /pingdom
907Disallow: /reviews/
908Disallow: /review/*/transparency
909Disallow: /product-reviews/
910Disallow: /*?*editmode=
911Disallow: /api/*
912Disallow: /*?*languages=
913Disallow: /*?*stars=
914Disallow: /*?*sort=
915Disallow: /*?*verified=
916Disallow: /*?*topics=
917Disallow: /*?*replies=
918Disallow: /*?*date=
919Disallow: /*?*trustscore=
920Disallow: /*?*claimed=
921Disallow: /*?*country=
922Disallow: /*?*location=
923Disallow: /*?*subcategories=
924Disallow: /search
925Disallow: /sessions
926Disallow: /submitted/review
927Disallow: /users/
928Allow: /users/connect
929Disallow: /users/connect?redirect
930Disallow: /users/connect?signup
931Disallow: /users/connect?intent
932Disallow: /users/activate
933Disallow: /users/embed-login-to-edit
934Disallow: /users/create
935Disallow: /unsubscribe/invitation
936
937User-agent: meta-externalfetcher
938Disallow: /error
939Disallow: /evaluate/
940Disallow: /evaluate-link
941Disallow: /evaluate-unique-link
942Disallow: /pingdom
943Disallow: /reviews/
944Disallow: /review/*/transparency
945Disallow: /product-reviews/
946Disallow: /*?*editmode=
947Disallow: /api/*
948Disallow: /*?*languages=
949Disallow: /*?*stars=
950Disallow: /*?*sort=
951Disallow: /*?*verified=
952Disallow: /*?*topics=
953Disallow: /*?*replies=
954Disallow: /*?*date=
955Disallow: /*?*trustscore=
956Disallow: /*?*claimed=
957Disallow: /*?*country=
958Disallow: /*?*location=
959Disallow: /*?*subcategories=
960Disallow: /search
961Disallow: /sessions
962Disallow: /submitted/review
963Disallow: /users/
964Allow: /users/connect
965Disallow: /users/connect?redirect
966Disallow: /users/connect?signup
967Disallow: /users/connect?intent
968Disallow: /users/activate
969Disallow: /users/embed-login-to-edit
970Disallow: /users/create
971Disallow: /unsubscribe/invitation
972
973User-agent: meta-externalagent
974Disallow: /error
975Disallow: /evaluate/
976Disallow: /evaluate-link
977Disallow: /evaluate-unique-link
978Disallow: /pingdom
979Disallow: /reviews/
980Disallow: /review/*/transparency
981Disallow: /product-reviews/
982Disallow: /*?*editmode=
983Disallow: /api/*
984Disallow: /*?*languages=
985Disallow: /*?*stars=
986Disallow: /*?*sort=
987Disallow: /*?*verified=
988Disallow: /*?*topics=
989Disallow: /*?*replies=
990Disallow: /*?*date=
991Disallow: /*?*trustscore=
992Disallow: /*?*claimed=
993Disallow: /*?*country=
994Disallow: /*?*location=
995Disallow: /*?*subcategories=
996Disallow: /search
997Disallow: /sessions
998Disallow: /submitted/review
999Disallow: /users/
1000Allow: /users/connect
1001Disallow: /users/connect?redirect
1002Disallow: /users/connect?signup
1003Disallow: /users/connect?intent
1004Disallow: /users/activate
1005Disallow: /users/embed-login-to-edit
1006Disallow: /users/create
1007Disallow: /unsubscribe/invitation
1008
1009User-agent: sentibot
1010Disallow: /
1011
1012User-agent: Yandex
1013Disallow: /
1014
1015User-agent: AwarioRssBot
1016Disallow: /
1017
1018User-agent: AwarioSmartBot
1019Disallow: /
1020
1021User-agent: ImagesiftBot
1022Disallow: /
1023
1024User-agent: Applebot-Extended
1025Disallow: /
1026
1027User-agent: anthropic-ai
1028Disallow: /
1029
1030User-agent: Omgilibot
1031Disallow: /
1032
1033User-agent: Omgili
1034Disallow: /
1035
1036User-agent: Diffbot
1037Disallow: /
1038
1039User-agent: Bytespider
1040Disallow: /
1041
1042User-agent: ImagesiftBot
1043Disallow: /
1044
1045User-agent: Kangaroo Bot
1046Disallow: /
1047
1048User-agent: cohere-ai
1049Disallow: /
1050
1051User-agent: Timpibot
1052Disallow: /
1053
1054User-agent: Webzio-Extended
1055Disallow: /
1056
1057User-agent: YouBot
1058Disallow: /
1059
1060User-agent: *
1061Disallow: /
1062
1063Sitemap: https://sitemaps.trustpilot.com/index_en-us.xml
1064Sitemap: https://trustpilot.com/trust/sitemaps/domain_en-us.xml
1065Sitemap: https://trustpilot.com/blog/sitemaps/domain_en-us.xml
1066
1067
What do the bots actually get?
This page reads robots.txt, which states what trustpilot.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on trustpilot.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does trustpilot.com block GPTBot?
Partly. trustpilot.com’s robots.txt keeps GPTBot out of some paths, such as /error, and lets it crawl the rest. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does trustpilot.com block ClaudeBot?
Partly. trustpilot.com’s robots.txt keeps ClaudeBot out of some paths, such as /error, and lets it crawl the rest. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does trustpilot.com block Google-Extended?
Partly. trustpilot.com’s robots.txt keeps Google-Extended out of some paths, such as /error, and lets it crawl the rest. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does trustpilot.com block PerplexityBot?
Partly. trustpilot.com’s robots.txt keeps PerplexityBot out of some paths, such as /error, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find trustpilot.com?
Partly. trustpilot.com’s robots.txt keeps OAI-SearchBot out of some paths, such as /error, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.