Who blocks AI · #150 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does unsplash.com block AI crawlers?
Yes. unsplash.com’s robots.txt blocks 3 of the 19 AI crawlers we track, including ClaudeBot, Bytespider and Claude-SearchBot. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- unsplash.com/robots.txt · 320 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Blocked | line 274 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Blocked | line 274 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Blocked | line 278 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 292 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Blocked | line 287 · Disallow: / under User-agent: dotbot |
View unsplash.com/robots.txt · 320 lines
1User-Agent: *
2
3Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/de/sitemap.xml
4
5Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/de/sitemap.xml
6
7Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/de/sitemap.xml
8
9Sitemap: https://sitemaps.unsplash.com/sitemaps/static/en/sitemap.xml
10
11Sitemap: https://sitemaps.unsplash.com/sitemaps/landing_pages/en/sitemap.xml
12
13Sitemap: https://sitemaps.unsplash.com/sitemaps/collections/en/sitemap.xml
14
15Sitemap: https://sitemaps.unsplash.com/sitemaps/illustration_searches/en/sitemap.xml
16
17Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/en/sitemap.xml
18
19Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/en/sitemap.xml
20
21Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/en/sitemap.xml
22
23Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/es/sitemap.xml
24
25Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/es/sitemap.xml
26
27Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/es/sitemap.xml
28
29Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/fr/sitemap.xml
30
31Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/fr/sitemap.xml
32
33Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/fr/sitemap.xml
34
35Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/id/sitemap.xml
36
37Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/id/sitemap.xml
38
39Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/id/sitemap.xml
40
41Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/it/sitemap.xml
42
43Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/it/sitemap.xml
44
45Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/it/sitemap.xml
46
47Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/ja/sitemap.xml
48
49Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/ja/sitemap.xml
50
51Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/ja/sitemap.xml
52
53Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/ko/sitemap.xml
54
55Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/ko/sitemap.xml
56
57Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/ko/sitemap.xml
58
59Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/pt/sitemap.xml
60
61Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/pt/sitemap.xml
62
63Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/pt/sitemap.xml
64
65Allow: /blog
66
67Allow: /de/backgrounds
68
69Allow: /backgrounds
70
71Allow: /es/backgrounds
72
73Allow: /fr/backgrounds
74
75Allow: /id/backgrounds
76
77Allow: /it/backgrounds
78
79Allow: /ja/backgrounds
80
81Allow: /ko/backgrounds
82
83Allow: /pt-br/backgrounds
84
85Disallow: /photos/*/download
86
87Disallow: /illustrations/*/download
88
89Disallow: /ngetty
90
91Disallow: /nautocomplete
92
93Disallow: /nabc
94
95Disallow: /de/account
96Disallow: /de/account/*
97Disallow: /de/b
98Disallow: /de/b/*
99Disallow: /de/community
100Disallow: /de/einstellung
101Disallow: /de/einstellung/job-posts/*
102Disallow: /de/kollektionen
103Disallow: /de/kollektionen/*
104Disallow: /de/lesezeichen
105Disallow: /de/login
106Disallow: /de/plus/r/*
107Disallow: /de/presse
108Disallow: /de/registrieren
109Disallow: /de/s/benutzer/*
110Disallow: /de/s/kollektionen/*
111Disallow: /de/s/visuell/*
112Disallow: /de/werben
113Disallow: /de/werben/produktplatzierung
114
115Disallow: /account
116Disallow: /account/*
117Disallow: /b
118Disallow: /b/*
119Disallow: /bookmarks
120Disallow: /join
121Disallow: /login
122Disallow: /plus/r/*
123Disallow: /s/collections/*
124Disallow: /s/users/*
125Disallow: /s/visual/*
126
127Disallow: /es/%C3%BAnete
128Disallow: /es/account
129Disallow: /es/account/*
130Disallow: /es/anunciarse
131Disallow: /es/anunciarse/colocaci%C3%B3n-de-producto
132Disallow: /es/b
133Disallow: /es/b/*
134Disallow: /es/colecciones
135Disallow: /es/colecciones/*
136Disallow: /es/comunidad
137Disallow: /es/contrataci%C3%B3n
138Disallow: /es/contrataci%C3%B3n/job-posts/*
139Disallow: /es/guardados
140Disallow: /es/inicia-sesi%C3%B3n
141Disallow: /es/plus/r/*
142Disallow: /es/prensa
143Disallow: /es/s/colecciones/*
144Disallow: /es/s/imagen/*
145Disallow: /es/s/usuarios/*
146
147Disallow: /fr/account
148Disallow: /fr/account/*
149Disallow: /fr/b
150Disallow: /fr/b/*
151Disallow: /fr/collections
152Disallow: /fr/collections/*
153Disallow: /fr/communaut%C3%A9
154Disallow: /fr/connexion
155Disallow: /fr/faire-votre-publicit%C3%A9
156Disallow: /fr/faire-votre-publicit%C3%A9/placement-de-produit
157Disallow: /fr/favoris
158Disallow: /fr/newsroom
159Disallow: /fr/plus/r/*
160Disallow: /fr/recrutement
161Disallow: /fr/recrutement/job-posts/*
162Disallow: /fr/s%E2%80%99inscrire
163Disallow: /fr/s/collections/*
164Disallow: /fr/s/utilisateurs/*
165Disallow: /fr/s/visuel/*
166
167Disallow: /id/account
168Disallow: /id/account/*
169Disallow: /id/ada-lowongan
170Disallow: /id/ada-lowongan/job-posts/*
171Disallow: /id/b
172Disallow: /id/b/*
173Disallow: /id/berita
174Disallow: /id/bookmark
175Disallow: /id/daftar
176Disallow: /id/iklankan
177Disallow: /id/iklankan/penempatan-produk
178Disallow: /id/koleksi
179Disallow: /id/koleksi/*
180Disallow: /id/komunitas
181Disallow: /id/masuk
182Disallow: /id/plus/r/*
183Disallow: /id/s/koleksi/*
184Disallow: /id/s/pengguna/*
185Disallow: /id/s/visual/*
186
187Disallow: /it/accedi
188Disallow: /it/account
189Disallow: /it/account/*
190Disallow: /it/assunzioni
191Disallow: /it/assunzioni/job-posts/*
192Disallow: /it/b
193Disallow: /it/b/*
194Disallow: /it/collezioni
195Disallow: /it/collezioni/*
196Disallow: /it/community
197Disallow: /it/diventa-inserzionista
198Disallow: /it/diventa-inserzionista/posizionamento-del-prodotto
199Disallow: /it/plus/r/*
200Disallow: /it/press-room
201Disallow: /it/registrati
202Disallow: /it/s/collezioni/*
203Disallow: /it/s/immagini/*
204Disallow: /it/s/utenti/*
205Disallow: /it/segnalibri
206
207Disallow: /ja/%E3%82%B3%E3%83%9F%E3%83%A5%E3%83%8B%E3%83%86%E3%82%A3
208Disallow: /ja/%E3%82%B3%E3%83%AC%E3%82%AF%E3%82%B7%E3%83%A7%E3%83%B3
209Disallow: /ja/%E3%82%B3%E3%83%AC%E3%82%AF%E3%82%B7%E3%83%A7%E3%83%B3/*
210Disallow: /ja/%E3%82%B5%E3%82%A4%E3%83%B3%E3%82%A4%E3%83%B3
211Disallow: /ja/%E3%83%96%E3%83%83%E3%82%AF%E3%83%9E%E3%83%BC%E3%82%AF
212Disallow: /ja/%E3%83%97%E3%83%AC%E3%82%B9
213Disallow: /ja/%E5%BA%83%E5%91%8A
214Disallow: /ja/%E5%BA%83%E5%91%8A/%E3%83%97%E3%83%AD%E3%83%80%E3%82%AF%E3%83%88%E3%83%97%E3%83%AC%E3%82%A4%E3%82%B9%E3%83%A1%E3%83%B3%E3%83%88
215Disallow: /ja/%E6%A1%88%E4%BB%B6%E4%BE%9D%E9%A0%BC%E4%B8%AD
216Disallow: /ja/%E6%A1%88%E4%BB%B6%E4%BE%9D%E9%A0%BC%E4%B8%AD/job-posts/*
217Disallow: /ja/%E7%99%BB%E9%8C%B2
218Disallow: /ja/account
219Disallow: /ja/account/*
220Disallow: /ja/b
221Disallow: /ja/b/*
222Disallow: /ja/plus/r/*
223Disallow: /ja/s/%E3%82%B3%E3%83%AC%E3%82%AF%E3%82%B7%E3%83%A7%E3%83%B3/*
224Disallow: /ja/s/%E3%83%93%E3%82%B8%E3%83%A5%E3%82%A2%E3%83%AB/*
225Disallow: /ja/s/%E3%83%A6%E3%83%BC%E3%82%B6%E3%83%BC/*
226
227Disallow: /ko/%EA%B0%80%EC%9E%85
228Disallow: /ko/%EA%B3%A0%EC%9A%A9-%EC%A4%91
229Disallow: /ko/%EA%B3%A0%EC%9A%A9-%EC%A4%91/job-posts/*
230Disallow: /ko/%EA%B4%91%EA%B3%A0
231Disallow: /ko/%EA%B4%91%EA%B3%A0/%EA%B0%84%EC%A0%91-%EA%B4%91%EA%B3%A0
232Disallow: /ko/%EB%A1%9C%EA%B7%B8%EC%9D%B8
233Disallow: /ko/%EB%B3%B4%EB%8F%84%EC%9E%90%EB%A3%8C
234Disallow: /ko/%EB%B6%81%EB%A7%88%ED%81%AC
235Disallow: /ko/%EC%BB%A4%EB%AE%A4%EB%8B%88%ED%8B%B0
236Disallow: /ko/%EC%BB%AC%EB%A0%89%EC%85%98
237Disallow: /ko/%EC%BB%AC%EB%A0%89%EC%85%98/*
238Disallow: /ko/%ED%94%8C%EB%9F%AC%EC%8A%A4/r/*
239Disallow: /ko/account
240Disallow: /ko/account/*
241Disallow: /ko/b
242Disallow: /ko/b/*
243Disallow: /ko/s/%EC%82%AC%EC%9A%A9%EC%9E%90/*
244Disallow: /ko/s/%EC%8B%9C%EA%B0%81-%EC%9E%90%EB%A3%8C/*
245Disallow: /ko/s/%EC%BB%AC%EB%A0%89%EC%85%98/*
246
247Disallow: /pt-br/account
248Disallow: /pt-br/account/*
249Disallow: /pt-br/anunciar
250Disallow: /pt-br/anunciar/posicionamento-de-produto
251Disallow: /pt-br/b
252Disallow: /pt-br/b/*
253Disallow: /pt-br/cadastre-se
254Disallow: /pt-br/cole%C3%A7%C3%B5es
255Disallow: /pt-br/cole%C3%A7%C3%B5es/*
256Disallow: /pt-br/comunidade
257Disallow: /pt-br/contratando
258Disallow: /pt-br/contratando/job-posts/*
259Disallow: /pt-br/favoritos
260Disallow: /pt-br/imprensa
261Disallow: /pt-br/login
262Disallow: /pt-br/plus/r/*
263Disallow: /pt-br/s/cole%C3%A7%C3%B5es/*
264Disallow: /pt-br/s/usu%C3%A1rios/*
265Disallow: /pt-br/s/visual/*
266
267User-Agent: ImagesiftBot
268
269Disallow: /
270
271User-Agent: Claude-SearchBot
272User-Agent: ClaudeBot
273
274Disallow: /
275
276User-Agent: Bytespider
277
278Disallow: /
279
280User-Agent: AwarioRssBot
281User-Agent: AwarioSmartBot
282
283Disallow: /
284
285User-Agent: dotbot
286
287Disallow: /
288
289User-Agent: AhrefsBot
290User-Agent: AhrefsSiteAudit
291
292Disallow: /
293
294User-Agent: MJ12bot
295
296User-Agent: SemrushBot-BA
297User-Agent: SemrushBot-ESI
298User-Agent: SemrushBot-FT
299User-Agent: SemrushBot-OCOB
300User-Agent: SemrushBot-SI
301User-Agent: SemrushBot-SWA
302User-Agent: SiteAuditBot
303
304Disallow: /
305
306User-Agent: BrightEdge
307
308Disallow: /
309
310User-agent: proximic
311
312Disallow: /
313
314User-Agent: DataForSeoBot
315
316Disallow: /
317
318User-Agent: SeobilityBot
319
320Disallow: /
What do the bots actually get?
This page reads robots.txt, which states what unsplash.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on unsplash.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does unsplash.com block GPTBot?
No. unsplash.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does unsplash.com block ClaudeBot?
Yes. unsplash.com’s robots.txt disallows ClaudeBot from the whole site (line 274: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does unsplash.com block Google-Extended?
No. unsplash.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does unsplash.com block PerplexityBot?
No. unsplash.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find unsplash.com?
Yes. unsplash.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.