Who blocks AI · #896 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does last.fm block AI crawlers?
No. last.fm’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.last.fm/robots.txt · 339 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | line 310 · Allow: / under User-agent: * | |
Bingbot | Microsoft | Partly blocked | line 5 · Disallow: /music/+noredirect/ line 6 · Disallow: /de/music/+noredirect/ and 286 more |
DuckDuckBot | DuckDuckGo | Allowed | line 310 · Allow: / under User-agent: * |
Applebot | Apple | Allowed | line 310 · Allow: / under User-agent: * |
PetalBot | Huawei | Allowed | line 310 · Allow: / under User-agent: * |
YandexBot | Yandex | Partly blocked | line 5 · Disallow: /music/+noredirect/ line 6 · Disallow: /de/music/+noredirect/ and 286 more |
Baiduspider | Baidu | Allowed | line 310 · Allow: / under User-agent: * |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | line 310 · Allow: / under User-agent: * |
PerplexityBot | Perplexity | Allowed | line 310 · Allow: / under User-agent: * |
Claude-SearchBot | Anthropic | Allowed | line 310 · Allow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Allowed | line 310 · Allow: / under User-agent: * |
Amazonbot | Amazon | Allowed | line 310 · Allow: / under User-agent: * |
YouBot | You.com | Allowed | line 310 · Allow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | line 310 · Allow: / under User-agent: * |
ClaudeBot | Anthropic | Allowed | line 310 · Allow: / under User-agent: * |
Google-Extended | Allowed | line 310 · Allow: / under User-agent: * | |
CCBot | Common Crawl | Allowed | line 310 · Allow: / under User-agent: * |
Applebot-Extended | Apple | Allowed | line 310 · Allow: / under User-agent: * |
GoogleOther | Allowed | line 310 · Allow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Allowed | line 310 · Allow: / under User-agent: * |
Bytespider | ByteDance | Allowed | line 310 · Allow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | line 310 · Allow: / under User-agent: * |
Claude-User | Anthropic | Allowed | line 310 · Allow: / under User-agent: * |
Perplexity-User | Perplexity | Allowed | line 310 · Allow: / under User-agent: * |
MistralAI-User | Mistral | Allowed | line 310 · Allow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Allowed | line 310 · Allow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | line 310 · Allow: / under User-agent: * |
SemrushBot | Semrush | Allowed | line 310 · Allow: / under User-agent: * |
MJ12bot | Majestic | Allowed | line 310 · Allow: / under User-agent: * |
DotBot | Moz | Allowed | line 310 · Allow: / under User-agent: * |
View www.last.fm/robots.txt · 339 lines
1User-Agent: Bingbot
2User-Agent: Yandexbot
3User-Agent: *
4
5Disallow: /music/+noredirect/
6Disallow: /de/music/+noredirect/
7Disallow: /es/music/+noredirect/
8Disallow: /fr/music/+noredirect/
9Disallow: /it/music/+noredirect/
10Disallow: /ja/music/+noredirect/
11Disallow: /pl/music/+noredirect/
12Disallow: /pt/music/+noredirect/
13Disallow: /ru/music/+noredirect/
14Disallow: /sv/music/+noredirect/
15Disallow: /tr/music/+noredirect/
16Disallow: /zh/music/+noredirect/
17
18Disallow: /search*
19Disallow: /de/search*
20Disallow: /es/search*
21Disallow: /fr/search*
22Disallow: /it/search*
23Disallow: /ja/search*
24Disallow: /pl/search*
25Disallow: /pt/search*
26Disallow: /ru/search*
27Disallow: /sv/search*
28Disallow: /tr/search*
29Disallow: /zh/search*
30
31# Old pages
32Disallow: /forum*
33Disallow: /de/forum*
34Disallow: /es/forum*
35Disallow: /fr/forum*
36Disallow: /it/forum*
37Disallow: /ja/forum*
38Disallow: /pl/forum*
39Disallow: /pt/forum*
40Disallow: /ru/forum*
41Disallow: /sv/forum*
42Disallow: /tr/forum*
43Disallow: /zh/forum*
44
45Disallow: /label*
46Disallow: /de/label*
47Disallow: /es/label*
48Disallow: /fr/label*
49Disallow: /it/label*
50Disallow: /ja/label*
51Disallow: /pl/label*
52Disallow: /pt/label*
53Disallow: /ru/label*
54Disallow: /sv/label*
55Disallow: /tr/label*
56Disallow: /zh/label*
57
58Disallow: /group*
59Disallow: /de/group*
60Disallow: /es/group*
61Disallow: /fr/group*
62Disallow: /it/group*
63Disallow: /ja/group*
64Disallow: /pl/group*
65Disallow: /pt/group*
66Disallow: /ru/group*
67Disallow: /sv/group*
68Disallow: /tr/group*
69Disallow: /zh/group*
70
71# Shouts
72Disallow: /event/*/shoutbox?
73Disallow: /de/event/*/shoutbox?
74Disallow: /es/event/*/shoutbox?
75Disallow: /fr/event/*/shoutbox?
76Disallow: /it/event/*/shoutbox?
77Disallow: /ja/event/*/shoutbox?
78Disallow: /pl/event/*/shoutbox?
79Disallow: /pt/event/*/shoutbox?
80Disallow: /ru/event/*/shoutbox?
81Disallow: /sv/event/*/shoutbox?
82Disallow: /tr/event/*/shoutbox?
83Disallow: /zh/event/*/shoutbox?
84
85Disallow: /event/*/shoutbox*
86Disallow: /de/event/*/shoutbox*
87Disallow: /es/event/*/shoutbox*
88Disallow: /fr/event/*/shoutbox*
89Disallow: /it/event/*/shoutbox*
90Disallow: /ja/event/*/shoutbox*
91Disallow: /pl/event/*/shoutbox*
92Disallow: /pt/event/*/shoutbox*
93Disallow: /ru/event/*/shoutbox*
94Disallow: /sv/event/*/shoutbox*
95Disallow: /tr/event/*/shoutbox*
96Disallow: /zh/event/*/shoutbox*
97
98Disallow: /festival/*/shoutbox*
99Disallow: /de/festival/*/shoutbox*
100Disallow: /es/festival/*/shoutbox*
101Disallow: /fr/festival/*/shoutbox*
102Disallow: /it/festival/*/shoutbox*
103Disallow: /ja/festival/*/shoutbox*
104Disallow: /pl/festival/*/shoutbox*
105Disallow: /pt/festival/*/shoutbox*
106Disallow: /ru/festival/*/shoutbox*
107Disallow: /sv/festival/*/shoutbox*
108Disallow: /tr/festival/*/shoutbox*
109Disallow: /zh/festival/*/shoutbox*
110
111Disallow: /tag/*
112Disallow: /de/tag/*
113Disallow: /es/tag/*
114Disallow: /fr/tag/*
115Disallow: /it/tag/*
116Disallow: /ja/tag/*
117Disallow: /pl/tag/*
118Disallow: /pt/tag/*
119Disallow: /ru/tag/*
120Disallow: /sv/tag/*
121Disallow: /tr/tag/*
122Disallow: /zh/tag/*
123
124Disallow: /music/*?
125Disallow: /de/music/*?
126Disallow: /es/music/*?
127Disallow: /fr/music/*?
128Disallow: /it/music/*?
129Disallow: /ja/music/*?
130Disallow: /pl/music/*?
131Disallow: /pt/music/*?
132Disallow: /ru/music/*?
133Disallow: /sv/music/*?
134Disallow: /tr/music/*?
135Disallow: /zh/music/*?
136
137Disallow: /music/*/+tags*
138Disallow: /de/music/*/+tags*
139Disallow: /es/music/*/+tags*
140Disallow: /fr/music/*/+tags*
141Disallow: /it/music/*/+tags*
142Disallow: /ja/music/*/+tags*
143Disallow: /pl/music/*/+tags*
144Disallow: /pt/music/*/+tags*
145Disallow: /ru/music/*/+tags*
146Disallow: /sv/music/*/+tags*
147Disallow: /tr/music/*/+tags*
148Disallow: /zh/music/*/+tags*
149
150Disallow: /music/*/+shoutbox?
151Disallow: /de/music/*/+shoutbox?
152Disallow: /es/music/*/+shoutbox?
153Disallow: /fr/music/*/+shoutbox?
154Disallow: /it/music/*/+shoutbox?
155Disallow: /ja/music/*/+shoutbox?
156Disallow: /pl/music/*/+shoutbox?
157Disallow: /pt/music/*/+shoutbox?
158Disallow: /ru/music/*/+shoutbox?
159Disallow: /sv/music/*/+shoutbox?
160Disallow: /tr/music/*/+shoutbox?
161Disallow: /zh/music/*/+shoutbox?
162
163Disallow: /music/*/+shoutbox*
164Disallow: /de/music/*/+shoutbox*
165Disallow: /es/music/*/+shoutbox*
166Disallow: /fr/music/*/+shoutbox*
167Disallow: /it/music/*/+shoutbox*
168Disallow: /ja/music/*/+shoutbox*
169Disallow: /pl/music/*/+shoutbox*
170Disallow: /pt/music/*/+shoutbox*
171Disallow: /ru/music/*/+shoutbox*
172Disallow: /sv/music/*/+shoutbox*
173Disallow: /tr/music/*/+shoutbox*
174Disallow: /zh/music/*/+shoutbox*
175
176# N.B: these are not covered by the above /music/ rule
177# (shoutbox vs +shoutbox)
178Disallow: /music/*/+images/*/shoutbox?
179Disallow: /de/music/*/+images/*/shoutbox?
180Disallow: /es/music/*/+images/*/shoutbox?
181Disallow: /fr/music/*/+images/*/shoutbox?
182Disallow: /it/music/*/+images/*/shoutbox?
183Disallow: /ja/music/*/+images/*/shoutbox?
184Disallow: /pl/music/*/+images/*/shoutbox?
185Disallow: /pt/music/*/+images/*/shoutbox?
186Disallow: /ru/music/*/+images/*/shoutbox?
187Disallow: /sv/music/*/+images/*/shoutbox?
188Disallow: /tr/music/*/+images/*/shoutbox?
189Disallow: /zh/music/*/+images/*/shoutbox?
190
191Disallow: /music/*/+images/*/shoutbox*
192Disallow: /de/music/*/+images/*/shoutbox*
193Disallow: /es/music/*/+images/*/shoutbox*
194Disallow: /fr/music/*/+images/*/shoutbox*
195Disallow: /it/music/*/+images/*/shoutbox*
196Disallow: /ja/music/*/+images/*/shoutbox*
197Disallow: /pl/music/*/+images/*/shoutbox*
198Disallow: /pt/music/*/+images/*/shoutbox*
199Disallow: /ru/music/*/+images/*/shoutbox*
200Disallow: /sv/music/*/+images/*/shoutbox*
201Disallow: /tr/music/*/+images/*/shoutbox*
202Disallow: /zh/music/*/+images/*/shoutbox*
203
204Disallow: /music/*/+wiki/shoutbox*
205Disallow: /de/music/*/+wiki/shoutbox*
206Disallow: /es/music/*/+wiki/shoutbox*
207Disallow: /fr/music/*/+wiki/shoutbox*
208Disallow: /it/music/*/+wiki/shoutbox*
209Disallow: /ja/music/*/+wiki/shoutbox*
210Disallow: /pl/music/*/+wiki/shoutbox*
211Disallow: /pt/music/*/+wiki/shoutbox*
212Disallow: /ru/music/*/+wiki/shoutbox*
213Disallow: /sv/music/*/+wiki/shoutbox*
214Disallow: /tr/music/*/+wiki/shoutbox*
215Disallow: /zh/music/*/+wiki/shoutbox*
216
217Disallow: /music/*/+images*
218Disallow: /de/music/*/+images*
219Disallow: /es/music/*/+images*
220Disallow: /fr/music/*/+images*
221Disallow: /it/music/*/+images*
222Disallow: /ja/music/*/+images*
223Disallow: /pl/music/*/+images*
224Disallow: /pt/music/*/+images*
225Disallow: /ru/music/*/+images*
226Disallow: /sv/music/*/+images*
227Disallow: /tr/music/*/+images*
228Disallow: /zh/music/*/+images*
229
230Disallow: /music/*/+listeners*
231Disallow: /de/music/*/+listeners*
232Disallow: /es/music/*/+listeners*
233Disallow: /fr/music/*/+listeners*
234Disallow: /it/music/*/+listeners*
235Disallow: /ja/music/*/+listeners*
236Disallow: /pl/music/*/+listeners*
237Disallow: /pt/music/*/+listeners*
238Disallow: /ru/music/*/+listeners*
239Disallow: /sv/music/*/+listeners*
240Disallow: /tr/music/*/+listeners*
241Disallow: /zh/music/*/+listeners*
242
243# AJAX content
244Disallow: /search/autocomplete
245Disallow: /de/search/autocomplete
246Disallow: /es/search/autocomplete
247Disallow: /fr/search/autocomplete
248Disallow: /it/search/autocomplete
249Disallow: /ja/search/autocomplete
250Disallow: /pl/search/autocomplete
251Disallow: /pt/search/autocomplete
252Disallow: /ru/search/autocomplete
253Disallow: /sv/search/autocomplete
254Disallow: /tr/search/autocomplete
255Disallow: /zh/search/autocomplete
256
257Disallow: /player
258Disallow: /de/player
259Disallow: /es/player
260Disallow: /fr/player
261Disallow: /it/player
262Disallow: /ja/player
263Disallow: /pl/player
264Disallow: /pt/player
265Disallow: /ru/player
266Disallow: /sv/player
267Disallow: /tr/player
268Disallow: /zh/player
269
270Disallow: /login?next=/*
271Disallow: /de/login?next=/*
272Disallow: /es/login?next=/*
273Disallow: /fr/login?next=/*
274Disallow: /it/login?next=/*
275Disallow: /ja/login?next=/*
276Disallow: /pl/login?next=/*
277Disallow: /pt/login?next=/*
278Disallow: /ru/login?next=/*
279Disallow: /sv/login?next=/*
280Disallow: /tr/login?next=/*
281Disallow: /zh/login?next=/*
282
283Disallow: /features*
284Disallow: /de/features*
285Disallow: /es/features*
286Disallow: /fr/features*
287Disallow: /it/features*
288Disallow: /ja/features*
289Disallow: /pl/features*
290Disallow: /pt/features*
291Disallow: /ru/features*
292Disallow: /sv/features*
293Disallow: /tr/features*
294Disallow: /zh/features*
295
296Disallow: /settings*
297Disallow: /de/settings*
298Disallow: /es/settings*
299Disallow: /fr/settings*
300Disallow: /it/settings*
301Disallow: /ja/settings*
302Disallow: /pl/settings*
303Disallow: /pt/settings*
304Disallow: /ru/settings*
305Disallow: /sv/settings*
306Disallow: /tr/settings*
307Disallow: /zh/settings*
308
309
310Allow: /
311
312Sitemap: https://www.last.fm/sitemap-index-secure.xml
313
314User-Agent: Bingbot
315Crawl-Delay: 1
316
317User-Agent: Yandexbot
318Crawl-Delay: 1
319Disallow: /user/*
320Disallow: /de/user/*
321Disallow: /es/user/*
322Disallow: /fr/user/*
323Disallow: /it/user/*
324Disallow: /ja/user/*
325Disallow: /pl/user/*
326Disallow: /pt/user/*
327Disallow: /ru/user/*
328Disallow: /sv/user/*
329Disallow: /tr/user/*
330Disallow: /zh/user/*
331
332User-Agent: *
333Allow: /ads.txt
334Allow: /app-ads.txt
335
336Disallow: /harming/humans
337Disallow: /ignoring/human/orders
338Disallow: /harm/to/self
339
What do the bots actually get?
This page reads robots.txt, which states what last.fm asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on last.fm Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does last.fm block GPTBot?
No. last.fm’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does last.fm block ClaudeBot?
No. last.fm’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does last.fm block Google-Extended?
No. last.fm’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does last.fm block PerplexityBot?
No. last.fm’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find last.fm?
Yes. last.fm’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.