Who blocks AI · #124 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does linktr.ee block AI crawlers?
Yes. linktr.ee’s robots.txt blocks 4 of the 19 AI crawlers we track, including Meta-ExternalAgent, Bytespider, YouBot and GoogleOther. It also blocks DuckDuckBot, PetalBot and YandexBot among search engines.
GPTBotPartly blockedOAI-SearchBotPartly blockedClaudeBotPartly blockedPerplexityBotPartly blockedGoogle-ExtendedAllowedCCBotPartly blocked- robots.txt
- linktr.ee/robots.txt · 450 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 67 · Disallow: /admin$ line 68 · Disallow: /admin/ and 2 more | |
Bingbot | Microsoft | Partly blocked | line 46 · Disallow: /admin$ line 47 · Disallow: /admin/ and 2 more |
DuckDuckBot | DuckDuckGo | Blocked | line 443 · Disallow: / under User-agent: * |
Applebot | Apple | Partly blocked | line 28 · Disallow: /admin$ line 29 · Disallow: /admin/ and 2 more |
PetalBot | Huawei | Blocked | line 443 · Disallow: / under User-agent: * |
YandexBot | Yandex | Blocked | line 443 · Disallow: / under User-agent: * |
Baiduspider | Baidu | Partly blocked | line 39 · Disallow: /admin$ line 40 · Disallow: /admin/ and 2 more |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Partly blocked | line 194 · Disallow: /admin$ line 195 · Disallow: /admin/ and 2 more |
PerplexityBot | Perplexity | Partly blocked | line 266 · Disallow: /admin$ line 267 · Disallow: /admin/ and 2 more |
Claude-SearchBot | Anthropic | Partly blocked | line 237 · Disallow: /admin$ line 238 · Disallow: /admin/ and 2 more |
DuckAssistBot | DuckDuckGo | Partly blocked | line 422 · Disallow: /admin$ line 423 · Disallow: /admin/ and 2 more |
Amazonbot | Amazon | Partly blocked | line 351 · Disallow: /admin$ line 352 · Disallow: /admin/ and 2 more |
YouBot | You.com | Blocked | line 443 · Disallow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Partly blocked | line 229 · Disallow: /admin$ line 230 · Disallow: /admin/ and 2 more |
ClaudeBot | Anthropic | Partly blocked | line 258 · Disallow: /admin$ line 259 · Disallow: /admin/ and 2 more |
Google-Extended | Allowed | line 304 · Allow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Partly blocked | line 380 · Disallow: /admin$ line 381 · Disallow: /admin/ and 2 more |
Applebot-Extended | Apple | Allowed | line 35 · Allow: / under User-agent: Applebot-Extended |
GoogleOther | Blocked | line 443 · Disallow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Blocked | line 443 · Disallow: / under User-agent: * |
Bytespider | ByteDance | Blocked | line 443 · Disallow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Partly blocked | line 201 · Disallow: /admin$ line 202 · Disallow: /admin/ and 6 more |
Claude-User | Anthropic | Partly blocked | line 244 · Disallow: /admin$ line 245 · Disallow: /admin/ and 2 more |
Perplexity-User | Perplexity | Partly blocked | line 273 · Disallow: /admin$ line 274 · Disallow: /admin/ and 2 more |
MistralAI-User | Mistral | Partly blocked | line 281 · Disallow: /admin$ line 282 · Disallow: /admin/ and 6 more |
Meta-ExternalFetcher | Meta | Partly blocked | line 296 · Disallow: /admin$ line 297 · Disallow: /admin/ and 2 more |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Partly blocked | line 436 · Disallow: /admin$ line 437 · Disallow: /admin/ and 2 more |
SemrushBot | Semrush | Partly blocked | line 186 · Disallow: /admin$ line 187 · Disallow: /admin/ and 2 more |
MJ12bot | Majestic | Blocked | line 443 · Disallow: / under User-agent: * |
DotBot | Moz | Blocked | line 443 · Disallow: / under User-agent: * |
View linktr.ee/robots.txt · 450 lines
1
2# As a condition of accessing this website, you agree to abide by the following
3# content signals:
4# (a) If a content-signal = yes, you may collect content for the corresponding use.
5# (b) If a content-signal = no, you may not collect content for the corresponding use.
6# (c) If no content signal is present for a corresponding use, the website operator
7# neither grants nor restricts permission via content signal for that use.
8# The content signals and their meanings are:
9# search: building a search index and providing search results (e.g., returning
10# hyperlinks and short excerpts). Search does not include providing
11# AI-generated search summaries.
12# ai-input: inputting content into one or more AI models (e.g., retrieval augmented
13# generation, grounding, or other real-time taking of content for
14# generative AI search answers).
15# ai-train: training or fine-tuning AI models.
16# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF RIGHTS
17# UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED
18# RIGHTS IN THE DIGITAL SINGLE MARKET.
19#
20# llms.txt: https://linktr.ee/llms.txt
21#
22# /s/about/trust-center/report is disallowed because every profile footer's "Report"
23# link appends a unique ?field86145911=<profile url> param, so crawlers discover one
24# report URL per profile
25
26User-agent: Applebot
27Content-Signal: ai-train=no, search=yes
28Disallow: /admin$
29Disallow: /admin/
30Disallow: /admin?
31Disallow: /s/about/trust-center/report
32
33# Applebot-Extended controls use of content for Apple Intelligence / generative model training
34User-agent: Applebot-Extended
35Allow: /
36
37User-agent: baiduspider
38Content-Signal: ai-train=no, search=yes
39Disallow: /admin$
40Disallow: /admin/
41Disallow: /admin?
42Disallow: /s/about/trust-center/report
43
44User-agent: Bingbot
45Content-Signal: ai-train=no, search=yes
46Disallow: /admin$
47Disallow: /admin/
48Disallow: /admin?
49Disallow: /s/about/trust-center/report
50
51User-agent: Discordbot
52Content-Signal: ai-train=no, search=yes
53Disallow: /admin$
54Disallow: /admin/
55Disallow: /admin?
56Disallow: /s/about/trust-center/report
57
58User-agent: facebookexternalhit
59Content-Signal: ai-train=no, search=yes
60Disallow: /admin$
61Disallow: /admin/
62Disallow: /admin?
63Disallow: /s/about/trust-center/report
64
65User-agent: Googlebot
66Content-Signal: ai-train=no, search=yes
67Disallow: /admin$
68Disallow: /admin/
69Disallow: /admin?
70Disallow: /s/about/trust-center/report
71
72User-agent: Googlebot-Image
73Content-Signal: ai-train=no, search=yes
74Disallow: /admin$
75Disallow: /admin/
76Disallow: /admin?
77Disallow: /s/about/trust-center/report
78
79User-agent: Google-InspectionTool
80Content-Signal: ai-train=no, search=yes
81Disallow: /admin$
82Disallow: /admin/
83Disallow: /admin?
84Disallow: /s/about/trust-center/report
85
86User-agent: ia_archiver
87Content-Signal: ai-train=no, search=yes
88Disallow: /admin$
89Disallow: /admin/
90Disallow: /admin?
91Disallow: /s/about/trust-center/report
92
93User-agent: LinkedInBot
94Content-Signal: ai-train=no, search=yes
95Disallow: /admin$
96Disallow: /admin/
97Disallow: /admin?
98Disallow: /s/about/trust-center/report
99
100User-agent: msnbot
101Content-Signal: ai-train=no, search=yes
102Disallow: /admin$
103Disallow: /admin/
104Disallow: /admin?
105Disallow: /s/about/trust-center/report
106
107User-agent: Naverbot
108Content-Signal: ai-train=no, search=yes
109Disallow: /admin$
110Disallow: /admin/
111Disallow: /admin?
112Disallow: /s/about/trust-center/report
113
114User-agent: Pinterestbot
115Content-Signal: ai-train=no, search=yes
116Disallow: /admin$
117Disallow: /admin/
118Disallow: /admin?
119Disallow: /s/about/trust-center/report
120
121User-agent: Screaming Frog SEO Spider
122Content-Signal: ai-train=no, search=yes
123Disallow: /admin$
124Disallow: /admin/
125Disallow: /admin?
126Disallow: /s/about/trust-center/report
127
128User-agent: seznambot
129Content-Signal: ai-train=no, search=yes
130Disallow: /admin$
131Disallow: /admin/
132Disallow: /admin?
133Disallow: /s/about/trust-center/report
134
135User-agent: Slurp
136Content-Signal: ai-train=no, search=yes
137Disallow: /admin$
138Disallow: /admin/
139Disallow: /admin?
140Disallow: /s/about/trust-center/report
141
142User-agent: teoma
143Content-Signal: ai-train=no, search=yes
144Disallow: /admin$
145Disallow: /admin/
146Disallow: /admin?
147Disallow: /s/about/trust-center/report
148
149User-agent: TelegramBot
150Content-Signal: ai-train=no, search=yes
151Disallow: /admin$
152Disallow: /admin/
153Disallow: /admin?
154Disallow: /s/about/trust-center/report
155
156User-agent: Twitterbot
157Content-Signal: ai-train=no, search=yes
158Disallow: /admin$
159Disallow: /admin/
160Disallow: /admin?
161Disallow: /s/about/trust-center/report
162
163User-agent: Yandex
164Content-Signal: ai-train=no, search=yes
165Disallow: /admin$
166Disallow: /admin/
167Disallow: /admin?
168Disallow: /s/about/trust-center/report
169
170User-agent: Yeti
171Content-Signal: ai-train=no, search=yes
172Disallow: /admin$
173Disallow: /admin/
174Disallow: /admin?
175Disallow: /s/about/trust-center/report
176
177User-agent: SnapchatAdsBot
178Content-Signal: ai-train=no, search=yes
179Disallow: /admin$
180Disallow: /admin/
181Disallow: /admin?
182Disallow: /s/about/trust-center/report
183
184User-agent: SemrushBot
185Content-Signal: ai-train=no, search=yes
186Disallow: /admin$
187Disallow: /admin/
188Disallow: /admin?
189Disallow: /s/about/trust-center/report
190
191# OpenAI
192User-agent: OAI-SearchBot
193Content-Signal: ai-train=no, search=yes
194Disallow: /admin$
195Disallow: /admin/
196Disallow: /admin?
197Disallow: /s/about/trust-center/report
198
199User-agent: ChatGPT-User
200Content-Signal: ai-train=no, search=yes
201Disallow: /admin$
202Disallow: /admin/
203Disallow: /admin?
204Disallow: /s/about/trust-center/report
205
206User-agent: ChatGPT-User/2.0
207Content-Signal: ai-train=no, search=yes
208Disallow: /admin$
209Disallow: /admin/
210Disallow: /admin?
211Disallow: /s/about/trust-center/report
212
213User-agent: ChatGPT Agent
214Content-Signal: ai-train=no, search=yes
215Disallow: /admin$
216Disallow: /admin/
217Disallow: /admin?
218Disallow: /s/about/trust-center/report
219
220User-agent: OpenAI
221Content-Signal: ai-train=no, search=yes
222Disallow: /admin$
223Disallow: /admin/
224Disallow: /admin?
225Disallow: /s/about/trust-center/report
226
227User-agent: GPTBot
228Content-Signal: ai-train=no, search=yes
229Disallow: /admin$
230Disallow: /admin/
231Disallow: /admin?
232Disallow: /s/about/trust-center/report
233
234# Anthropic
235User-agent: Claude-SearchBot
236Content-Signal: ai-train=no, search=yes
237Disallow: /admin$
238Disallow: /admin/
239Disallow: /admin?
240Disallow: /s/about/trust-center/report
241
242User-agent: Claude-User
243Content-Signal: ai-train=no, search=yes
244Disallow: /admin$
245Disallow: /admin/
246Disallow: /admin?
247Disallow: /s/about/trust-center/report
248
249User-agent: Claude-Web
250Content-Signal: ai-train=no, search=yes
251Disallow: /admin$
252Disallow: /admin/
253Disallow: /admin?
254Disallow: /s/about/trust-center/report
255
256User-agent: ClaudeBot
257Content-Signal: ai-train=no, search=yes
258Disallow: /admin$
259Disallow: /admin/
260Disallow: /admin?
261Disallow: /s/about/trust-center/report
262
263# Perplexity
264User-agent: PerplexityBot
265Content-Signal: ai-train=no, search=yes
266Disallow: /admin$
267Disallow: /admin/
268Disallow: /admin?
269Disallow: /s/about/trust-center/report
270
271User-agent: Perplexity-User
272Content-Signal: ai-train=no, search=yes
273Disallow: /admin$
274Disallow: /admin/
275Disallow: /admin?
276Disallow: /s/about/trust-center/report
277
278# Mistral
279User-agent: MistralAI-User
280Content-Signal: ai-train=no, search=yes
281Disallow: /admin$
282Disallow: /admin/
283Disallow: /admin?
284Disallow: /s/about/trust-center/report
285
286User-agent: MistralAI-User/1.0
287Content-Signal: ai-train=no, search=yes
288Disallow: /admin$
289Disallow: /admin/
290Disallow: /admin?
291Disallow: /s/about/trust-center/report
292
293# Meta
294User-agent: Meta-ExternalFetcher
295Content-Signal: ai-train=no, search=yes
296Disallow: /admin$
297Disallow: /admin/
298Disallow: /admin?
299Disallow: /s/about/trust-center/report
300
301# Google
302# Google-Extended controls use of content for Gemini training and grounding (Gemini Apps + Vertex AI)
303User-agent: Google-Extended
304Allow: /
305
306User-agent: Google
307Content-Signal: ai-train=no, search=yes
308Disallow: /admin$
309Disallow: /admin/
310Disallow: /admin?
311Disallow: /s/about/trust-center/report
312
313User-agent: Google-NotebookLM
314Content-Signal: ai-train=no, search=yes
315Disallow: /admin$
316Disallow: /admin/
317Disallow: /admin?
318Disallow: /s/about/trust-center/report
319
320User-agent: NotebookLM
321Content-Signal: ai-train=no, search=yes
322Disallow: /admin$
323Disallow: /admin/
324Disallow: /admin?
325Disallow: /s/about/trust-center/report
326
327User-agent: Gemini-Deep-Research
328Content-Signal: ai-train=no, search=yes
329Disallow: /admin$
330Disallow: /admin/
331Disallow: /admin?
332Disallow: /s/about/trust-center/report
333
334User-agent: GoogleAgent-Mariner
335Content-Signal: ai-train=no, search=yes
336Disallow: /admin$
337Disallow: /admin/
338Disallow: /admin?
339Disallow: /s/about/trust-center/report
340
341User-agent: Google-CloudVertexBot
342Content-Signal: ai-train=no, search=yes
343Disallow: /admin$
344Disallow: /admin/
345Disallow: /admin?
346Disallow: /s/about/trust-center/report
347
348# Amazon
349User-agent: Amazonbot
350Content-Signal: ai-train=no, search=yes
351Disallow: /admin$
352Disallow: /admin/
353Disallow: /admin?
354Disallow: /s/about/trust-center/report
355
356User-agent: amazon-kendra
357Content-Signal: ai-train=no, search=yes
358Disallow: /admin$
359Disallow: /admin/
360Disallow: /admin?
361Disallow: /s/about/trust-center/report
362
363User-agent: AmazonBuyForMe
364Content-Signal: ai-train=no, search=yes
365Disallow: /admin$
366Disallow: /admin/
367Disallow: /admin?
368Disallow: /s/about/trust-center/report
369
370User-agent: NovaAct
371Content-Signal: ai-train=no, search=yes
372Disallow: /admin$
373Disallow: /admin/
374Disallow: /admin?
375Disallow: /s/about/trust-center/report
376
377# Misc AI Bots
378User-agent: CCBot
379Content-Signal: ai-train=no, search=yes
380Disallow: /admin$
381Disallow: /admin/
382Disallow: /admin?
383Disallow: /s/about/trust-center/report
384
385User-agent: atlassian-bot
386Content-Signal: ai-train=no, search=yes
387Disallow: /admin$
388Disallow: /admin/
389Disallow: /admin?
390Disallow: /s/about/trust-center/report
391
392User-agent: Cloudflare-AutoRAG
393Content-Signal: ai-train=no, search=yes
394Disallow: /admin$
395Disallow: /admin/
396Disallow: /admin?
397Disallow: /s/about/trust-center/report
398
399User-agent: KlaviyoAIBot
400Content-Signal: ai-train=no, search=yes
401Disallow: /admin$
402Disallow: /admin/
403Disallow: /admin?
404Disallow: /s/about/trust-center/report
405
406User-agent: ShapBot
407Content-Signal: ai-train=no, search=yes
408Disallow: /admin$
409Disallow: /admin/
410Disallow: /admin?
411Disallow: /s/about/trust-center/report
412
413User-agent: Bravebot
414Content-Signal: ai-train=no, search=yes
415Disallow: /admin$
416Disallow: /admin/
417Disallow: /admin?
418Disallow: /s/about/trust-center/report
419
420User-agent: DuckAssistBot
421Content-Signal: ai-train=no, search=yes
422Disallow: /admin$
423Disallow: /admin/
424Disallow: /admin?
425Disallow: /s/about/trust-center/report
426
427User-agent: Devin
428Content-Signal: ai-train=no, search=yes
429Disallow: /admin$
430Disallow: /admin/
431Disallow: /admin?
432Disallow: /s/about/trust-center/report
433
434User-agent: AhrefsBot
435Content-Signal: ai-train=no, search=yes
436Disallow: /admin$
437Disallow: /admin/
438Disallow: /admin?
439Disallow: /s/about/trust-center/report
440
441# Fallback, disallow scraping
442User-agent: *
443Disallow: /
444
445Sitemap: https://linktr.ee/discover/profile-directory/sitemapindex.xml
446Sitemap: https://linktr.ee/marketplace/sitemap.xml
447Sitemap: https://linktr.ee/s/wf/sitemap.xml
448Sitemap: https://linktr.ee/s/templates/sitemap.xml
449Sitemap: https://linktr.ee/help/sitemap.xml
450
What do the bots actually get?
This page reads robots.txt, which states what linktr.ee asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on linktr.ee Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does linktr.ee block GPTBot?
Partly. linktr.ee’s robots.txt keeps GPTBot out of some paths, such as /admin$, and lets it crawl the rest. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does linktr.ee block ClaudeBot?
Partly. linktr.ee’s robots.txt keeps ClaudeBot out of some paths, such as /admin$, and lets it crawl the rest. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does linktr.ee block Google-Extended?
No. linktr.ee’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does linktr.ee block PerplexityBot?
Partly. linktr.ee’s robots.txt keeps PerplexityBot out of some paths, such as /admin$, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find linktr.ee?
Partly. linktr.ee’s robots.txt keeps OAI-SearchBot out of some paths, such as /admin$, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.