Who blocks AI · #791 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does about.me block AI crawlers?
No. about.me’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- about.me/robots.txt · 347 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Blocked | line 19 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View about.me/robots.txt · 347 lines
1User-agent: *
2Disallow: /facebook/
3Disallow: /twitter/
4Disallow: /linkedin/
5Disallow: /random/
6Disallow: /content/
7Disallow: /n/
8Disallow: /ajax/
9Disallow: /me/
10Disallow: /dw/
11
12
13Sitemap: https://aboutme-public.s3.amazonaws.com/sitemap/SitemapIndex.xml
14
15User-agent: Twitterbot
16Allow: /s3/
17
18User-agent: SemrushBot
19Disallow: /
20
21User-agent: SemrushBot-SA
22Disallow: /
23
24User-agent: asterias
25Disallow: /
26
27User-agent: BackDoorBot/1.0
28Disallow: /
29
30User-agent: Black Hole
31Disallow: /
32
33User-agent: BlowFish/1.0
34Disallow: /
35
36User-agent: BotALot
37Disallow: /
38
39User-agent: BuiltBotTough
40Disallow: /
41
42User-agent: Bullseye/1.0
43Disallow: /
44
45User-agent: BunnySlippers
46Disallow: /
47
48User-agent: Cegbfeieh
49Disallow: /
50
51User-agent: CheeseBot
52Disallow: /
53
54User-agent: CherryPicker
55Disallow: /
56
57User-agent: CherryPickerElite/1.0
58Disallow: /
59
60User-agent: CherryPickerSE/1.0
61Disallow: /
62
63User-agent: CopyRightCheck
64Disallow: /
65
66User-agent: cosmos
67Disallow: /
68
69User-agent: Crescent
70Disallow: /
71
72User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
73Disallow: /
74
75User-agent: DittoSpyder
76Disallow: /
77
78User-agent: EmailCollector
79Disallow: /
80
81User-agent: EmailSiphon
82Disallow: /
83
84User-agent: EmailWolf
85Disallow: /
86
87User-agent: EroCrawler
88Disallow: /
89
90User-agent: ExtractorPro
91Disallow: /
92
93User-agent: Foobot
94Disallow: /
95
96User-agent: Harvest/1.5
97Disallow: /
98
99User-agent: hloader
100Disallow: /
101
102User-agent: httplib
103Disallow: /
104
105User-agent: humanlinks
106Disallow: /
107
108User-agent: InfoNaviRobot
109Disallow: /
110
111User-agent: JennyBot
112Disallow: /
113
114User-agent: Kenjin Spider
115Disallow: /
116
117User-agent: Keyword Density/0.9
118Disallow: /
119
120User-agent: LexiBot
121Disallow: /
122
123User-agent: libWeb/clsHTTP
124Disallow: /
125
126User-agent: LinkextractorPro
127Disallow: /
128
129User-agent: LinkScan/8.1a Unix
130Disallow: /
131
132User-agent: LinkWalker
133Disallow: /
134
135User-agent: LNSpiderguy
136Disallow: /
137
138User-agent: lwp-trivial
139Disallow: /
140
141User-agent: lwp-trivial/1.34
142Disallow: /
143
144User-agent: Mata Hari
145Disallow: /
146
147User-agent: Microsoft URL Control - 5.01.4511
148Disallow: /
149
150User-agent: Microsoft URL Control - 6.00.8169
151Disallow: /
152
153User-agent: MIIxpc
154Disallow: /
155
156User-agent: MIIxpc/4.2
157Disallow: /
158
159User-agent: Mister PiX
160Disallow: /
161
162User-agent: moget
163Disallow: /
164
165User-agent: moget/2.1
166Disallow: /
167
168User-agent: mozilla/4
169Disallow: /
170
171User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
172Disallow: /
173
174User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
175Disallow: /
176
177User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
178Disallow: /
179
180User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
181Disallow: /
182
183User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
184Disallow: /
185
186User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
187Disallow: /
188
189User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
190Disallow: /
191
192User-agent: mozilla/5
193Disallow: /
194
195User-agent: NetAnts
196Disallow: /
197
198User-agent: NICErsPRO
199Disallow: /
200
201User-agent: Offline Explorer
202Disallow: /
203
204User-agent: Openfind
205Disallow: /
206
207User-agent: Openfind data gathere
208Disallow: /
209
210User-agent: ProPowerBot/2.14
211Disallow: /
212
213User-agent: ProWebWalker
214Disallow: /
215
216User-agent: QueryN Metasearch
217Disallow: /
218
219User-agent: RepoMonkey
220Disallow: /
221
222User-agent: RepoMonkey Bait & Tackle/v1.01
223Disallow: /
224
225User-agent: RMA
226Disallow: /
227
228User-agent: SiteSnagger
229Disallow: /
230
231User-agent: SpankBot
232Disallow: /
233
234User-agent: spanner
235Disallow: /
236
237User-agent: suzuran
238Disallow: /
239
240User-agent: Szukacz/1.4
241Disallow: /
242
243User-agent: Teleport
244Disallow: /
245
246User-agent: TeleportPro
247Disallow: /
248
249User-agent: Telesoft
250Disallow: /
251
252User-agent: The Intraformant
253Disallow: /
254
255User-agent: TheNomad
256Disallow: /
257
258User-agent: TightTwatBot
259Disallow: /
260
261User-agent: Titan
262Disallow: /
263
264User-agent: toCrawl/UrlDispatcher
265Disallow: /
266
267User-agent: True_Robot
268Disallow: /
269
270User-agent: True_Robot/1.0
271Disallow: /
272
273User-agent: turingos
274Disallow: /
275
276User-agent: URLy Warning
277Disallow: /
278
279User-agent: VCI
280Disallow: /
281
282User-agent: VCI WebViewer VCI WebViewer Win32
283Disallow: /
284
285User-agent: Web Image Collector
286Disallow: /
287
288User-agent: WebAuto
289Disallow: /
290
291User-agent: WebBandit
292Disallow: /
293
294User-agent: WebBandit/3.50
295Disallow: /
296
297User-agent: WebCopier
298Disallow: /
299
300User-agent: WebEnhancer
301Disallow: /
302
303User-agent: WebmasterWorldForumBot
304Disallow: /
305
306User-agent: WebSauger
307Disallow: /
308
309User-agent: Website Quester
310Disallow: /
311
312User-agent: Webster Pro
313Disallow: /
314
315User-agent: WebStripper
316Disallow: /
317
318User-agent: WebZip
319Disallow: /
320
321User-agent: WebZip/4.0
322Disallow: /
323
324User-agent: Wget
325Disallow: /
326
327User-agent: Wget/1.5.3
328Disallow: /
329
330User-agent: Wget/1.6
331Disallow: /
332
333User-agent: WWW-Collector-E
334Disallow: /
335
336User-agent: Xenu's
337Disallow: /
338
339User-agent: Xenu's Link Sleuth 1.1c
340Disallow: /
341
342User-agent: Zeus
343Disallow: /
344
345User-agent: Zeus 32297 Webster Pro V2.9 Win32
346Disallow: /
347
What do the bots actually get?
This page reads robots.txt, which states what about.me asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on about.me Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does about.me block GPTBot?
No. about.me’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does about.me block ClaudeBot?
No. about.me’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does about.me block Google-Extended?
No. about.me’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does about.me block PerplexityBot?
No. about.me’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find about.me?
Yes. about.me’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.