Who blocks AI · #356 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does healthline.com block AI crawlers?
Yes. healthline.com’s robots.txt blocks 7 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, CCBot and 4 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotBlocked- robots.txt
- www.healthline.com/robots.txt · 567 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Blocked | line 75 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 523 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 538 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Blocked | line 535 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 526 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 547 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 532 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.healthline.com/robots.txt · 567 lines
1User-agent: *
2Crawl-delay: 5
3Allow: /corporate/microsites/tysabri/ms-assessment-tool.html
4Allow: /corporate/microsites/tysabri/ms-assessment-tool-page1.html
5Allow: /corporate/microsites/tysabri/ms-assessment-tool-page2.html
6Allow: /corporate/microsites/tysabri/ms-assessment-tool-page3.html
7Allow: /corporate/microsites/tysabri/ms-assessment-tool-page4.html
8Allow: /corporate/microsites/tysabri/ms-assessment-tool-page5.html
9Allow: /corporate/microsites/tysabri/ms-assessment-tool-page6.html
10Allow: /corporate/microsites/tysabri/ms-assessment-tool-page7.html
11Allow: /corporate/microsites/tysabri/ms-assessment-tool-pageresults.html
12Allow: /ads.txt
13Disallow: /bmajax/
14Disallow: /v2/
15Disallow: /vp/
16Disallow: /corporate/
17Disallow: /health/mirgraine-headaches
18Disallow: /health/wp-*
19Disallow: /nutrition/wp-*
20Disallow: /program/wp-*
21Disallow: /health-news/wp-*
22Disallow: /health/*/wp-*
23Disallow: /nutrition/*/wp-*
24Disallow: /health-news/*/wp-*
25Disallow: /cdn.jwplayer.com/previews/*
26Disallow: /healthy/
27
28# Internal Frontend Test Pages - Not for Public Indexing
29Disallow: /health/sponsored-article-test-do-not-edit-this-ever
30Disallow: /program/hms-test-program-page
31Disallow: /health/QA-Tabbed-ManyTabs-21
32Disallow: /health/post-block-widget-sampler
33Disallow: /health/test-reference-article-for-front-end-tests
34Disallow: /health/wp-2485828
35Disallow: /health/this-is-an-infinite-scroll-test-page-please-do-not-modify-this-page-without-checking-with-frontend-site-teams
36Disallow: /health/hm-test-17-proven-tips-to-sleep-better-at-night
37Disallow: /healthy/hm-test-page-2-17-proven-tips-to-sleep-better-at-night
38Disallow: /healthy/hm-test-page-3-17-proven-tips-to-sleep-better-at-night
39Disallow: /healthy/hm-test-page-4-diabetes-and-sweets
40Disallow: /healthy/hm-test-page-5-diabetes-and-sweets
41Disallow: /health/drugs/hm-test-all-about-humira
42Disallow: /health/hm-test-feature-template-for-sponsorship
43Disallow: /this-is-a-test-page
44
45# Invalid URLs
46Disallow: */null$
47Disallow: */inline$
48
49# Static Test Articles
50Disallow: /test/
51
52# Media Revenue Test Pages
53Disallow: /*/hm-test
54
55# API Routes
56Disallow: /api/
57Allow: /api/auth/signup
58
59Disallow: /hlcmsresource/*.txt$
60Disallow: /hlcmsresource/*.xml$
61
62User-agent: Mediapartners-Google
63Disallow:
64
65User-agent: Adsbot-Google
66Disallow:
67
68# --- Amazon / AWS crawlers (begin) ---
69
70# AmazonAdbot is intentionally allowed (ad targeting — see HLMR-6504)
71User-agent: AmazonAdbot
72Disallow:
73
74User-agent: Amazonbot
75Disallow: /
76
77User-agent: Amzn-SearchBot
78Disallow: /
79
80User-agent: Amzn-User
81Disallow: /
82
83User-agent: amazon-kendra
84Disallow: /
85
86User-agent: amazon-QBusiness
87Disallow: /
88
89User-agent: bedrockbot
90Disallow: /
91
92User-agent: aws-quick-on-behalf-of-
93Disallow: /
94
95User-agent: AmazonProductDiscoverybot
96Disallow: /
97
98User-agent: AmazonBuyForMe
99Disallow: /
100
101User-agent: AmazonSellerInitiatedListing
102Disallow: /
103
104User-agent: NovaAct
105Disallow: /
106
107# --- Amazon / AWS crawlers (end) ---
108
109User-agent: Twitterbot
110Disallow:
111
112User-agent: NimbleCrawler
113Disallow: /
114
115User-agent: grapeshot
116Disallow:
117
118User-agent: BotRightHere
119Disallow: /
120
121User-agent: WebZip
122Disallow: /
123
124User-agent: larbin
125Disallow: /
126
127User-agent: b2w/0.1
128Disallow: /
129
130User-agent: Copernic
131Disallow: /
132
133User-agent: psbot
134Disallow: /
135
136User-agent: Python-urllib
137Disallow: /
138
139User-agent: URL_Spider_Pro
140Disallow: /
141
142User-agent: CherryPicker
143Disallow: /
144
145User-agent: EmailCollector
146Disallow: /
147
148User-agent: EmailSiphon
149Disallow: /
150
151User-agent: WebBandit
152Disallow: /
153
154User-agent: EmailWolf
155Disallow: /
156
157User-agent: ExtractorPro
158Disallow: /
159
160User-agent: CopyRightCheck
161Disallow: /
162
163User-agent: Crescent
164Disallow: /
165
166User-agent: SiteSnagger
167Disallow: /
168
169User-agent: ProWebWalker
170Disallow: /
171
172User-agent: CheeseBot
173Disallow: /
174
175User-agent: LNSpiderguy
176Disallow: /
177
178User-agent: Alexibot
179Disallow: /
180
181User-agent: Teleport
182Disallow: /
183
184User-agent: TeleportPro
185Disallow: /
186
187User-agent: MIIxpc
188Disallow: /
189
190User-agent: Telesoft
191Disallow: /
192
193User-agent: Website Quester
194Disallow: /
195
196User-agent: WebZip
197Disallow: /
198
199User-agent: moget/2.1
200Disallow: /
201
202User-agent: WebZip/4.0
203Disallow: /
204
205User-agent: WebStripper
206Disallow: /
207
208User-agent: WebSauger
209Disallow: /
210
211User-agent: WebCopier
212Disallow: /
213
214User-agent: NetAnts
215Disallow: /
216
217User-agent: Mister PiX
218Disallow: /
219
220User-agent: WebAuto
221Disallow: /
222
223User-agent: TheNomad
224Disallow: /
225
226User-agent: WWW-Collector-E
227Disallow: /
228
229User-agent: RMA
230Disallow: /
231
232User-agent: libWeb/clsHTTP
233Disallow: /
234
235User-agent: asterias
236Disallow: /
237
238User-agent: httplib
239Disallow: /
240
241User-agent: turingos
242Disallow: /
243
244User-agent: spanner
245Disallow: /
246
247User-agent: InfoNaviRobot
248Disallow: /
249
250User-agent: Harvest/1.5
251Disallow: /
252
253User-agent: Bullseye/1.0
254Disallow: /
255
256User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
257Disallow: /
258
259User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
260Disallow: /
261
262User-agent: CherryPickerSE/1.0
263Disallow: /
264
265User-agent: CherryPickerElite/1.0
266Disallow: /
267
268User-agent: WebBandit/3.50
269Disallow: /
270
271User-agent: NICErsPRO
272Disallow: /
273
274User-agent: Microsoft URL Control - 5.01.4511
275Disallow: /
276
277User-agent: DittoSpyder
278Disallow: /
279
280User-agent: Foobot
281Disallow: /
282
283User-agent: SpankBot
284Disallow: /
285
286User-agent: BotALot
287Disallow: /
288
289User-agent: lwp-trivial/1.34
290Disallow: /
291
292User-agent: lwp-trivial
293Disallow: /
294
295User-agent: BunnySlippers
296Disallow: /
297
298User-agent: Microsoft URL Control - 6.00.8169
299Disallow: /
300
301User-agent: URLy Warning
302Disallow: /
303
304User-agent: Wget/1.6
305Disallow: /
306
307User-agent: Wget/1.5.3
308Disallow: /
309
310User-agent: Wget
311Disallow: /
312
313User-agent: LinkWalker
314Disallow: /
315
316User-agent: cosmos
317Disallow: /
318
319User-agent: moget
320Disallow: /
321
322User-agent: hloader
323Disallow: /
324
325User-agent: humanlinks
326Disallow: /
327
328User-agent: LinkextractorPro
329Disallow: /
330
331User-agent: Offline Explorer
332Disallow: /
333
334User-agent: Mata Hari
335Disallow: /
336
337User-agent: LexiBot
338Disallow: /
339
340User-agent: Web Image Collector
341Disallow: /
342
343User-agent: The Intraformant
344Disallow: /
345
346User-agent: True_Robot/1.0
347Disallow: /
348
349User-agent: True_Robot
350Disallow: /
351
352User-agent: BlowFish/1.0
353Disallow: /
354
355User-agent: JennyBot
356Disallow: /
357
358User-agent: MIIxpc/4.2
359Disallow: /
360
361User-agent: BuiltBotTough
362Disallow: /
363
364User-agent: ProPowerBot/2.14
365Disallow: /
366
367User-agent: BackDoorBot/1.0
368Disallow: /
369
370User-agent: toCrawl/UrlDispatcher
371Disallow: /
372
373User-agent: WebEnhancer
374Disallow: /
375
376User-agent: suzuran
377Disallow: /
378
379User-agent: TightTwatBot
380Disallow: /
381
382User-agent: VCI WebViewer VCI WebViewer Win32
383Disallow: /
384
385User-agent: VCI
386Disallow: /
387
388User-agent: Szukacz/1.4
389Disallow: /
390
391User-agent: QueryN Metasearch
392Disallow: /
393
394User-agent: Openfind data gatherer
395Disallow: /
396
397User-agent: Openfind
398Disallow: /
399
400User-agent: Zeus
401Disallow: /
402
403User-agent: RepoMonkey Bait & Tackle/v1.01
404Disallow: /
405
406User-agent: RepoMonkey
407Disallow: /
408
409User-agent: Microsoft URL Control
410Disallow: /
411
412User-agent: Openbot
413Disallow: /
414
415User-agent: URL Control
416Disallow: /
417
418User-agent: Zeus Link Scout
419Disallow: /
420
421User-agent: Zeus 32297 Webster Pro V2.9 Win32
422Disallow: /
423
424User-agent: Webster Pro
425Disallow: /
426
427User-agent: EroCrawler
428Disallow: /
429
430User-agent: LinkScan/8.1a Unix
431Disallow: /
432
433User-agent: Keyword Density/0.9
434Disallow: /
435
436User-agent: Kenjin Spider
437Disallow: /
438
439User-agent: Iron33/1.0.2
440Disallow: /
441
442User-agent: GetRight/4.2
443Disallow: /
444
445User-agent: FairAd Client
446Disallow: /
447
448User-agent: Gaisbot
449Disallow: /
450
451User-agent: Aqua_Products
452Disallow: /
453
454User-agent: Radiation Retriever 1.1
455Disallow: /
456
457User-agent: Flaming AttackBot
458Disallow: /
459
460User-agent: Oracle Ultra Search
461Disallow: /
462
463User-agent: MSIECrawler
464Disallow: /
465
466User-agent: PerMan
467Disallow: /
468
469User-agent: searchpreview
470Disallow: /
471
472User-agent: TurnitinBot
473Disallow: /
474
475User-agent: wget
476Disallow: /
477
478User-agent: ExtractorPro
479Disallow: /
480
481User-agent: WebZIP/4.21
482Disallow: /
483
484User-agent: WebZIP/5.0
485Disallow: /
486
487User-agent: HTTrack 3.0
488Disallow: /
489
490User-agent: TurnitinBot/1.5
491Disallow: /
492
493User-agent: WebCopier v3.2a
494Disallow: /
495
496User-agent: WebCapture 2.0
497Disallow: /
498
499User-agent: WebCopier v.2.2
500Disallow: /
501
502User-agent: Callpod Keeper
503Disallow: /
504
505User-agent: Go-http-client
506Disallow: /
507
508User-agent: Java Browser
509Disallow: /
510
511User-agent: PHP
512Disallow: /
513
514User-agent: Clickagy Intelligence
515Disallow: /
516
517User-agent: uipbot
518Disallow: /
519
520User-agent: GPTBot
521Allow: /brand/
522Allow: /health/drugs/ozempic-for-weight-loss
523Disallow: /
524
525User-agent: Applebot-Extended
526Disallow: /
527
528User-agent: anthropic-ai
529Disallow: /
530
531User-agent: Bytespider
532Disallow: /
533
534User-agent: CCBot
535Disallow: /
536
537User-agent: ClaudeBot
538Disallow: /
539
540User-agent: Diffbot
541Disallow: /
542
543User-agent: FacebookBot
544Disallow: /
545
546User-agent: Meta-ExternalAgent
547Disallow: /
548
549User-agent: omgili
550Disallow: /
551
552User-agent: Timpibot
553Disallow: /
554
555User-agent: Webzio-Extended
556Disallow: /
557
558User-agent: DDM-DCipher/1.0.7
559Disallow: /
560
561Sitemap: https://www.healthline.com/sitemap.xml
562Sitemap: https://www.healthline.com/hlcms-articles-99.xml
563
564Disallow /account/*
565
566Disallow: /*?*
567
What do the bots actually get?
This page reads robots.txt, which states what healthline.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on healthline.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does healthline.com block GPTBot?
Yes. healthline.com’s robots.txt disallows GPTBot from the whole site (line 523: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does healthline.com block ClaudeBot?
Yes. healthline.com’s robots.txt disallows ClaudeBot from the whole site (line 538: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does healthline.com block Google-Extended?
No. healthline.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does healthline.com block PerplexityBot?
No. healthline.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find healthline.com?
Yes. healthline.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.