Who blocks AI · #1280 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does bleepingcomputer.com block AI crawlers?
Yes. bleepingcomputer.com’s robots.txt blocks 3 of the 19 AI crawlers we track, including ClaudeBot, CCBot and Applebot-Extended. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotBlocked- robots.txt
- www.bleepingcomputer.com/robots.txt · 145 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | line 141 · Allow: / under User-agent: * | |
Bingbot | Microsoft | Allowed | line 141 · Allow: / under User-agent: * |
DuckDuckBot | DuckDuckGo | Allowed | line 141 · Allow: / under User-agent: * |
Applebot | Apple | Allowed | line 141 · Allow: / under User-agent: * |
PetalBot | Huawei | Allowed | line 141 · Allow: / under User-agent: * |
YandexBot | Yandex | Allowed | line 141 · Allow: / under User-agent: * |
Baiduspider | Baidu | Allowed | line 141 · Allow: / under User-agent: * |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | line 141 · Allow: / under User-agent: * |
PerplexityBot | Perplexity | Allowed | line 141 · Allow: / under User-agent: * |
Claude-SearchBot | Anthropic | Allowed | line 141 · Allow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Allowed | line 141 · Allow: / under User-agent: * |
Amazonbot | Amazon | Allowed | line 141 · Allow: / under User-agent: * |
YouBot | You.com | Allowed | line 141 · Allow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | line 141 · Allow: / under User-agent: * |
ClaudeBot | Anthropic | Blocked | line 76 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Allowed | line 141 · Allow: / under User-agent: * | |
CCBot | Common Crawl | Blocked | line 40 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 6 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | line 141 · Allow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Allowed | line 141 · Allow: / under User-agent: * |
Bytespider | ByteDance | Allowed | line 141 · Allow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | line 141 · Allow: / under User-agent: * |
Claude-User | Anthropic | Allowed | line 141 · Allow: / under User-agent: * |
Perplexity-User | Perplexity | Allowed | line 141 · Allow: / under User-agent: * |
MistralAI-User | Mistral | Allowed | line 141 · Allow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Allowed | line 141 · Allow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 19 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Allowed | line 141 · Allow: / under User-agent: * |
MJ12bot | Majestic | Allowed | line 141 · Allow: / under User-agent: * |
DotBot | Moz | Blocked | line 31 · Disallow: / under User-agent: DotBot/1.1 |
View www.bleepingcomputer.com/robots.txt · 145 lines
1Sitemap: http://www.bleepingcomputer.com/sitemap_index.xml.gz
2User-agent: Slurp
3Crawl-delay: 2
4
5User-agent: Applebot-Extended
6Disallow: /
7
8User-agent: SeekportBot
9Crawl-delay: 10
10
11User-agent: AhrefsBot
12Crawl-Delay: 5
13
14User-agent: StractBot
15Crawl-Delay: 5
16
17User-agent: AwarioRssBot
18User-agent: AwarioSmartBot
19Disallow: /
20
21User-agent: ias_crawler
22Crawl-delay: 10
23
24User-Agent: MJ12bot
25Crawl-Delay: 10
26
27User-agent: Cliqzbot
28Disallow: /
29
30User-agent: DotBot/1.1
31Disallow: /
32
33User-agent: MauiBot
34Disallow: /
35
36User-agent: NTENTbot
37Disallow: /
38
39User-agent: CCBot
40Disallow: /
41
42User-agent: sentibot
43Disallow: /
44
45User-Agent: trendictionbot
46Disallow: /
47
48User-Agent: yacybot
49Disallow: /
50
51User-Agent: iSec_Bot
52Disallow: /
53
54User-Agent: DotBot/1.1
55Disallow: /
56
57User-agent: BLEXBot
58Disallow: /
59
60User-Agent: AhrefsBot/6.1
61Disallow: /
62
63User-Agent: Qwantify/Bleriot/1.1
64Disallow: /
65
66User-agent: BLEXBot
67Disallow: /
68
69User-agent: DemandbasePublisherAnalyzer/0.1
70Disallow: /
71
72User-agent: weborama-fetcher
73Disallow: /
74
75User-agent: ClaudeBot
76Disallow: /
77
78User-agent: *
79Crawl-delay: 1
80Disallow: /cgi-bin/
81Disallow: /forums/style_images/
82Disallow: /forums/index.php?act=Search
83Disallow: /forums/index.php?act=Login
84Disallow: /forums/index.php?act=Reg
85Disallow: /forums/index.php?act=calendar
86Disallow: /forums/index.php?act=Msg
87Disallow: /forums/index.php?act=Mail
88Disallow: /forums/index.php?act=Forward
89Disallow: /forums/index.php?act=forward
90Disallow: /forums/index.php?act=Track
91Disallow: /forums/index.php?act=Post
92Disallow: /forums/index.php?act=post
93Disallow: /forums/index.php?showuser
94Disallow: /forums/index.php?act=Print
95Disallow: /forums/index.php?act=ST
96Disallow: /forums/index.php?act=UserCP
97Disallow: /forums/index.php?act=usercp
98Disallow: /forums/index.php?act=Arcade
99Disallow: /forums/index.php?act=findpost
100Disallow: /forums/index.php?act=SF
101Disallow: /forums/index.php?act=Stats
102Disallow: /forums/index.php?&act=Online
103Disallow: /forums/index.php?act=Members
104Disallow: /forums/index.php?act=Help
105Disallow: /forums/index.php?act=Profile
106Disallow: /forums/index.php?act=profile
107Disallow: /forums/index.php?act=Attach
108Disallow: /forums/index.php?act=attach
109Disallow: /forums/index.php?app=core&attach_id
110Disallow: /forums/index.php?app=core&module=attach
111Disallow: /forums/index.php?act=Print
112Disallow: /forums/index.php?act=print
113Disallow: /forums/index.php?app=core&module=global§ion=login&do=deleteCookies&k=*
114Disallow: /forums/index.php?app=core&module=global§ion=login&do=logout*
115Disallow: /forums/index.php?app=forums&module=forums§ion=markasread&marktype=all&k=*
116Disallow: /forums/index.php?app=forums&module=moderate§ion=moderate&f=*
117Disallow: /forums/index.php?app=core&module=search*
118Disallow: /forums/index.php?sharelink=*
119Disallow: /forums/u/*
120Disallow: /forums/user-*
121Disallow: /forums/*&view=findpost*
122Disallow: /forums/*&view=old
123Disallow: /forums/*&view=new
124Disallow: /forums/*&mode=linear
125Disallow: /forums/*&mode=linearplus
126Disallow: /forums/*&mode=threaded
127Disallow: /forums/*&mode=threaded
128Disallow: /forums/*&mode=linearplus
129Disallow: /forums/*&mode=linear
130Disallow: /forums/help.html
131Disallow: /forums/lastpost
132Disallow: /resources/link
133Disallow: /tutorials/index.php?act=print
134Disallow: /tutorials/index.php?act=friend
135Disallow: /logreply/
136Disallow: /download/search/*
137Disallow: /forums/interface/twitter/*
138Disallow: /forums/index.php?andor_type=&app=core&do=search*
139Disallow: /forums/index.php?app=core&module=global*
140Allow: /ads.txt
141Allow: /
142
143User-agent: AmazonAdBot
144Allow: /
145
What do the bots actually get?
This page reads robots.txt, which states what bleepingcomputer.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on bleepingcomputer.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does bleepingcomputer.com block GPTBot?
No. bleepingcomputer.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does bleepingcomputer.com block ClaudeBot?
Yes. bleepingcomputer.com’s robots.txt disallows ClaudeBot from the whole site (line 76: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does bleepingcomputer.com block Google-Extended?
No. bleepingcomputer.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does bleepingcomputer.com block PerplexityBot?
No. bleepingcomputer.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find bleepingcomputer.com?
Yes. bleepingcomputer.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.