Who blocks AI · #1074 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does computerworld.com block AI crawlers?
Yes. computerworld.com’s robots.txt blocks 6 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, CCBot and 3 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotBlocked- robots.txt
- www.computerworld.com/robots.txt · 160 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 93 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 66 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Blocked | line 63 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 45 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 120 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 60 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Blocked | line 78 · Disallow: / under User-agent: dotbot |
View www.computerworld.com/robots.txt · 160 lines
1# ------------------------------------------------------
2# Global Rules
3# ------------------------------------------------------
4
5User-agent: *
6Disallow: /search*
7Disallow: */filter/*
8Disallow: /*?utm_*
9Disallow: /*//*
10Disallow: /wp-admin/
11Disallow: /wp-login.php
12Disallow: /wp-register.php
13Disallow: /wp-includes/
14Disallow: /xmlrpc.php
15Disallow: /wp-json/
16Disallow: /feed/
17Disallow: /*/feed/
18Disallow: /comments/feed/
19Disallow: /*LICENSE.txt$
20Disallow: /*README.md$
21Disallow: /*README.txt$
22Disallow: /*license.txt$
23Disallow: /*readme.html$
24Disallow: /*readme.md$
25Disallow: /*readme.txt$
26Disallow: /*package.json$
27Disallow: /*composer.json$
28Disallow: /*composer.lock$
29Disallow: /*yarn.lock$
30
31# ------------------------------------------------------
32# Unwanted Bots
33# ------------------------------------------------------
34
35User-agent: Ai2Bot-Dolma
36Disallow: /
37
38User-agent: AI2Bot
39Disallow: /
40
41User-agent: anthropic-ai
42Disallow: /
43
44User-agent: Applebot-Extended
45Disallow: /
46
47User-agent: Arquivo-web-crawler
48Disallow: /
49
50User-agent: AwarioRssBot
51Disallow: /
52
53User-agent: AwarioSmartBot
54Disallow: /
55
56User-agent: Barkrowler
57Disallow: /
58
59User-agent: Bytespider
60Disallow: /
61
62User-agent: CCBot
63Disallow: /
64
65User-agent: ClaudeBot
66Disallow: /
67
68User-agent: cohere-ai
69Disallow: /
70
71User-agent: cohere-training-data-crawler
72Disallow: /
73
74User-agent: crawler4j
75Disallow: /
76
77User-agent: dotbot
78Disallow: /
79
80User-agent: Diffbot
81Disallow: /
82
83User-agent: FacebookBot
84Disallow: /
85
86User-agent: FriendlyCrawler
87Disallow: /
88
89User-agent: Google-CloudVertexBot
90Disallow: /
91
92User-agent: GPTBot
93Disallow: /
94
95User-agent: https://hada.news
96Disallow: /
97
98User-agent: ICC-Crawler
99Disallow: /
100
101User-agent: ImagesiftBot
102Disallow: /
103
104User-agent: img2dataset
105Disallow: /
106
107User-agent: ISSCyberRiskCrawler
108Disallow: /
109
110User-agent: JenkersBot
111Disallow: /
112
113User-agent: Kangaroo Bot
114Disallow: /
115
116User-agent: Meltwater
117Disallow: /
118
119User-agent: Meta-ExternalAgent
120Disallow: /
121
122User-agent: MJ12bot
123Disallow: /
124
125User-agent: PanguBot
126Disallow: /
127
128User-agent: Scrapy
129Disallow: /
130
131User-agent: Seekr
132Disallow: /
133
134User-agent: Timpibot
135Disallow: /
136
137User-agent: VelenPublicWebCrawler
138Disallow: /
139
140User-agent: Webzio-Extended
141Disallow: /
142
143# ------------------------------------------------------
144# Sitemaps
145# ------------------------------------------------------
146
147Sitemap: https://www.computerworld.com/sitemap-index.xml
148Sitemap: https://www.computerworld.com/sitemap-news-en.xml
149Sitemap: https://www.computerworld.com/sitemap-articles-en.xml
150Sitemap: https://www.computerworld.com/sitemap-video-episodes-en.xml
151Sitemap: https://www.computerworld.com/sitemap-video-series.xml
152Sitemap: https://www.computerworld.com/sitemap-podcast-episodes-en.xml
153Sitemap: https://www.computerworld.com/sitemap-podcast-series.xml
154Sitemap: https://www.computerworld.com/sitemap-blog-series.xml
155Sitemap: https://www.computerworld.com/sitemap-pages.xml
156Sitemap: https://www.computerworld.com/sitemap-pages-au.xml
157Sitemap: https://www.computerworld.com/sitemap-pages-nz.xml
158Sitemap: https://www.computerworld.com/sitemap-pages-uk.xml
159Sitemap: https://www.computerworld.com/sitemap-profiles.xml
160Sitemap: https://www.computerworld.com/sitemap-brandposts.xml
What do the bots actually get?
This page reads robots.txt, which states what computerworld.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on computerworld.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
- 26 October 2026: started blocking Google-Extended; stopped blocking Meta-ExternalFetcher.
- 9 November 2026: stopped blocking Google-Extended.
Questions
Does computerworld.com block GPTBot?
Yes. computerworld.com’s robots.txt disallows GPTBot from the whole site (line 93: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does computerworld.com block ClaudeBot?
Yes. computerworld.com’s robots.txt disallows ClaudeBot from the whole site (line 66: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does computerworld.com block Google-Extended?
No. computerworld.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does computerworld.com block PerplexityBot?
No. computerworld.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find computerworld.com?
Yes. computerworld.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.