Who blocks AI · #285 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does ubuntu.com block AI crawlers?
No. ubuntu.com’s robots.txt lets all 19 AI crawlers we track crawl the site, though it keeps OAI-SearchBot, PerplexityBot and 9 more out of some paths. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotPartly blockedOAI-SearchBotPartly blockedClaudeBotPartly blockedPerplexityBotPartly blockedGoogle-ExtendedPartly blockedCCBotAllowed- robots.txt
- ubuntu.com/robots.txt · 224 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Partly blocked | line 132 · Disallow: /search line 133 · Disallow: /search* and 69 more |
PerplexityBot | Perplexity | Partly blocked | line 132 · Disallow: /search line 133 · Disallow: /search* and 69 more |
Claude-SearchBot | Anthropic | Partly blocked | line 132 · Disallow: /search line 133 · Disallow: /search* and 69 more |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Partly blocked | line 132 · Disallow: /search line 133 · Disallow: /search* and 69 more |
ClaudeBot | Anthropic | Partly blocked | line 132 · Disallow: /search line 133 · Disallow: /search* and 69 more |
Google-Extended | Partly blocked | line 132 · Disallow: /search line 133 · Disallow: /search* and 69 more | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Partly blocked | line 132 · Disallow: /search line 133 · Disallow: /search* and 69 more |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Partly blocked | line 132 · Disallow: /search line 133 · Disallow: /search* and 69 more |
Bytespider | ByteDance | Partly blocked | line 132 · Disallow: /search line 133 · Disallow: /search* and 69 more |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Partly blocked | line 132 · Disallow: /search line 133 · Disallow: /search* and 69 more |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Partly blocked | line 132 · Disallow: /search line 133 · Disallow: /search* and 69 more |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View ubuntu.com/robots.txt · 224 lines
1# ===========================================
2# ubuntu.com robots.txt
3# Strategy:
4# - Block private/transactional/search paths for all.
5# - Group major AI Retrieval & Crawling bots to optimize crawl budget.
6# - Allow high-value product/doc paths; block noise.
7# ===========================================
8
9# ===========================================
10# DEFAULT RULES — all crawlers
11# ===========================================
12User-Agent: *
13Disallow: /search
14Disallow: /search*
15Disallow: /*/search*
16Allow: /certified/search
17Disallow: /account
18Disallow: /account/*
19Disallow: /login
20Disallow: /logout
21Disallow: /pro/dashboard
22Disallow: /pro/users
23Disallow: /pro/account-users
24Disallow: /pro/subscribe
25Disallow: /pro/activate
26Disallow: /pro/attach
27Disallow: /pro/offer
28Disallow: /pro/offers
29Disallow: /pro/renewals/
30Disallow: /pro/contracts/
31Disallow: /pro/trial/
32Disallow: /pro/set-auto-renewal
33Disallow: /pro/user-subscriptions
34Disallow: /pro/distributor/users
35Disallow: /pro/distributor/invoice
36Disallow: /pro/distributor/thank-you
37Disallow: /account.json
38Disallow: /mirrors.json
39Disallow: /pro/subscriptions.json
40Disallow: /pro/offers.json
41Disallow: /pro/channel-offers.json
42Disallow: /thank-you
43Disallow: /*/thank-you
44Disallow: /blog/draft-blogs
45Disallow: /blog/draft-blogs/*
46Disallow: /tests/
47Disallow: /tests/*
48Disallow: /sentry-debug
49Disallow: /mobile
50Disallow: /mobile/*
51Disallow: /phone
52Disallow: /phone/*
53Disallow: /tablet
54Disallow: /tablet/*
55Disallow: /tv
56Disallow: /tv/*
57Disallow: /devices
58Disallow: /devices/*
59Disallow: /credentials/exam*
60Crawl-delay: 1
61
62# ===========================================
63# AI OPTIMIZED RULES
64# Includes: OpenAI (Browsing & Crawling), Perplexity, Anthropic, and Apple
65# ===========================================
66User-agent: GPTBot
67User-agent: ChatGPT-User
68User-agent: OAI-SearchBot
69User-agent: ClaudeBot
70User-agent: Claude-Web
71User-agent: anthropic-ai
72User-agent: Claude-SearchBot
73User-agent: Google-Extended
74User-agent: meta-externalagent
75User-agent: PerplexityBot
76User-agent: Perplexity-User
77User-agent: Applebot-Extended
78User-agent: cohere-ai
79User-agent: Bytespider
80
81# Nudge toward Markdown endpoints
82Allow: /*?format=md
83
84# High-value Content Priority
85Allow: /server
86Allow: /desktop
87Allow: /cloud
88Allow: /openstack
89Allow: /kubernetes
90Allow: /ceph
91Allow: /containers
92Allow: /core
93Allow: /ai
94Allow: /pro
95Allow: /landscape
96Allow: /security
97Allow: /internet-of-things
98Allow: /embedded
99Allow: /hpc
100Allow: /real-time
101Allow: /confidential-computing
102Allow: /enterprise-store
103Allow: /kernel
104Allow: /toolchains
105Allow: /robotics
106Allow: /certified
107Allow: /about
108Allow: /community
109Allow: /download
110Allow: /pricing
111Allow: /training
112Allow: /credentials
113Allow: /support
114Allow: /managed
115Allow: /managed-infrastructure
116Allow: /aws
117Allow: /azure
118Allow: /gcp
119Allow: /dell
120Allow: /ibm
121Allow: /nvidia
122Allow: /hpe
123Allow: /supermicro
124Allow: /blender
125Allow: /blog
126Allow: /tutorials
127Allow: /appliance
128Allow: /cpu-compatibility
129Allow: /what-is-enterprise-linux
130
131# Block "Noise" (Forms, fragments, and archives that exhaust context windows)
132Disallow: /search
133Disallow: /search*
134Disallow: /*/search*
135Allow: /certified/search
136Disallow: /account
137Disallow: /account/*
138Disallow: /account.json
139Disallow: /mirrors.json
140Disallow: /login
141Disallow: /logout
142Disallow: /contact-us
143Disallow: /contact-us/*
144Disallow: /*/contact-us
145Disallow: /thank-you
146Disallow: /*/thank-you
147Disallow: /engage
148Disallow: /engage/*
149Disallow: /takeovers
150Disallow: /takeovers.json
151Disallow: /templates/
152Disallow: /frame
153Disallow: /marketo/submit
154Disallow: /blog/feed
155Disallow: /blog/archives
156Disallow: /blog/tag/
157Disallow: /blog/author/
158Disallow: /blog/topic/
159Disallow: /blog/group/
160Disallow: /blog/latest-news
161Disallow: /blog/events-and-webinars
162Disallow: /blog/draft-blogs
163Disallow: /blog/draft-blogs/*
164Disallow: /engage/resources.json
165Disallow: /engage/metadata.json
166Disallow: /pro/dashboard
167Disallow: /pro/users
168Disallow: /pro/account-users
169Disallow: /pro/subscribe
170Disallow: /pro/activate
171Disallow: /pro/attach
172Disallow: /pro/offer
173Disallow: /pro/offers
174Disallow: /pro/renewals/
175Disallow: /pro/contracts/
176Disallow: /pro/trial/
177Disallow: /pro/set-auto-renewal
178Disallow: /pro/user-subscriptions
179Disallow: /pro/subscriptions.json
180Disallow: /pro/offers.json
181Disallow: /pro/channel-offers.json
182Disallow: /pro/distributor/users
183Disallow: /pro/distributor/invoice
184Disallow: /pro/distributor/thank-you
185Disallow: /subscription-centre
186Disallow: /rfp
187Disallow: /tests/
188Disallow: /tests/*
189Disallow: /_image-testing/
190Disallow: /sentry-debug
191Disallow: /credentials/exam*
192Disallow: /mobile
193Disallow: /mobile/*
194Disallow: /phone
195Disallow: /phone/*
196Disallow: /tablet
197Disallow: /tablet/*
198Disallow: /tv
199Disallow: /tv/*
200Disallow: /devices
201Disallow: /devices/*
202Disallow: /kubeconeurope2020
203Disallow: /masters-conference
204
205# Performance
206Crawl-delay: 2
207
208# ===========================================
209# SITEMAPS
210# ===========================================
211Sitemap: https://ubuntu.com/sitemap.xml
212
213# High-Value Content Trees (Prioritized for AI Crawlers)
214Sitemap: https://ubuntu.com/sitemap_tree.xml
215Sitemap: https://ubuntu.com/blog/sitemap.xml
216Sitemap: https://ubuntu.com/tutorials/sitemap.xml
217Sitemap: https://ubuntu.com/server/docs/doc-sitemap.xml
218Sitemap: https://ubuntu.com/openstack/docs/sitemap.xml
219Sitemap: https://ubuntu.com/ceph/docs/sitemap.xml
220Sitemap: https://ubuntu.com/security/notices/sitemap.xml
221Sitemap: https://ubuntu.com/security/cves/sitemap.xml
222Sitemap: https://ubuntu.com/security/vulnerabilities/sitemap.xml
223Sitemap: https://ubuntu.com/community/docs/sitemap.xml
224
What do the bots actually get?
This page reads robots.txt, which states what ubuntu.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on ubuntu.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does ubuntu.com block GPTBot?
Partly. ubuntu.com’s robots.txt keeps GPTBot out of some paths, such as /search, and lets it crawl the rest. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does ubuntu.com block ClaudeBot?
Partly. ubuntu.com’s robots.txt keeps ClaudeBot out of some paths, such as /search, and lets it crawl the rest. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does ubuntu.com block Google-Extended?
Partly. ubuntu.com’s robots.txt keeps Google-Extended out of some paths, such as /search, and lets it crawl the rest. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does ubuntu.com block PerplexityBot?
Partly. ubuntu.com’s robots.txt keeps PerplexityBot out of some paths, such as /search, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find ubuntu.com?
Partly. ubuntu.com’s robots.txt keeps OAI-SearchBot out of some paths, such as /search, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.