canaicrawl Get early access

Who blocks AI · #360 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does android.com block AI crawlers?

No. android.com’s robots.txt lets all 19 AI crawlers we track crawl the site, though it keeps OAI-SearchBot, PerplexityBot and 8 more out of some paths. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotPartly blockedOAI-SearchBotPartly blockedClaudeBotPartly blockedPerplexityBotPartly blockedGoogle-ExtendedPartly blockedCCBotPartly blocked
robots.txt
www.android.com/robots.txt · 106 lines
llms.txt
none
Checked
9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGoogleAllowedno rule matches the home page
BingbotMicrosoftAllowedno rule matches the home page
DuckDuckBotDuckDuckGoAllowedno rule matches the home page
ApplebotApplePartly blockedline 86 · Disallow: /search line 87 · Disallow: /404 and 3 more
PetalBotHuaweiAllowedno rule matches the home page
YandexBotYandexAllowedno rule matches the home page
BaiduspiderBaiduAllowedno rule matches the home page

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIPartly blockedline 16 · Disallow: /search line 17 · Disallow: /404 and 3 more
PerplexityBotPerplexityPartly blockedline 30 · Disallow: /search line 31 · Disallow: /404 and 3 more
Claude-SearchBotAnthropicPartly blockedline 44 · Disallow: /search line 45 · Disallow: /404 and 3 more
DuckAssistBotDuckDuckGoAllowedno rule matches the home page
AmazonbotAmazonAllowedno rule matches the home page
YouBotYou.comPartly blockedline 93 · Disallow: /search line 94 · Disallow: /404 and 3 more

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIPartly blockedline 9 · Disallow: /search line 10 · Disallow: /404 and 3 more
ClaudeBotAnthropicPartly blockedline 37 · Disallow: /search line 38 · Disallow: /404 and 3 more
Google-ExtendedGooglePartly blockedline 23 · Disallow: /search line 24 · Disallow: /404 and 3 more
CCBotCommon CrawlPartly blockedline 65 · Disallow: /search line 66 · Disallow: /404 and 3 more
Applebot-ExtendedApplePartly blockedline 79 · Disallow: /search line 80 · Disallow: /404 and 3 more
GoogleOtherGoogleAllowedno rule matches the home page
Meta-ExternalAgentMetaAllowedno rule matches the home page
BytespiderByteDanceAllowedno rule matches the home page

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIAllowedno rule matches the home page
Claude-UserAnthropicAllowedno rule matches the home page
Perplexity-UserPerplexityAllowedno rule matches the home page
MistralAI-UserMistralPartly blockedline 100 · Disallow: /search line 101 · Disallow: /404 and 3 more
Meta-ExternalFetcherMetaAllowedno rule matches the home page

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsAllowedno rule matches the home page
SemrushBotSemrushAllowedno rule matches the home page
MJ12botMajesticAllowedno rule matches the home page
DotBotMozAllowedno rule matches the home page
View www.android.com/robots.txt · 106 lines
1User-Agent: *
2Disallow: /search
3Disallow: /404
4Disallow: /payapp/
5Disallow: /results/
6Disallow: /intl/*/results/
7
8User-agent: GPTBot
9Disallow: /search
10Disallow: /404
11Disallow: /payapp/
12Disallow: /results/
13Disallow: /intl/*/results/
14
15User-agent: OAI-SearchBot
16Disallow: /search
17Disallow: /404
18Disallow: /payapp/
19Disallow: /results/
20Disallow: /intl/*/results/
21
22User-agent: Google-Extended
23Disallow: /search
24Disallow: /404
25Disallow: /payapp/
26Disallow: /results/
27Disallow: /intl/*/results/
28
29User-agent: PerplexityBot
30Disallow: /search
31Disallow: /404
32Disallow: /payapp/
33Disallow: /results/
34Disallow: /intl/*/results/
35
36User-agent: ClaudeBot
37Disallow: /search
38Disallow: /404
39Disallow: /payapp/
40Disallow: /results/
41Disallow: /intl/*/results/
42
43User-agent: Claude-SearchBot
44Disallow: /search
45Disallow: /404
46Disallow: /payapp/
47Disallow: /results/
48Disallow: /intl/*/results/
49
50User-agent: anthropic-ai
51Disallow: /search
52Disallow: /404
53Disallow: /payapp/
54Disallow: /results/
55Disallow: /intl/*/results/
56
57User-agent: cohere-ai
58Disallow: /search
59Disallow: /404
60Disallow: /payapp/
61Disallow: /results/
62Disallow: /intl/*/results/
63
64User-agent: CCBot
65Disallow: /search
66Disallow: /404
67Disallow: /payapp/
68Disallow: /results/
69Disallow: /intl/*/results/
70
71User-agent: Diffbot
72Disallow: /search
73Disallow: /404
74Disallow: /payapp/
75Disallow: /results/
76Disallow: /intl/*/results/
77
78User-agent: Applebot-Extended
79Disallow: /search
80Disallow: /404
81Disallow: /payapp/
82Disallow: /results/
83Disallow: /intl/*/results/
84
85User-agent: Applebot
86Disallow: /search
87Disallow: /404
88Disallow: /payapp/
89Disallow: /results/
90Disallow: /intl/*/results/
91
92User-agent: YouBot
93Disallow: /search
94Disallow: /404
95Disallow: /payapp/
96Disallow: /results/
97Disallow: /intl/*/results/
98
99User-agent: MistralAI-User
100Disallow: /search
101Disallow: /404
102Disallow: /payapp/
103Disallow: /results/
104Disallow: /intl/*/results/
105
106Sitemap: https://www.android.com/sitemap.xml

What do the bots actually get?

This page reads robots.txt, which states what android.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on android.com Test a path in the robots.txt tester

History

Questions

Does android.com block GPTBot?

Partly. android.com’s robots.txt keeps GPTBot out of some paths, such as /search, and lets it crawl the rest. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does android.com block ClaudeBot?

Partly. android.com’s robots.txt keeps ClaudeBot out of some paths, such as /search, and lets it crawl the rest. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does android.com block Google-Extended?

Partly. android.com’s robots.txt keeps Google-Extended out of some paths, such as /search, and lets it crawl the rest. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does android.com block PerplexityBot?

Partly. android.com’s robots.txt keeps PerplexityBot out of some paths, such as /search, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find android.com?

Partly. android.com’s robots.txt keeps OAI-SearchBot out of some paths, such as /search, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.