Who blocks AI · #593 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does techradar.com block AI crawlers?
Yes. techradar.com’s robots.txt blocks 5 of the 19 AI crawlers we track, including Meta-ExternalAgent, Applebot-Extended, Bytespider and 2 more. It also blocks Applebot among search engines.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.techradar.com/robots.txt · 96 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Blocked | line 51 · Disallow: / under User-agent: Applebot |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Blocked | line 51 · Disallow: / under User-agent: amazonbot |
YouBot | You.com | Blocked | line 51 · Disallow: / under User-agent: youbot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Blocked | line 51 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 51 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 51 · Disallow: / under User-agent: bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.techradar.com/robots.txt · 96 lines
1##################################################
2# Vanilla-wide rules #
3##################################################
4User-agent: *
5
6# Common path patterns (* prefix to handle localisation)
7Disallow: */deals/compare
8Disallow: */html/
9Disallow: */p/*/embed/captioned
10Disallow: */outlink*
11Disallow: *.freyr.*
12Disallow: */comments/*
13Disallow: */.events-handler*
14Disallow: *CoreWebVitalsThresholdEvent*
15Disallow: */partials/*
16Disallow: */future.adtech.freyr.v1.PageViewEvent
17
18# Common query string patterns
19Disallow: *searchTerm=*
20Disallow: *sortBy=*
21Disallow: *productBrand=*
22Disallow: *%7B*%7D*
23Disallow: *seenMatchId=*
24Disallow: *q=*
25
26# Infinite scroll paths
27Disallow: /infinite-scroll-article/*
28Disallow: /infinite-scroll-review/*
29Disallow: /infinite-scroll-recipe/*
30
31# Amazon AdBot
32User-agent: AmazonAdBot
33Disallow:
34
35# AI crawlers
36User-agent: bytespider
37User-agent: mistralai
38User-agent: cohere
39User-agent: ai2bot
40User-agent: youbot
41User-agent: omgili
42User-agent: diffbot
43User-agent: kangaroo
44User-agent: img2dataset
45User-agent: amazonbot
46User-agent: amazon-qbusiness
47User-Agent: meta-externalagent
48User-Agent: meta-webindexer
49User-Agent: Applebot
50User-Agent: Applebot-Extended
51Disallow: /
52
53##################################################
54# Site-specific rules #
55##################################################
56User-Agent: *
57Disallow: */price-comparison
58Disallow: */specification
59
60##################################################
61# Sitemaps #
62##################################################
63User-agent: *
64Sitemap: https://www.techradar.com/sitemap.xml
65Sitemap: https://www.techradar.com/sg/sitemap.xml
66Sitemap: https://www.techradar.com/uk/sitemap.xml
67Sitemap: https://www.techradar.com/au/sitemap.xml
68Sitemap: https://www.techradar.com/nz/sitemap.xml
69Sitemap: https://global.techradar.com/es-mx/sitemap.xml
70Sitemap: https://global.techradar.com/da-dk/sitemap.xml
71Sitemap: https://global.techradar.com/fi-fi/sitemap.xml
72Sitemap: https://global.techradar.com/no-no/sitemap.xml
73Sitemap: https://global.techradar.com/sv-se/sitemap.xml
74Sitemap: https://global.techradar.com/it-it/sitemap.xml
75Sitemap: https://global.techradar.com/nl-nl/sitemap.xml
76Sitemap: https://global.techradar.com/nl-be/sitemap.xml
77Sitemap: https://global.techradar.com/fr-fr/sitemap.xml
78Sitemap: https://global.techradar.com/de-de/sitemap.xml
79Sitemap: https://global.techradar.com/es-es/sitemap.xml
80Sitemap: https://www.techradar.com/sitemap-news.xml
81Sitemap: https://www.techradar.com/sg/sitemap-news.xml
82Sitemap: https://www.techradar.com/uk/sitemap-news.xml
83Sitemap: https://www.techradar.com/au/sitemap-news.xml
84Sitemap: https://www.techradar.com/nz/sitemap-news.xml
85Sitemap: https://global.techradar.com/es-mx/sitemap-news.xml
86Sitemap: https://global.techradar.com/da-dk/sitemap-news.xml
87Sitemap: https://global.techradar.com/fi-fi/sitemap-news.xml
88Sitemap: https://global.techradar.com/no-no/sitemap-news.xml
89Sitemap: https://global.techradar.com/sv-se/sitemap-news.xml
90Sitemap: https://global.techradar.com/it-it/sitemap-news.xml
91Sitemap: https://global.techradar.com/nl-nl/sitemap-news.xml
92Sitemap: https://global.techradar.com/nl-be/sitemap-news.xml
93Sitemap: https://global.techradar.com/fr-fr/sitemap-news.xml
94Sitemap: https://global.techradar.com/de-de/sitemap-news.xml
95Sitemap: https://global.techradar.com/es-es/sitemap-news.xml
96
What do the bots actually get?
This page reads robots.txt, which states what techradar.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on techradar.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does techradar.com block GPTBot?
No. techradar.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does techradar.com block ClaudeBot?
No. techradar.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does techradar.com block Google-Extended?
No. techradar.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does techradar.com block PerplexityBot?
No. techradar.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find techradar.com?
Yes. techradar.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.