Who blocks AI · #1217 in the Majestic Million
Does ctvnews.ca block AI crawlers?
Yes. ctvnews.ca’s robots.txt blocks 5 of the 19 AI crawlers we track, including GPTBot, PerplexityBot, OAI-SearchBot and 2 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotAllowedPerplexityBotBlockedGoogle-ExtendedAllowedCCBotBlocked- robots.txt
- www.ctvnews.ca/robots.txt · 69 lines
- llms.txt
- none
- Checked
- 16 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | line 6 · Allow: / under User-agent: * | |
Bingbot | Microsoft | Allowed | line 6 · Allow: / under User-agent: * |
DuckDuckBot | DuckDuckGo | Allowed | line 6 · Allow: / under User-agent: * |
Applebot | Apple | Allowed | line 6 · Allow: / under User-agent: * |
PetalBot | Huawei | Allowed | line 6 · Allow: / under User-agent: * |
YandexBot | Yandex | Allowed | line 6 · Allow: / under User-agent: * |
Baiduspider | Baidu | Allowed | line 6 · Allow: / under User-agent: * |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 51 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 54 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Allowed | line 6 · Allow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Allowed | line 6 · Allow: / under User-agent: * |
Amazonbot | Amazon | Allowed | line 6 · Allow: / under User-agent: * |
YouBot | You.com | Allowed | line 6 · Allow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 45 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Allowed | line 6 · Allow: / under User-agent: * |
Google-Extended | Allowed | line 6 · Allow: / under User-agent: * | |
CCBot | Common Crawl | Blocked | line 57 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | line 6 · Allow: / under User-agent: * |
GoogleOther | Allowed | line 6 · Allow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Allowed | line 6 · Allow: / under User-agent: * |
Bytespider | ByteDance | Allowed | line 6 · Allow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 48 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Allowed | line 6 · Allow: / under User-agent: * |
Perplexity-User | Perplexity | Allowed | line 6 · Allow: / under User-agent: * |
MistralAI-User | Mistral | Allowed | line 6 · Allow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Allowed | line 6 · Allow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | line 6 · Allow: / under User-agent: * |
SemrushBot | Semrush | Allowed | line 6 · Allow: / under User-agent: * |
MJ12bot | Majestic | Allowed | line 6 · Allow: / under User-agent: * |
DotBot | Moz | Allowed | line 6 · Allow: / under User-agent: * |
View www.ctvnews.ca/robots.txt · 69 lines
What do the bots actually get?
This page reads robots.txt, which states what ctvnews.ca asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on ctvnews.ca Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does ctvnews.ca block GPTBot?
Yes. ctvnews.ca’s robots.txt disallows GPTBot from the whole site (line 45: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 16 November 2026.
Does ctvnews.ca block ClaudeBot?
No. ctvnews.ca’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 16 November 2026.
Does ctvnews.ca block Google-Extended?
No. ctvnews.ca’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 16 November 2026.
Does ctvnews.ca block PerplexityBot?
Yes. ctvnews.ca’s robots.txt disallows PerplexityBot from the whole site (line 54: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 16 November 2026.
Can ChatGPT search find ctvnews.ca?
Not through its own crawler. ctvnews.ca’s robots.txt disallows OAI-SearchBot from the whole site (line 51: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 16 November 2026.