canaicrawl Get early access

Who blocks AI · #1056 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does deadline.com block AI crawlers?

Yes. deadline.com’s robots.txt blocks 12 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 9 more. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked
robots.txt
deadline.com/robots.txt · 93 lines
llms.txt
none
Checked
9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGoogleAllowedno rule matches the home page
BingbotMicrosoftAllowedno rule matches the home page
DuckDuckBotDuckDuckGoAllowedno rule matches the home page
ApplebotAppleAllowedno rule matches the home page
PetalBotHuaweiAllowedno rule matches the home page
YandexBotYandexAllowedno rule matches the home page
BaiduspiderBaiduAllowedno rule matches the home page

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIBlockedline 54 · Disallow: / under User-agent: OAI-SearchBot
PerplexityBotPerplexityBlockedline 78 · Disallow: / under User-agent: PerplexityBot
Claude-SearchBotAnthropicAllowedno rule matches the home page
DuckAssistBotDuckDuckGoAllowedno rule matches the home page
AmazonbotAmazonAllowedno rule matches the home page
YouBotYou.comBlockedline 87 · Disallow: / under User-agent: YouBot

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIBlockedline 51 · Disallow: / under User-agent: GPTBot
ClaudeBotAnthropicBlockedline 57 · Disallow: / under User-agent: ClaudeBot
Google-ExtendedGoogleBlockedline 69 · Disallow: / under User-agent: Google-Extended
CCBotCommon CrawlBlockedline 36 · Disallow: / under User-agent: CCBot
Applebot-ExtendedAppleBlockedline 30 · Disallow: / under User-agent: Applebot-Extended
GoogleOtherGoogleAllowedno rule matches the home page
Meta-ExternalAgentMetaBlockedline 72 · Disallow: / under User-agent: Meta-ExternalAgent
BytespiderByteDanceBlockedline 33 · Disallow: / under User-agent: Bytespider

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIBlockedline 39 · Disallow: / under User-agent: ChatGPT-User
Claude-UserAnthropicAllowedno rule matches the home page
Perplexity-UserPerplexityAllowedno rule matches the home page
MistralAI-UserMistralAllowedno rule matches the home page
Meta-ExternalFetcherMetaBlockedline 75 · Disallow: / under User-agent: Meta-ExternalFetcher

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsAllowedno rule matches the home page
SemrushBotSemrushAllowedno rule matches the home page
MJ12botMajesticAllowedno rule matches the home page
DotBotMozAllowedno rule matches the home page
View deadline.com/robots.txt · 93 lines
1Sitemap: https://deadline.com/sitemap_index.xml
2Sitemap: https://deadline.com/news-sitemap.xml
3User-agent: *
4Disallow: /wp-admin/
5Allow: /wp-admin/admin-ajax.php
6Disallow: /?s=
7Disallow: /*/?s=
8Disallow: /search/
9Disallow: /search?
10Disallow: *?v02
11Disallow: *?replytocom
12Disallow: /genre/
13Disallow: /genre/*
14Disallow: /*preview=true
15Disallow: /*theme_preview=true
16
17User-agent: anthropic-ai
18Disallow: /
19
20User-agent: cohere-ai
21Disallow: /
22
23User-agent: omgili
24Disallow: /
25
26User-agent: omgilibot
27Disallow: /
28
29User-agent: Applebot-Extended
30Disallow: /
31
32User-agent: Bytespider
33Disallow: /
34
35User-agent: CCBot
36Disallow: /
37
38User-agent: ChatGPT-User
39Disallow: /
40
41User-agent: ChatGPT
42Disallow: /
43
44User-agent: DeepSeek
45Disallow: /
46
47User-agent: DeepSeekBot
48Disallow: /
49
50User-agent: GPTBot
51Disallow: /
52
53User-agent: OAI-SearchBot
54Disallow: /
55
56User-agent: ClaudeBot
57Disallow: /
58
59User-agent: Claude-Web
60Disallow: /
61
62User-agent: Diffbot
63Disallow: /
64
65User-agent: FacebookBot
66Disallow: /
67
68User-agent: Google-Extended
69Disallow: /
70
71User-agent: Meta-ExternalAgent
72Disallow: /
73
74User-agent: Meta-ExternalFetcher
75Disallow: /
76
77User-agent: PerplexityBot
78Disallow: /
79
80User-agent: Timpibot
81Disallow: /
82
83User-agent: Webzio-Extended
84Disallow: /
85
86User-agent: YouBot
87Disallow: /
88User-agent: cXensebot
89Crawl-delay: 5
90
91
92
93

What do the bots actually get?

This page reads robots.txt, which states what deadline.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on deadline.com Test a path in the robots.txt tester

History

Questions

Does deadline.com block GPTBot?

Yes. deadline.com’s robots.txt disallows GPTBot from the whole site (line 51: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does deadline.com block ClaudeBot?

Yes. deadline.com’s robots.txt disallows ClaudeBot from the whole site (line 57: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does deadline.com block Google-Extended?

Yes. deadline.com’s robots.txt disallows Google-Extended from the whole site (line 69: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does deadline.com block PerplexityBot?

Yes. deadline.com’s robots.txt disallows PerplexityBot from the whole site (line 78: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find deadline.com?

Not through its own crawler. deadline.com’s robots.txt disallows OAI-SearchBot from the whole site (line 54: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.