canaicrawl Get early access

Who blocks AI · #758 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does abcnews.com block AI crawlers?

Yes. abcnews.com’s robots.txt blocks 5 of the 19 AI crawlers we track, including GPTBot, Google-Extended, CCBot and 2 more. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotBlockedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedBlockedCCBotBlocked
robots.txt
abcnews.com/robots.txt · 108 lines
llms.txt
none
Checked
9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGoogleAllowedno rule matches the home page
BingbotMicrosoftAllowedno rule matches the home page
DuckDuckBotDuckDuckGoAllowedno rule matches the home page
ApplebotApplePartly blockedline 108 · Disallow: /private/
PetalBotHuaweiAllowedno rule matches the home page
YandexBotYandexAllowedno rule matches the home page
BaiduspiderBaiduAllowedno rule matches the home page

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIAllowedno rule matches the home page
PerplexityBotPerplexityAllowedno rule matches the home page
Claude-SearchBotAnthropicAllowedno rule matches the home page
DuckAssistBotDuckDuckGoAllowedno rule matches the home page
AmazonbotAmazonAllowedno rule matches the home page
YouBotYou.comAllowedno rule matches the home page

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIBlockedline 13 · Disallow: / under User-agent: GPTBot
ClaudeBotAnthropicAllowedno rule matches the home page
Google-ExtendedGoogleBlockedline 16 · Disallow: / under User-agent: Google-Extended
CCBotCommon CrawlBlockedline 19 · Disallow: / under User-agent: CCBot
Applebot-ExtendedAppleAllowedno rule matches the home page
GoogleOtherGoogleAllowedno rule matches the home page
Meta-ExternalAgentMetaAllowedno rule matches the home page
BytespiderByteDanceBlockedline 37 · Disallow: / under User-agent: Bytespider

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIBlockedline 22 · Disallow: / under User-agent: ChatGPT-User
Claude-UserAnthropicAllowedno rule matches the home page
Perplexity-UserPerplexityAllowedno rule matches the home page
MistralAI-UserMistralAllowedno rule matches the home page
Meta-ExternalFetcherMetaAllowedno rule matches the home page

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsAllowedno rule matches the home page
SemrushBotSemrushAllowedno rule matches the home page
MJ12botMajesticAllowedno rule matches the home page
DotBotMozAllowedno rule matches the home page
View abcnews.com/robots.txt · 108 lines
1
2# robots.txt for https://abcnews.com/
3#
4
5Sitemap: https://abcnews.com/xmap
6Sitemap: https://abcnews.com/xmlLatestStories
7Sitemap: https://abcnews.com/xmlLatestVideos
8
9User-agent: claritybot
10Disallow: /
11
12User-agent: GPTBot
13Disallow: /
14
15User-agent: Google-Extended
16Disallow: /
17
18User-agent: CCBot
19Disallow: /
20
21User-agent: ChatGPT-User
22Disallow: /
23
24User-agent: anthropic-ai
25Disallow: /
26
27User-agent: Omgilibot
28Disallow: /
29
30User-agent: Omgili
31Disallow: /
32
33User-agent: FacebookBot
34Disallow: /
35
36User-agent: Bytespider
37Disallow: /
38
39User-agent: *
40Disallow: /search?searchtext=*
41Disallow: /disneyid/*
42Disallow: /assets/static/ads/*
43
44Disallow: /cgi
45Disallow: /xls
46Disallow: /imp
47Disallow: /kmail
48Disallow: /map
49Disallow: /log
50Disallow: /gif
51Disallow: /panel
52
53Disallow: /0/
54
55Disallow: /promo/
56
57Disallow: /abclinks/
58
59Disallow: /houseads/
60
61# Disallow: /xmldata/
62Allow: /xmldata/mrss
63Allow: /xmldata/rss
64Allow: /xmldata/xmlPodcast
65Allow: /xmldata/config
66Allow: /xmldata/feed
67
68Disallow: /sendtofriend/
69Allow: /meta/sitemap
70Disallow: /meta/
71
72Disallow: /staging/
73Disallow: /test/
74Disallow: /swen/
75Disallow: /intro/
76
77Disallow: /go/
78Disallow: /news/go/
79
80Disallow: /widgets/
81Disallow: /vp2/
82Disallow: /Video/*playerIndex
83
84Disallow: /*carousel/
85Disallow: /*videoLogin?
86Disallow: /video/browse/
87Disallow: /*popup?
88
89Disallow: /alerts-v1/
90
91Disallow: /not-allowed/
92
93Disallow: /beta-story-container/*
94Disallow: /video/embed/*
95Disallow: /video/amp/embed/*
96Disallow: /responder
97Disallow: /video/itemfeed*
98Disallow: /video/caption*
99Disallow: /content/news/liveBlog/*
100Disallow: /configuration/*
101Disallow: /rsidxfeed/*
102Disallow: /xmldata/*
103Disallow: /gma/api/*
104Disallow: /proxy/*
105
106User-agent: Applebot
107Allow: /
108Disallow: /private/

What do the bots actually get?

This page reads robots.txt, which states what abcnews.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on abcnews.com Test a path in the robots.txt tester

History

Questions

Does abcnews.com block GPTBot?

Yes. abcnews.com’s robots.txt disallows GPTBot from the whole site (line 13: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does abcnews.com block ClaudeBot?

No. abcnews.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does abcnews.com block Google-Extended?

Yes. abcnews.com’s robots.txt disallows Google-Extended from the whole site (line 16: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does abcnews.com block PerplexityBot?

No. abcnews.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find abcnews.com?

Yes. abcnews.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.