Who blocks AI · #688 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does espn.com block AI crawlers?
Yes. espn.com’s robots.txt blocks 5 of the 19 AI crawlers we track, including GPTBot, Google-Extended, CCBot and 2 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.espn.com/robots.txt · 142 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 7 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Blocked | line 10 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 13 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Blocked | line 31 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 16 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.espn.com/robots.txt · 142 lines
1# robots.txt for www.espn.com
2
3User-agent: claritybot
4Disallow: /
5
6User-agent: GPTBot
7Disallow: /
8
9User-agent: Google-Extended
10Disallow: /
11
12User-agent: CCBot
13Disallow: /
14
15User-agent: ChatGPT-User
16Disallow: /
17
18User-agent: anthropic-ai
19Disallow: /
20
21User-agent: Omgilibot
22Disallow: /
23
24User-agent: Omgili
25Disallow: /
26
27User-agent: FacebookBot
28Disallow: /
29
30User-agent: Bytespider
31Disallow: /
32
33User-agent: *
34Disallow: */admin/
35Disallow: */boxscore?
36Disallow: */calendar/
37Disallow: */cat/
38Disallow: */conversation?
39Disallow: */conversation/
40Disallow: */conversations?
41Disallow: */conversations/
42Disallow: */databaseresults/
43Disallow: */date/
44Disallow: */deportes/
45Disallow: */flash/
46Disallow: */login?
47Disallow: */login/
48Disallow: */order/
49Disallow: */photoId/
50Disallow: */playbyplay?
51Disallow: /watch/*?*lang=
52Disallow: /watch/*?*rb=
53Disallow: /watch/*?*country=
54Disallow: /watch/*?*search=
55Disallow: /watch/*?*device=
56Disallow: /watch/*?*partner=
57Disallow: /watch/*?*redirected=
58Disallow: /watch/_/redirected/*
59Disallow: /watch/*?*om-navmethod=
60Disallow: */print?id
61Disallow: */printschedule/
62Disallow: */season/200
63Disallow: */split/
64Disallow: */start/
65Disallow: */swf/
66Disallow: */util/
67Disallow: */webslices/
68Disallow: */year/200
69Disallow: /*?ex_cid=
70Disallow: /*?innings=
71Disallow: /*?user=
72Disallow: /*&action=login
73Disallow: /*&full=
74Disallow: /*&order=
75Disallow: /*&photoId=
76Disallow: /*&sort=*
77Disallow: /ad/
78Disallow: /broadband/
79Disallow: /cgi
80Disallow: /community/
81Disallow: /composer/
82Disallow: /contests/
83Disallow: /espn/now
84Disallow: /espnradio/podcast/feeds/easports/
85Disallow: /index?sport=*&topId=*
86Disallow: /index?sport=*&type=replay
87Disallow: /members/
88Disallow: /personalization/
89Disallow: /sports*/index?topId=*
90Disallow: /travel/passport/
91Disallow: /video/clipDeportes
92Disallow: /video/search
93Disallow: /nba/tradeMachine/?tradeId=*
94Disallow: */wire?
95Disallow: /load_video_player/*
96Disallow: /football/conversation?gameId=*
97Disallow: */channels/*
98Disallow: */crossdomain/*
99Disallow: */makecorsrequest/*
100Disallow: */utils/*
101Disallow: */strategies/*
102Disallow: */child/*
103Disallow: */message/*
104Disallow: */units/*
105Disallow: */om-navmethod/*
106Disallow: */null
107Disallow: */null/*
108Disallow: */undefined
109Disallow: */undefined/*
110Disallow: *ESPNPlusAdobeTracking/
111Disallow: */modifier/*
112Disallow: /*&xhr=1
113Disallow: /*&wsVar=*
114Disallow: /*&_slug_=*
115Disallow: /*.com
116Disallow: /*.net
117Disallow: /*.org
118Disallow: /*.co.uk
119Disallow: */platform/*
120Disallow: /*&platform=
121Disallow: /*?platform=
122Disallow: /football/matchstats
123Disallow: /football/lineups
124Disallow: /football/commentary
125Disallow: /soccer/matchstats
126Disallow: /football/video
127Disallow: /soccer/lineups
128Disallow: /soccer/commentary
129Disallow: /soccer/video
130Disallow: //*
131Disallow: /*?*::*
132Disallow: /rugby?*
133Disallow: /jayski/tracks/archive/story/_/id/*/*/*
134Disallow: */baltimore-ravens-head-coach-john-harbaugh-clocks-long-hours-prep-game-day-espn-magazine
135Disallow: */kansas-city-royals-manager-ned-yost-lightning-rod-criticism
136Disallow: */wwe-aj-styles-retains-top-spot-narrowest-margins-kevin-owens-john-cena
137Allow: */platform/amp*
138Allow: /*&platform=amp
139Allow: /*?platform=amp
140Allow: /.well-known/amphtml/apikey.pub
141Sitemap: https://www.espn.com/sitemap.xml
142
What do the bots actually get?
This page reads robots.txt, which states what espn.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on espn.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does espn.com block GPTBot?
Yes. espn.com’s robots.txt disallows GPTBot from the whole site (line 7: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does espn.com block ClaudeBot?
No. espn.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does espn.com block Google-Extended?
Yes. espn.com’s robots.txt disallows Google-Extended from the whole site (line 10: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does espn.com block PerplexityBot?
No. espn.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find espn.com?
Yes. espn.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.