Who blocks AI · #920 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does nba.com block AI crawlers?
Yes. nba.com’s robots.txt blocks 8 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 5 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.nba.com/robots.txt · 445 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | line 380 · Allow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 414 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Blocked | line 156 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 72 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 316 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 114 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 252 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 220 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 284 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.nba.com/robots.txt · 445 lines
1Sitemap: https://www.nba.com/sitemap_index.xml
2User-Agent: *
3Disallow: /amp/
4Disallow: /api/*
5Disallow: /mediacentral/*
6Disallow: /search
7Disallow: /video/partners/
8Disallow: /stats/events/
9Disallow: /how-to-watch-game-redirect
10Disallow: /suns/tickets/vervelounge_requests.html
11Disallow: /blazers/node/
12Disallow: /hornets/node/
13Disallow: /bucks/node/
14Disallow: /bulls/node/
15Disallow: /cavaliers/node/
16Disallow: /celtics/node/
17Disallow: /clippers/node/
18Disallow: /grizzlies/node/
19Disallow: /hawks/node/
20Disallow: /heat/node/
21Disallow: /jazz/node/
22Disallow: /kings/node/
23Disallow: /knicks/node/
24Disallow: /lakers/node/
25Disallow: /magic/node/
26Disallow: /mavs/node/
27Disallow: /nets/node/
28Disallow: /nuggets/node/
29Disallow: /pacers/node/
30Disallow: /pelicans/node/
31Disallow: /pistons/node/
32Disallow: /raptors/node/
33Disallow: /rockets/node/
34Disallow: /sixers/node/
35Disallow: /spurs/node/
36Disallow: /suns/node/
37Disallow: /thunder/node/
38Disallow: /timberwolves/node/
39Disallow: /warriors/node/
40Disallow: /wizards/node/
41Disallow: /blazers/users/
42Disallow: /hornets/users/
43Disallow: /bucks/users/
44Disallow: /bulls/users/
45Disallow: /cavaliers/users/
46Disallow: /celtics/users/
47Disallow: /clippers/users/
48Disallow: /grizzlies/users/
49Disallow: /hawks/users/
50Disallow: /heat/users/
51Disallow: /jazz/users/
52Disallow: /kings/users/
53Disallow: /knicks/users/
54Disallow: /lakers/users/
55Disallow: /magic/users/
56Disallow: /mavs/users/
57Disallow: /nets/users/
58Disallow: /nuggets/users/
59Disallow: /pacers/users/
60Disallow: /pelicans/users/
61Disallow: /pistons/users/
62Disallow: /raptors/users/
63Disallow: /rockets/users/
64Disallow: /sixers/users/
65Disallow: /spurs/users/
66Disallow: /suns/users/
67Disallow: /thunder/users/
68Disallow: /timberwolves/users/
69Disallow: /warriors/users/
70Disallow: /wizards/users/
71User-agent: GPTBot
72Disallow: /
73Allow: /blazers/game/*
74Allow: /hornets/game/*
75Allow: /bucks/game/*
76Allow: /bulls/game/*
77Allow: /cavaliers/game/*
78Allow: /celtics/game/*
79Allow: /clippers/game/*
80Allow: /grizzlies/game/*
81Allow: /hawks/game/*
82Allow: /heat/game/*
83Allow: /jazz/game/*
84Allow: /kings/game/*
85Allow: /knicks/game/*
86Allow: /lakers/game/*
87Allow: /magic/game/*
88Allow: /mavs/game/*
89Allow: /nets/game/*
90Allow: /nuggets/game/*
91Allow: /pacers/game/*
92Allow: /pelicans/game/*
93Allow: /pistons/game/*
94Allow: /raptors/game/*
95Allow: /rockets/game/*
96Allow: /sixers/game/*
97Allow: /spurs/game/*
98Allow: /suns/game/*
99Allow: /thunder/game/*
100Allow: /timberwolves/game/*
101Allow: /warriors/game/*
102Allow: /wizards/game/*
103Allow: /standings
104Allow: /schedule
105Allow: /stats/help/glossary
106Allow: /stats/draft/history
107Allow: /stats/help/statminimums
108Allow: /stats/history
109Allow: /players
110Allow: /player/*profile$
111Allow: /team/*
112Disallow: /team/*/schedule$
113User-agent: Google-Extended
114Disallow: /
115Allow: /blazers/game/*
116Allow: /hornets/game/*
117Allow: /bucks/game/*
118Allow: /bulls/game/*
119Allow: /cavaliers/game/*
120Allow: /celtics/game/*
121Allow: /clippers/game/*
122Allow: /grizzlies/game/*
123Allow: /hawks/game/*
124Allow: /heat/game/*
125Allow: /jazz/game/*
126Allow: /kings/game/*
127Allow: /knicks/game/*
128Allow: /lakers/game/*
129Allow: /magic/game/*
130Allow: /mavs/game/*
131Allow: /nets/game/*
132Allow: /nuggets/game/*
133Allow: /pacers/game/*
134Allow: /pelicans/game/*
135Allow: /pistons/game/*
136Allow: /raptors/game/*
137Allow: /rockets/game/*
138Allow: /sixers/game/*
139Allow: /spurs/game/*
140Allow: /suns/game/*
141Allow: /thunder/game/*
142Allow: /timberwolves/game/*
143Allow: /warriors/game/*
144Allow: /wizards/game/*
145Allow: /standings
146Allow: /schedule
147Allow: /stats/help/glossary
148Allow: /stats/draft/history
149Allow: /stats/help/statminimums
150Allow: /stats/history
151Allow: /players
152Allow: /player/*profile$
153Allow: /team/*
154Disallow: /team/*/schedule$
155User-agent: Amazonbot
156Disallow: /
157Allow: /blazers/game/*
158Allow: /hornets/game/*
159Allow: /bucks/game/*
160Allow: /bulls/game/*
161Allow: /cavaliers/game/*
162Allow: /celtics/game/*
163Allow: /clippers/game/*
164Allow: /grizzlies/game/*
165Allow: /hawks/game/*
166Allow: /heat/game/*
167Allow: /jazz/game/*
168Allow: /kings/game/*
169Allow: /knicks/game/*
170Allow: /lakers/game/*
171Allow: /magic/game/*
172Allow: /mavs/game/*
173Allow: /nets/game/*
174Allow: /nuggets/game/*
175Allow: /pacers/game/*
176Allow: /pelicans/game/*
177Allow: /pistons/game/*
178Allow: /raptors/game/*
179Allow: /rockets/game/*
180Allow: /sixers/game/*
181Allow: /spurs/game/*
182Allow: /suns/game/*
183Allow: /thunder/game/*
184Allow: /timberwolves/game/*
185Allow: /warriors/game/*
186Allow: /wizards/game/*
187User-agent: anthropic-ai
188Disallow: /
189Allow: /blazers/game/*
190Allow: /hornets/game/*
191Allow: /bucks/game/*
192Allow: /bulls/game/*
193Allow: /cavaliers/game/*
194Allow: /celtics/game/*
195Allow: /clippers/game/*
196Allow: /grizzlies/game/*
197Allow: /hawks/game/*
198Allow: /heat/game/*
199Allow: /jazz/game/*
200Allow: /kings/game/*
201Allow: /knicks/game/*
202Allow: /lakers/game/*
203Allow: /magic/game/*
204Allow: /mavs/game/*
205Allow: /nets/game/*
206Allow: /nuggets/game/*
207Allow: /pacers/game/*
208Allow: /pelicans/game/*
209Allow: /pistons/game/*
210Allow: /raptors/game/*
211Allow: /rockets/game/*
212Allow: /sixers/game/*
213Allow: /spurs/game/*
214Allow: /suns/game/*
215Allow: /thunder/game/*
216Allow: /timberwolves/game/*
217Allow: /warriors/game/*
218Allow: /wizards/game/*
219User-agent: Applebot-Extended
220Disallow: /
221Allow: /blazers/game/*
222Allow: /hornets/game/*
223Allow: /bucks/game/*
224Allow: /bulls/game/*
225Allow: /cavaliers/game/*
226Allow: /celtics/game/*
227Allow: /clippers/game/*
228Allow: /grizzlies/game/*
229Allow: /hawks/game/*
230Allow: /heat/game/*
231Allow: /jazz/game/*
232Allow: /kings/game/*
233Allow: /knicks/game/*
234Allow: /lakers/game/*
235Allow: /magic/game/*
236Allow: /mavs/game/*
237Allow: /nets/game/*
238Allow: /nuggets/game/*
239Allow: /pacers/game/*
240Allow: /pelicans/game/*
241Allow: /pistons/game/*
242Allow: /raptors/game/*
243Allow: /rockets/game/*
244Allow: /sixers/game/*
245Allow: /spurs/game/*
246Allow: /suns/game/*
247Allow: /thunder/game/*
248Allow: /timberwolves/game/*
249Allow: /warriors/game/*
250Allow: /wizards/game/*
251User-agent: CCBot
252Disallow: /
253Allow: /blazers/game/*
254Allow: /hornets/game/*
255Allow: /bucks/game/*
256Allow: /bulls/game/*
257Allow: /cavaliers/game/*
258Allow: /celtics/game/*
259Allow: /clippers/game/*
260Allow: /grizzlies/game/*
261Allow: /hawks/game/*
262Allow: /heat/game/*
263Allow: /jazz/game/*
264Allow: /kings/game/*
265Allow: /knicks/game/*
266Allow: /lakers/game/*
267Allow: /magic/game/*
268Allow: /mavs/game/*
269Allow: /nets/game/*
270Allow: /nuggets/game/*
271Allow: /pacers/game/*
272Allow: /pelicans/game/*
273Allow: /pistons/game/*
274Allow: /raptors/game/*
275Allow: /rockets/game/*
276Allow: /sixers/game/*
277Allow: /spurs/game/*
278Allow: /suns/game/*
279Allow: /thunder/game/*
280Allow: /timberwolves/game/*
281Allow: /warriors/game/*
282Allow: /wizards/game/*
283User-agent: ChatGPT-User
284Disallow: /
285Allow: /blazers/game/*
286Allow: /hornets/game/*
287Allow: /bucks/game/*
288Allow: /bulls/game/*
289Allow: /cavaliers/game/*
290Allow: /celtics/game/*
291Allow: /clippers/game/*
292Allow: /grizzlies/game/*
293Allow: /hawks/game/*
294Allow: /heat/game/*
295Allow: /jazz/game/*
296Allow: /kings/game/*
297Allow: /knicks/game/*
298Allow: /lakers/game/*
299Allow: /magic/game/*
300Allow: /mavs/game/*
301Allow: /nets/game/*
302Allow: /nuggets/game/*
303Allow: /pacers/game/*
304Allow: /pelicans/game/*
305Allow: /pistons/game/*
306Allow: /raptors/game/*
307Allow: /rockets/game/*
308Allow: /sixers/game/*
309Allow: /spurs/game/*
310Allow: /suns/game/*
311Allow: /thunder/game/*
312Allow: /timberwolves/game/*
313Allow: /warriors/game/*
314Allow: /wizards/game/*
315User-agent: ClaudeBot
316Disallow: /
317Allow: /blazers/game/*
318Allow: /hornets/game/*
319Allow: /bucks/game/*
320Allow: /bulls/game/*
321Allow: /cavaliers/game/*
322Allow: /celtics/game/*
323Allow: /clippers/game/*
324Allow: /grizzlies/game/*
325Allow: /hawks/game/*
326Allow: /heat/game/*
327Allow: /jazz/game/*
328Allow: /kings/game/*
329Allow: /knicks/game/*
330Allow: /lakers/game/*
331Allow: /magic/game/*
332Allow: /mavs/game/*
333Allow: /nets/game/*
334Allow: /nuggets/game/*
335Allow: /pacers/game/*
336Allow: /pelicans/game/*
337Allow: /pistons/game/*
338Allow: /raptors/game/*
339Allow: /rockets/game/*
340Allow: /sixers/game/*
341Allow: /spurs/game/*
342Allow: /suns/game/*
343Allow: /thunder/game/*
344Allow: /timberwolves/game/*
345Allow: /warriors/game/*
346Allow: /wizards/game/*
347User-agent: Claude-Web
348Disallow: /
349Allow: /blazers/game/*
350Allow: /hornets/game/*
351Allow: /bucks/game/*
352Allow: /bulls/game/*
353Allow: /cavaliers/game/*
354Allow: /celtics/game/*
355Allow: /clippers/game/*
356Allow: /grizzlies/game/*
357Allow: /hawks/game/*
358Allow: /heat/game/*
359Allow: /jazz/game/*
360Allow: /kings/game/*
361Allow: /knicks/game/*
362Allow: /lakers/game/*
363Allow: /magic/game/*
364Allow: /mavs/game/*
365Allow: /nets/game/*
366Allow: /nuggets/game/*
367Allow: /pacers/game/*
368Allow: /pelicans/game/*
369Allow: /pistons/game/*
370Allow: /raptors/game/*
371Allow: /rockets/game/*
372Allow: /sixers/game/*
373Allow: /spurs/game/*
374Allow: /suns/game/*
375Allow: /thunder/game/*
376Allow: /timberwolves/game/*
377Allow: /warriors/game/*
378Allow: /wizards/game/*
379User-agent: OAI-SearchBot
380Allow: /
381User-agent: Quora-Bot
382Disallow: /
383Allow: /blazers/game/*
384Allow: /hornets/game/*
385Allow: /bucks/game/*
386Allow: /bulls/game/*
387Allow: /cavaliers/game/*
388Allow: /celtics/game/*
389Allow: /clippers/game/*
390Allow: /grizzlies/game/*
391Allow: /hawks/game/*
392Allow: /heat/game/*
393Allow: /jazz/game/*
394Allow: /kings/game/*
395Allow: /knicks/game/*
396Allow: /lakers/game/*
397Allow: /magic/game/*
398Allow: /mavs/game/*
399Allow: /nets/game/*
400Allow: /nuggets/game/*
401Allow: /pacers/game/*
402Allow: /pelicans/game/*
403Allow: /pistons/game/*
404Allow: /raptors/game/*
405Allow: /rockets/game/*
406Allow: /sixers/game/*
407Allow: /spurs/game/*
408Allow: /suns/game/*
409Allow: /thunder/game/*
410Allow: /timberwolves/game/*
411Allow: /warriors/game/*
412Allow: /wizards/game/*
413User-agent: PerplexityBot
414Disallow: /
415Allow: /blazers/game/*
416Allow: /hornets/game/*
417Allow: /bucks/game/*
418Allow: /bulls/game/*
419Allow: /cavaliers/game/*
420Allow: /celtics/game/*
421Allow: /clippers/game/*
422Allow: /grizzlies/game/*
423Allow: /hawks/game/*
424Allow: /heat/game/*
425Allow: /jazz/game/*
426Allow: /kings/game/*
427Allow: /knicks/game/*
428Allow: /lakers/game/*
429Allow: /magic/game/*
430Allow: /mavs/game/*
431Allow: /nets/game/*
432Allow: /nuggets/game/*
433Allow: /pacers/game/*
434Allow: /pelicans/game/*
435Allow: /pistons/game/*
436Allow: /raptors/game/*
437Allow: /rockets/game/*
438Allow: /sixers/game/*
439Allow: /spurs/game/*
440Allow: /suns/game/*
441Allow: /thunder/game/*
442Allow: /timberwolves/game/*
443Allow: /warriors/game/*
444Allow: /wizards/game/*
445
What do the bots actually get?
This page reads robots.txt, which states what nba.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on nba.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does nba.com block GPTBot?
Yes. nba.com’s robots.txt disallows GPTBot from the whole site (line 72: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does nba.com block ClaudeBot?
Yes. nba.com’s robots.txt disallows ClaudeBot from the whole site (line 316: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does nba.com block Google-Extended?
Yes. nba.com’s robots.txt disallows Google-Extended from the whole site (line 114: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does nba.com block PerplexityBot?
Yes. nba.com’s robots.txt disallows PerplexityBot from the whole site (line 414: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find nba.com?
Yes. nba.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.