canaicrawl Get early access

Who blocks AI · #51 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does yahoo.com block AI crawlers?

Yes. yahoo.com’s robots.txt blocks 8 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 5 more. It also blocks PetalBot among search engines.

GPTBotBlockedOAI-SearchBotPartly blockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked
robots.txt
www.yahoo.com/robots.txt · 72 lines
llms.txt
none
Checked
9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGoogleAllowedno rule matches the home page
BingbotMicrosoftAllowedno rule matches the home page
DuckDuckBotDuckDuckGoAllowedno rule matches the home page
ApplebotAppleAllowedno rule matches the home page
PetalBotHuaweiBlockedline 67 · Disallow: / under User-agent: PetalBot
YandexBotYandexAllowedno rule matches the home page
BaiduspiderBaiduAllowedno rule matches the home page

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIPartly blockedline 71 · Disallow: */articles/
PerplexityBotPerplexityBlockedline 67 · Disallow: / under User-agent: PerplexityBot
Claude-SearchBotAnthropicPartly blockedline 71 · Disallow: */articles/
DuckAssistBotDuckDuckGoAllowedno rule matches the home page
AmazonbotAmazonAllowedno rule matches the home page
YouBotYou.comBlockedline 67 · Disallow: / under User-agent: YouBot

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIBlockedline 67 · Disallow: / under User-agent: GPTBot
ClaudeBotAnthropicBlockedline 67 · Disallow: / under User-agent: ClaudeBot
Google-ExtendedGoogleBlockedline 67 · Disallow: / under User-agent: Google-Extended
CCBotCommon CrawlBlockedline 67 · Disallow: / under User-agent: CCBot
Applebot-ExtendedAppleAllowedno rule matches the home page
GoogleOtherGoogleAllowedno rule matches the home page
Meta-ExternalAgentMetaAllowedno rule matches the home page
BytespiderByteDanceBlockedline 67 · Disallow: / under User-agent: Bytespider

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIBlockedline 67 · Disallow: / under User-agent: ChatGPT-User
Claude-UserAnthropicAllowedno rule matches the home page
Perplexity-UserPerplexityAllowedno rule matches the home page
MistralAI-UserMistralAllowedno rule matches the home page
Meta-ExternalFetcherMetaAllowedno rule matches the home page

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsAllowedno rule matches the home page
SemrushBotSemrushAllowedno rule matches the home page
MJ12botMajesticAllowedno rule matches the home page
DotBotMozAllowedno rule matches the home page
View www.yahoo.com/robots.txt · 72 lines
1User-agent: *
2Disallow: /info/p.gif
3Disallow: /p/
4Disallow: /r/
5Disallow: /bin/
6Disallow: /caas/
7Disallow: /blank.html
8Disallow: /includes/
9Disallow: /_td_api
10Disallow: /tdv2_fp
11Disallow: /nel_ms
12Disallow: /fp_ms
13Disallow: /sports_fp_ms
14Disallow: /search_ms
15Disallow: /_tdpp_api
16Disallow: /_remote
17Disallow: /_multiremote
18Disallow: /_tdhl_api
19Disallow: /digest
20Disallow: /fpjs
21Disallow: /myjs
22Disallow: /news/m/
23
24User-agent: ADmantX
25User-agent: AlphaBot
26User-agent: anthropic-ai
27User-agent: AwarioRssBot
28User-agent: AwarioSmartBot
29User-agent: BLEXBot
30User-agent: Buzzbot
31User-agent: Bytespider
32User-agent: CCBot
33User-agent: ChatGPT-User
34User-agent: claritybot
35User-agent: Claude-Web
36User-agent: ClaudeBot
37User-agent: cohere-ai
38User-agent: Diffbot
39User-agent: FacebookBot
40User-agent: FriendlyCrawler
41User-agent: Google-Extended
42User-agent: GPTBot
43User-agent: huggingface
44User-agent: ImagesiftBot
45User-agent: img2dataset
46User-agent: magpie-crawler
47User-agent: Meltwater
48User-agent: Neevabot
49User-agent: news-please
50User-agent: NewsNow
51User-agent: Nutch
52User-agent: omgili
53User-agent: omgilibot
54User-agent: panscient.com
55User-agent: Perplexity-ai
56User-agent: PerplexityBot
57User-agent: PetalBot
58User-agent: PiplBot
59User-agent: scoop.it
60User-agent: Scrapy
61User-agent: Seekr
62User-agent: SentiBot
63User-agent: SeznamBot
64User-agent: TurnitinBot
65User-agent: YouBot
66User-agent: ZumBot
67Disallow: /
68
69User-agent: Claude-SearchBot
70User-agent: OAI-SearchBot
71Disallow: */articles/
72

What do the bots actually get?

This page reads robots.txt, which states what yahoo.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on yahoo.com Test a path in the robots.txt tester

History

Questions

Does yahoo.com block GPTBot?

Yes. yahoo.com’s robots.txt disallows GPTBot from the whole site (line 67: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does yahoo.com block ClaudeBot?

Yes. yahoo.com’s robots.txt disallows ClaudeBot from the whole site (line 67: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does yahoo.com block Google-Extended?

Yes. yahoo.com’s robots.txt disallows Google-Extended from the whole site (line 67: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does yahoo.com block PerplexityBot?

Yes. yahoo.com’s robots.txt disallows PerplexityBot from the whole site (line 67: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find yahoo.com?

Partly. yahoo.com’s robots.txt keeps OAI-SearchBot out of some paths, such as */articles/, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.