Who blocks AI · #934 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does fool.com block AI crawlers?

Yes. fool.com’s robots.txt blocks 4 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, CCBot and Bytespider. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotBlocked OAI-SearchBotAllowed ClaudeBotBlocked PerplexityBotAllowed Google-ExtendedAllowed CCBotBlocked

robots.txt: www.fool.com/robots.txt · 113 lines
llms.txt: found
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Allowed	no rule matches the home page
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Allowed	no rule matches the home page
`PerplexityBot`	Perplexity	Allowed	no rule matches the home page
`Claude-SearchBot`	Anthropic	Allowed	no rule matches the home page
`DuckAssistBot`	DuckDuckGo	Allowed	no rule matches the home page
`Amazonbot`	Amazon	Allowed	no rule matches the home page
`YouBot`	You.com	Allowed	no rule matches the home page

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 4 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 7 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Allowed	no rule matches the home page
`CCBot`	Common Crawl	Blocked	line 10 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Allowed	no rule matches the home page
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Allowed	no rule matches the home page
`Bytespider`	ByteDance	Blocked	line 16 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Allowed	no rule matches the home page
`Claude-User`	Anthropic	Allowed	no rule matches the home page
`Perplexity-User`	Perplexity	Allowed	no rule matches the home page
`MistralAI-User`	Mistral	Allowed	no rule matches the home page
`Meta-ExternalFetcher`	Meta	Allowed	no rule matches the home page

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	no rule matches the home page
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View www.fool.com/robots.txt · 113 lines

1# /robots.txt file for http://www.fool.com/ (prod)

2# OpenAI (Model Training)

3User-agent: GPTBot

4Disallow: /

5# Anthropic (Model Training)

6User-agent: ClaudeBot

7Disallow: /

8# Common Crawl (AI Training Dataset)

9User-agent: CCBot

10Disallow: /

11# Sensika (Media Monitoring & Web Scraping)

12User-agent: MauiBot

13Disallow: /

14# ByteDance (TikTok / Content Crawling)

15User-agent: Bytespider

16Disallow: /

17# Cloudflare Content Signal

18User-agent: *

19Content-Signal: ai-train=no, search=yes, ai-input=yes

20Allow: /a/feeds/

21Allow: /ads.txt

22Sitemap: https://www.fool.com/sitemap-recent.xml

23Sitemap: https://www.fool.com/news-sitemap.xml

24Sitemap: https://www.fool.com/money/news-sitemap.xml

25Sitemap: https://www.fool.com/wag-sitemap.xml

26Sitemap: https://www.fool.com/quotes-sitemap.xml

27Sitemap: https://www.fool.com/money/sitemap.xml

28Sitemap: https://www.fool.com/hreflang-tag-sitemap.xml

29Sitemap: https://www.fool.com/sitemap/

30Disallow: /3910

31Disallow: /a/

32Disallow: /account/

33Disallow: /Account/

34Disallow: /admin

35Disallow: /Admin

36Disallow: /amptest/

37Disallow: /art/

38Disallow: /Art/

39Disallow: /auth/

40Disallow: /betaboards/

41Disallow: /cdn-cgi/

42Disallow: /Common/

43Disallow: /dubs/

44Disallow: /ecap/

45Disallow: /ext-content/

46Disallow: /external/

47Disallow: /feeds

48Disallow: /Feeds

49Disallow: /fool/free-report

50Disallow: /foolpics

51Disallow: /foolPics

52Disallow: /FoolPics

53Disallow: /free-report

54Disallow: /help

55Disallow: /Help

56Disallow: /includes

57Disallow: /Includes

58Disallow: /investing/businesswire/

59Disallow: /investing/fiercemarkets/

60Disallow: /investing/FierceMarkets/

61Disallow: /investing/stocks/

62Disallow: /investor-alert/

63Disallow: /localize

64Disallow: /Localize

65Disallow: /login/

66Disallow: /mailemergency

67Disallow: /mailEmergency

68Disallow: /MailEmergency

69Disallow: /Marketing/EcapSurvey.aspx

70Disallow: /mms

71Disallow: /news/commentary/

72Disallow: /news/xt

73Disallow: /News/Xt

74Disallow: /News/XT

75Disallow: /newsletters/*

76Disallow: /nexus/

77Disallow: /now50/

78Disallow: /offers/

79Disallow: /order-results*

80Disallow: /order/

81Disallow: /p/*

82Disallow: /p/60/

83Disallow: /partners

84Disallow: /Partners

85Disallow: /pegulator

86Disallow: /Pegulator

87Disallow: /premium-reports

88Disallow: /private

89Disallow: /Private

90Disallow: /quote/failed-lookup/

91Disallow: /quote/unknown/

92Disallow: /reports

93Disallow: /scripts

94Disallow: /Scripts

95Disallow: /secure/

96Disallow: /server

97Disallow: /Server

98Disallow: /shop/

99Disallow: /Shop/Download/Event/

100Disallow: /Shop/OrderResults.aspx

101Disallow: /Shop/OrderResults.aspx*

102Disallow: /Shop/OrderResults*

103Disallow: /snap

104Disallow: /Snap

105Disallow: /stock-ranker/

106Disallow: /test

107Disallow: /Test

108Disallow: /thank-you/

109Disallow: /the-ascent/search/

110Disallow: /video-alert

111Disallow: /vip/

112Disallow: /api/historical/

113

What do the bots actually get?

This page reads robots.txt, which states what fool.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on fool.com Test a path in the robots.txt tester

History

19 October 2026: first checked.
26 October 2026: robots.txt readable again.
2 November 2026: started blocking GPTBot; started blocking ClaudeBot; started blocking CCBot.

Questions

Does fool.com block GPTBot?

Yes. fool.com’s robots.txt disallows GPTBot from the whole site (line 4: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does fool.com block ClaudeBot?

Yes. fool.com’s robots.txt disallows ClaudeBot from the whole site (line 7: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does fool.com block Google-Extended?

No. fool.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does fool.com block PerplexityBot?

No. fool.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find fool.com?

Yes. fool.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.