Who blocks AI · #565 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does marketwatch.com block AI crawlers?
Yes. marketwatch.com’s robots.txt blocks 15 of the 19 AI crawlers we track, including ClaudeBot, Google-Extended, PerplexityBot and 12 more. It also blocks YandexBot among search engines.
GPTBotPartly blockedOAI-SearchBotPartly blockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.marketwatch.com/robots.txt · 176 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 107 · Disallow: /2/ line 108 · Disallow: /3com/ and 54 more | |
Bingbot | Microsoft | Partly blocked | line 107 · Disallow: /2/ line 108 · Disallow: /3com/ and 51 more |
DuckDuckBot | DuckDuckGo | Partly blocked | line 107 · Disallow: /2/ line 108 · Disallow: /3com/ and 51 more |
Applebot | Apple | Partly blocked | line 107 · Disallow: /2/ line 108 · Disallow: /3com/ and 51 more |
PetalBot | Huawei | Partly blocked | line 107 · Disallow: /2/ line 108 · Disallow: /3com/ and 51 more |
YandexBot | Yandex | Blocked | line 15 · Disallow: / under User-agent: * |
Baiduspider | Baidu | Partly blocked | line 107 · Disallow: /2/ line 108 · Disallow: /3com/ and 51 more |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Partly blocked | line 107 · Disallow: /2/ line 108 · Disallow: /3com/ and 51 more |
PerplexityBot | Perplexity | Blocked | line 15 · Disallow: / under User-agent: * |
Claude-SearchBot | Anthropic | Blocked | line 15 · Disallow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Blocked | line 15 · Disallow: / under User-agent: * |
Amazonbot | Amazon | Blocked | line 15 · Disallow: / under User-agent: * |
YouBot | You.com | Blocked | line 15 · Disallow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Partly blocked | line 107 · Disallow: /2/ line 108 · Disallow: /3com/ and 51 more |
ClaudeBot | Anthropic | Blocked | line 15 · Disallow: / under User-agent: * |
Google-Extended | Blocked | line 15 · Disallow: / under User-agent: * | |
CCBot | Common Crawl | Blocked | line 15 · Disallow: / under User-agent: * |
Applebot-Extended | Apple | Blocked | line 15 · Disallow: / under User-agent: * |
GoogleOther | Partly blocked | line 107 · Disallow: /2/ line 108 · Disallow: /3com/ and 51 more | |
Meta-ExternalAgent | Meta | Blocked | line 15 · Disallow: / under User-agent: * |
Bytespider | ByteDance | Blocked | line 15 · Disallow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Partly blocked | line 107 · Disallow: /2/ line 108 · Disallow: /3com/ and 51 more |
Claude-User | Anthropic | Blocked | line 15 · Disallow: / under User-agent: * |
Perplexity-User | Perplexity | Blocked | line 15 · Disallow: / under User-agent: * |
MistralAI-User | Mistral | Blocked | line 15 · Disallow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Blocked | line 15 · Disallow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Partly blocked | line 107 · Disallow: /2/ line 108 · Disallow: /3com/ and 51 more |
SemrushBot | Semrush | Partly blocked | line 107 · Disallow: /2/ line 108 · Disallow: /3com/ and 51 more |
MJ12bot | Majestic | Blocked | line 15 · Disallow: / under User-agent: * |
DotBot | Moz | Blocked | line 15 · Disallow: / under User-agent: * |
View www.marketwatch.com/robots.txt · 176 lines
1# NOTICE: Collection of content and other data on https://www.marketwatch.com/ through
2# automated means is prohibited unless you have express written
3# permission from Dow Jones & Company, Inc. and may only be conducted for the
4# limited purpose contained in said permission.
5#
6# Dow Jones & Company, Inc. Terms of Use may be found at
7# https://www.dowjones.com/terms-of-use/
8#
9# If you would like to apply for permission to license the
10# intellectual property and/or other materials of Dow Jones & Company, Inc.’s
11# brands, please contact us via email at copyright@dowjones.com.
12
13
14User-agent: *
15Disallow: /
16
17User-agent: googlebot
18User-agent: googlebot-image
19User-agent: GoogleOther
20User-agent: Googlebot-Video
21User-agent: Google-InspectionTool
22User-agent: AdsBot-Google
23User-agent: AdsBot-Google-Mobile
24User-agent: AdsBot-Google-Mobile-Apps
25User-agent: Storebot-Google
26User-agent: google-read-aloud
27User-agent: bingbot
28User-agent: msnbot
29User-agent: bingpreview
30User-agent: slurp
31User-agent: yahoo
32User-agent: yandex
33User-agent: YandexImages
34User-agent: baiduspider
35User-agent: Pinterestbot
36User-agent: Yeti
37User-agent: MojeekBot
38User-agent: 360Spider
39User-agent: google-cloudvertexbot
40User-agent: duckduckbot
41User-agent: petalbot
42User-agent: Applebot
43User-agent: AppleNewsBot
44User-agent: flipboard
45User-agent: qwantbot
46User-agent: SeznamBot
47User-agent: yisouspider
48
49
50User-agent: proximic
51User-agent: admantx
52User-agent: thetradedesk
53User-agent: outbrain
54User-agent: ias_crawler
55User-agent: AmazonAdBot
56User-agent: pubmatic
57User-agent: smartologybot
58User-agent: parselybot
59User-agent: IllumaBot/1.0
60User-agent: BomboraBot
61User-agent: IllumaBot/1.0
62User-agent: Publication-Access-for-Facebook
63User-agent: Wallie
64User-agent: mediapartners-google
65User-agent: NewsRoom.BI
66
67
68User-agent: Screaming Frog SEO Spider
69User-agent: DeepCrawl
70User-agent: AhrefsBot
71User-agent: SemrushBot
72User-agent: SimilarWebBot
73User-agent: SISTRIX
74User-agent: OnCrawl
75User-agent: botify
76User-agent: Chrome-Lighthouse
77
78
79
80User-agent: ChatGPT-User
81User-agent: GPTBot
82User-agent: OAI-SearchBot
83
84
85User-agent: facebookexternalhit
86User-agent: facebot
87User-agent: twitterbot
88User-agent: linkedinbot
89User-agent: telegrambot
90User-agent: snapchat
91User-agent: newrelicpinger
92User-agent: siteimprovebot
93User-agent: uptimerobot
94User-agent: pingdom.com_bot_version_1.4
95User-agent: sentry
96User-agent: schema-validator
97User-agent: googleimageproxy
98User-agent: Iframely
99User-agent: Vocabtracker
100User-agent: EpvzCrawl6194680250
101User-agent: Citoid
102User-agent: ZoteroTranslationServer
103
104
105Allow: /
106
107Disallow: /2/
108Disallow: /3com/
109Disallow: /admin/
110Disallow: /bgpoxy/
111Disallow: /bin/
112Disallow: /cDN_contENt/
113Disallow: /cgi-bin/
114Disallow: /client/
115Disallow: /data/
116Disallow: /dbcfiles/
117Disallow: /dhtml/
118Disallow: /dhtmlmenu/
119Disallow: /doubleclick/
120Disallow: /dynamiclogic/
121Disallow: /email/
122Disallow: /feeds/
123Disallow: /free/
124Disallow: /http2_data/
125Disallow: /game/*
126Disallow: /games/*
127Disallow: /*?tickers=
128Disallow: /*?submit=
129Disallow: /*?exchange=
130Disallow: /login/
131Disallow: /MarketPerception/
132Disallow: /marketperception/
133Disallow: /MyPortfolio/
134Disallow: /myportfolio/
135Disallow: /news/current/
136Disallow: /portfolio/
137Disallow: /print/
138Disallow: /RealMedia/
139Disallow: /realmedia/
140Disallow: /registration/
141Disallow: /search/
142Disallow: /search
143Disallow: /story/twittershare
144Disallow: /tools/alerts/
145Disallow: /tpx/
146Disallow: /investing/secfile/
147Disallow: /tools/quotes/secfilings.asp
148Disallow: /tools/quotes/form10k.asp
149Disallow: /tools/quotes/form10q.asp
150Disallow: /tools/quotes/form8k.asp
151Disallow: /tools/quotes/registrationstate.asp
152Disallow: /tools/quotes/sched13d.asp
153Disallow: /tools/quotes/sched13f.asp
154Disallow: /tools/quotes/sched13g.asp
155Disallow: /tools/quotes/secarticle.asp
156Disallow: /wsj/
157Disallow: /xhtml/v/
158Disallow: /user/login/profile
159Disallow: /auth/
160
161User-agent: googlebot
162Disallow: /client
163Disallow: /subscribe/?inttrackingCode=*
164Disallow: /subscribe/?template=*
165
166
167
168Sitemap: https://www.marketwatch.com/sitemaps/web/mw/en/sitemap_mw_en_index.xml
169Sitemap: https://www.marketwatch.com/sitemaps/web/mw-video/en/sitemap_mw-video_en_index.xml
170Sitemap: https://www.marketwatch.com/mw_news_sitemap.xml
171Sitemap: https://www.marketwatch.com/mw_live_news_sitemap.xml
172Sitemap: https://www.marketwatch.com/api-video/sitemaps/google/sitemap-google-video-marketwatch.asp
173Sitemap: https://www.marketwatch.com/mw/mw_quote_index_sitemap.xml
174
175
176
What do the bots actually get?
This page reads robots.txt, which states what marketwatch.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on marketwatch.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does marketwatch.com block GPTBot?
Partly. marketwatch.com’s robots.txt keeps GPTBot out of some paths, such as /2/, and lets it crawl the rest. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does marketwatch.com block ClaudeBot?
Yes. marketwatch.com’s robots.txt disallows ClaudeBot from the whole site (line 15: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does marketwatch.com block Google-Extended?
Yes. marketwatch.com’s robots.txt disallows Google-Extended from the whole site (line 15: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does marketwatch.com block PerplexityBot?
Yes. marketwatch.com’s robots.txt disallows PerplexityBot from the whole site (line 15: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find marketwatch.com?
Partly. marketwatch.com’s robots.txt keeps OAI-SearchBot out of some paths, such as /2/, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.