Who blocks AI · #147 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does wsj.com block AI crawlers?
Yes. wsj.com’s robots.txt blocks 15 of the 19 AI crawlers we track, including ClaudeBot, Google-Extended, PerplexityBot and 12 more. It also blocks PetalBot and YandexBot among search engines.
GPTBotPartly blockedOAI-SearchBotPartly blockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.wsj.com/robots.txt · 158 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 90 · Disallow: /article_email/* line 91 · Disallow: /user/* and 47 more | |
Bingbot | Microsoft | Partly blocked | line 90 · Disallow: /article_email/* line 91 · Disallow: /user/* and 47 more |
DuckDuckBot | DuckDuckGo | Partly blocked | line 90 · Disallow: /article_email/* line 91 · Disallow: /user/* and 47 more |
Applebot | Apple | Partly blocked | line 90 · Disallow: /article_email/* line 91 · Disallow: /user/* and 47 more |
PetalBot | Huawei | Blocked | line 14 · Disallow: / under User-agent: * |
YandexBot | Yandex | Blocked | line 14 · Disallow: / under User-agent: * |
Baiduspider | Baidu | Partly blocked | line 90 · Disallow: /article_email/* line 91 · Disallow: /user/* and 47 more |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Partly blocked | line 90 · Disallow: /article_email/* line 91 · Disallow: /user/* and 47 more |
PerplexityBot | Perplexity | Blocked | line 14 · Disallow: / under User-agent: * |
Claude-SearchBot | Anthropic | Blocked | line 14 · Disallow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Blocked | line 14 · Disallow: / under User-agent: * |
Amazonbot | Amazon | Blocked | line 14 · Disallow: / under User-agent: * |
YouBot | You.com | Blocked | line 14 · Disallow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Partly blocked | line 90 · Disallow: /article_email/* line 91 · Disallow: /user/* and 47 more |
ClaudeBot | Anthropic | Blocked | line 14 · Disallow: / under User-agent: * |
Google-Extended | Blocked | line 14 · Disallow: / under User-agent: * | |
CCBot | Common Crawl | Blocked | line 14 · Disallow: / under User-agent: * |
Applebot-Extended | Apple | Blocked | line 14 · Disallow: / under User-agent: * |
GoogleOther | Partly blocked | line 90 · Disallow: /article_email/* line 91 · Disallow: /user/* and 47 more | |
Meta-ExternalAgent | Meta | Blocked | line 14 · Disallow: / under User-agent: * |
Bytespider | ByteDance | Blocked | line 14 · Disallow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Partly blocked | line 90 · Disallow: /article_email/* line 91 · Disallow: /user/* and 47 more |
Claude-User | Anthropic | Blocked | line 14 · Disallow: / under User-agent: * |
Perplexity-User | Perplexity | Blocked | line 14 · Disallow: / under User-agent: * |
MistralAI-User | Mistral | Blocked | line 14 · Disallow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Blocked | line 14 · Disallow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Partly blocked | line 90 · Disallow: /article_email/* line 91 · Disallow: /user/* and 47 more |
SemrushBot | Semrush | Partly blocked | line 90 · Disallow: /article_email/* line 91 · Disallow: /user/* and 47 more |
MJ12bot | Majestic | Blocked | line 14 · Disallow: / under User-agent: * |
DotBot | Moz | Blocked | line 14 · Disallow: / under User-agent: * |
View www.wsj.com/robots.txt · 158 lines
1# NOTICE: Collection of content and other data on https://www.wsj.com/ through
2# automated means is prohibited unless you have express written
3# permission from Dow Jones & Company, Inc. and may only be conducted for the
4# limited purpose contained in said permission.
5#
6# Dow Jones & Company, Inc. Terms of Use may be found at
7# https://www.dowjones.com/terms-of-use/
8#
9# If you would like to apply for permission to license the
10# intellectual property and/or other materials of Dow Jones & Company, Inc.’s
11# brands, please contact us via email at copyright@dowjones.com.
12
13User-agent: *
14Disallow: /
15
16User-agent: googlebot
17User-agent: googlebot-image
18User-agent: GoogleOther
19User-agent: Googlebot-Video
20User-agent: Google-InspectionTool
21User-agent: AdsBot-Google
22User-agent: AdsBot-Google-Mobile
23User-agent: AdsBot-Google-Mobile-Apps
24User-agent: Storebot-Google
25User-agent: google-read-aloud
26User-agent: bingbot
27User-agent: msnbot
28User-agent: bingpreview
29User-agent: slurp
30User-agent: yahoo
31User-agent: baiduspider
32User-agent: Pinterestbot
33User-agent: Yeti
34User-agent: MojeekBot
35User-agent: 360Spider
36User-agent: google-cloudvertexbot
37User-agent: duckduckbot
38User-agent: Applebot
39User-agent: flipboard
40User-agent: qwantbot
41User-agent: SeznamBot
42
43
44
45User-agent: proximic
46User-agent: admantx
47User-agent: thetradedesk
48User-agent: outbrain
49User-agent: ias_crawler
50User-agent: AmazonAdBot
51User-agent: pubmatic
52User-agent: smartologybot
53User-agent: parselybot
54User-agent: BomboraBot
55User-agent: IllumaBot/1.0
56User-agent: Publication-Access-for-Facebook
57User-agent: Wallie
58User-agent: mediapartners-google
59User-agent: NewsRoom.BI
60
61User-agent: Screaming Frog SEO Spider
62User-agent: AhrefsBot
63User-agent: SemrushBot
64User-agent: SimilarWebBot
65User-agent: SISTRIX
66User-agent: botify
67User-agent: Chrome-Lighthouse
68
69
70
71User-agent: ChatGPT-User
72User-agent: GPTBot
73User-agent: OAI-SearchBot
74
75
76User-agent: facebookexternalhit
77User-agent: facebot
78User-agent: twitterbot
79User-agent: linkedinbot
80User-agent: snapchat
81User-agent: sentry
82User-agent: Iframely
83User-agent: Vocabtracker
84User-agent: EpvzCrawl6194680250
85User-agent: Citoid
86User-agent: ZoteroTranslationServer
87
88
89Allow: /
90Disallow: /article_email/*
91Disallow: /user/*
92Disallow: /pdf/documents/*
93Disallow: /login/*
94Disallow: /acct/*
95Disallow: /msgcenter/*
96Disallow: /setup/*
97Disallow: /marketing/*
98Disallow: /public/article/*
99Disallow: /public/resources/documents/*
100Disallow: /public/search/
101Disallow: /public/search*
102Disallow: /search*
103Disallow: /public/page/wsj-x-marketing.html
104Disallow: /public/page/news-media-marketing.html
105Disallow: /public/page/0_0_WP_RT_MARKETING.html
106Disallow: /news/articles/SB2*
107Disallow: /news/articles/SB3*
108Disallow: /news/articles/SB4*
109Disallow: /articles/SB2*
110Disallow: /articles/SB3*
111Disallow: /articles/SB4*
112Disallow: /article/AP*
113Disallow: /article/BT-CO*
114Disallow: /article/DN-CO*
115Disallow: /article/PR-CO*
116Disallow: /article/HUG*
117Disallow: /video/search/*
118Disallow: /articles/BT-CO*
119Disallow: /articles/DN-CO*
120Disallow: /articles/PR-CO*
121Disallow: /news/articles/BT-CO*
122Disallow: /news/articles/DN-CO*
123Disallow: /news/articles/PR-CO*
124Disallow: /catchup/*
125Disallow: /articles/the-meaning-behind-juneteenth-11592413234
126Disallow: /emailservice/*
127Disallow: /emailsignup/*
128Disallow: /insetsrv/v1/*
129Disallow: /user/fpd/api/*
130Disallow: /Date(*
131Disallow: /auth/sso/proxy-login*
132Disallow: /client/
133
134# For Buyside Search Results
135Disallow: /buyside/search-results?*term=*
136# Don't crawl non-indexable sites
137Disallow: /*?type=mdc_*&id=*
138Disallow: /*?id=*&type=mdc_*
139Disallow: /market-data/quotes/*/options/*
140Disallow: /subscribe/?inttrackingCode=*
141Disallow: /subscribe/?template=*
142
143
144Sitemap: https://www.wsj.com/sitemap.xml
145Sitemap: https://www.wsj.com/wsjsitemaps/wsj_google_news.xml
146Sitemap: https://www.wsj.com/wsj_video_recent.xml
147Sitemap: https://www.wsj.com/sitemap_topics.xml
148Sitemap: https://www.wsj.com/sitemaps/web/wsj/en/sitemap_wsj_en_index.xml
149Sitemap: https://www.wsj.com/live_news_sitemap.xml
150Sitemap: https://www.wsj.com/authors_sitemap.xml
151Sitemap: https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml
152Sitemap: https://www.wsj.com/buyside/sitemap.xml
153Sitemap: https://www.wsj.com/wsj_quote_index_sitemap.xml
154Sitemap: https://www.wsj.com/wsjsitemaps/wsj_recipes.xml
155Sitemap: https://www.wsj.com/sitemap_topic_collections.xml
156
157
158
What do the bots actually get?
This page reads robots.txt, which states what wsj.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on wsj.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does wsj.com block GPTBot?
Partly. wsj.com’s robots.txt keeps GPTBot out of some paths, such as /article_email/*, and lets it crawl the rest. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does wsj.com block ClaudeBot?
Yes. wsj.com’s robots.txt disallows ClaudeBot from the whole site (line 14: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does wsj.com block Google-Extended?
Yes. wsj.com’s robots.txt disallows Google-Extended from the whole site (line 14: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does wsj.com block PerplexityBot?
Yes. wsj.com’s robots.txt disallows PerplexityBot from the whole site (line 14: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find wsj.com?
Partly. wsj.com’s robots.txt keeps OAI-SearchBot out of some paths, such as /article_email/*, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.