Who blocks AI · #116 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does wiley.com block AI crawlers?
Yes. wiley.com’s robots.txt blocks 9 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 6 more. It also blocks PetalBot and Baiduspider among search engines.
GPTBotBlockedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotAllowedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.wiley.com/robots.txt · 193 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | line 32 · Allow: / under User-agent: * | |
Bingbot | Microsoft | Allowed | line 32 · Allow: / under User-agent: * |
DuckDuckBot | DuckDuckGo | Allowed | line 32 · Allow: / under User-agent: * |
Applebot | Apple | Allowed | line 32 · Allow: / under User-agent: * |
PetalBot | Huawei | Blocked | line 84 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | line 32 · Allow: / under User-agent: * |
Baiduspider | Baidu | Blocked | line 81 · Disallow: / under User-agent: Baiduspider |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | line 32 · Allow: / under User-agent: * |
PerplexityBot | Perplexity | Allowed | line 32 · Allow: / under User-agent: * |
Claude-SearchBot | Anthropic | Allowed | line 32 · Allow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Allowed | line 32 · Allow: / under User-agent: * |
Amazonbot | Amazon | Blocked | line 36 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Allowed | line 32 · Allow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 57 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 48 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 54 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 45 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 39 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Blocked | line 78 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Blocked | line 75 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 42 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | line 32 · Allow: / under User-agent: * |
Claude-User | Anthropic | Allowed | line 32 · Allow: / under User-agent: * |
Perplexity-User | Perplexity | Allowed | line 32 · Allow: / under User-agent: * |
MistralAI-User | Mistral | Allowed | line 32 · Allow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Allowed | line 32 · Allow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 166 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Allowed | line 32 · Allow: / under User-agent: * |
MJ12bot | Majestic | Allowed | line 32 · Allow: / under User-agent: * |
DotBot | Moz | Blocked | line 157 · Disallow: / under User-agent: dotbot |
View www.wiley.com/robots.txt · 193 lines
1# As a condition of accessing this website, you agree to abide by the following
2# content signals:
3
4# (a) If a Content-Signal = yes, you may collect content for the corresponding
5# use.
6# (b) If a Content-Signal = no, you may not collect content for the
7# corresponding use.
8# (c) If the website operator does not include a Content-Signal for a
9# corresponding use, the website operator neither grants nor restricts
10# permission via Content-Signal with respect to the corresponding use.
11
12# The content signals and their meanings are:
13
14# search: building a search index and providing search results (e.g., returning
15# hyperlinks and short excerpts from your website's contents). Search does not
16# include providing AI-generated search summaries.
17# ai-input: inputting content into one or more AI models (e.g., retrieval
18# augmented generation, grounding, or other real-time taking of content for
19# generative AI search answers).
20# ai-train: training or fine-tuning AI models.
21# use: how AI systems may consume the content (immediate, reference, or full).
22
23# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
24# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
25# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
26
27# BEGIN Cloudflare Managed content
28
29
30User-agent: *
31Content-Signal: search=yes,ai-train=no,use=reference
32Allow: /
33
34# Training crawlers
35User-agent: Amazonbot
36Disallow: /
37
38User-agent: Applebot-Extended
39Disallow: /
40
41User-agent: Bytespider
42Disallow: /
43
44User-agent: CCBot
45Disallow: /
46
47User-agent: ClaudeBot
48Disallow: /
49
50User-agent: Diffbot
51Disallow: /
52
53User-agent: Google-Extended
54Disallow: /
55
56User-agent: GPTBot
57Disallow: /
58
59User-agent: omgili
60Disallow: /
61
62User-agent: anthropic-ai
63Disallow: /
64
65User-agent: Claude-Web
66Disallow: /
67
68User-agent: cohere-ai
69Disallow: /
70
71User-agent: MistralAI-Training
72Disallow: /
73
74User-agent: meta-externalagent
75Disallow: /
76
77User-agent: GoogleOther
78Disallow: /
79
80User-agent: Baiduspider
81Disallow: /
82
83User-agent: PetalBot
84Disallow: /
85
86User-agent: AwarioSmartBot
87User-agent: AwarioRssBot
88Disallow: /
89
90User-agent: Google-CloudVertexBot
91Disallow: /
92
93User-agent: QualifiedBot
94Disallow: /
95
96User-agent: Cotoyogi
97Disallow: /
98
99User-agent: ICC-Crawler
100Disallow: /
101
102User-agent: atlassian-bot
103Disallow: /
104
105User-agent: FishBot
106Disallow: /
107
108User-agent: BorderxBot
109Disallow: /
110
111User-agent: NavuBot
112Disallow: /
113
114User-agent: SemrushBot-SWA
115Disallow: /
116
117User-agent: WARDBot
118Disallow: /
119
120User-agent: magpie-crawler
121Disallow: /
122
123User-agent: KimiBot
124Disallow: /
125
126User-agent: CitibotSiteCrawler
127Disallow: /
128
129# END Cloudflare Managed Content
130
131# For all robots
132User-agent: *
133Allow: /
134
135# Allow search crawlers to discover the sitemap
136Sitemap: https://www.wiley.com/sitemap.xml
137
138# Block access to specific groups of pages
139Disallow: /*/cart/
140Disallow: /*/checkout/
141Disallow: /*/my-account/
142Disallow: /*/school/
143
144# Block common duplicate/low-value parameterised URLs
145Disallow: /*?sessionid=
146Disallow: /*?ref=
147Disallow: /*?utm_
148
149# Block unwanted crawlers
150User-agent: CazoodleBot
151Disallow: /
152
153User-agent: MJ12bot
154Disallow: /
155
156User-agent: dotbot
157Disallow: /
158
159User-agent: Gigabot
160Disallow: /
161
162User-agent: trendkite-akashic-crawler
163Disallow: /
164
165User-agent: AhrefsBot
166Disallow: /
167
168User-agent: MauiBot
169Disallow: /
170
171User-agent: Barkrowler
172Disallow: /
173
174
175# Account Pages
176Disallow: /login
177Disallow: /register
178Disallow: /wishlist
179Disallow: /order-confirmation
180
181# Search Pages
182Disallow: /search
183Disallow: /*?q=
184Disallow: /*?query=
185
186# URL Parameters
187Disallow: /*?sort=
188Disallow: /*?view=
189Disallow: /*?pageSize=
190
191# Administrative and API Paths
192Disallow: /admin/
193Disallow: /api/
What do the bots actually get?
This page reads robots.txt, which states what wiley.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on wiley.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
- 26 October 2026: started blocking PetalBot; started blocking Baiduspider; started blocking Amazonbot; started blocking GPTBot; started blocking ClaudeBot; started blocking Google-Extended; started blocking CCBot; started blocking Applebot-Extended; started blocking GoogleOther; started blocking Meta-ExternalAgent; started blocking Bytespider.
Questions
Does wiley.com block GPTBot?
Yes. wiley.com’s robots.txt disallows GPTBot from the whole site (line 57: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does wiley.com block ClaudeBot?
Yes. wiley.com’s robots.txt disallows ClaudeBot from the whole site (line 48: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does wiley.com block Google-Extended?
Yes. wiley.com’s robots.txt disallows Google-Extended from the whole site (line 54: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does wiley.com block PerplexityBot?
No. wiley.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find wiley.com?
Yes. wiley.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.