Who blocks AI · #962 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does yumpu.com block AI crawlers?

Yes. yumpu.com’s robots.txt blocks 11 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 8 more. It also blocks YandexBot among search engines.

GPTBotBlocked OAI-SearchBotAllowed ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: yumpu.com/robots.txt · 318 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Partly blocked	line 93 · `Disallow: /document/forward/` line 94 · `Disallow: /*/document/forward/` and 60 more
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Allowed	no rule matches the home page
`YandexBot`	Yandex	Blocked	line 185 · `Disallow: /` under `User-agent: YandexBot`
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Allowed	no rule matches the home page
`PerplexityBot`	Perplexity	Blocked	line 271 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Allowed	no rule matches the home page
`DuckAssistBot`	DuckDuckGo	Allowed	no rule matches the home page
`Amazonbot`	Amazon	Blocked	line 231 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Allowed	no rule matches the home page

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 191 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 196 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Blocked	line 205 · `Disallow: /` under `User-agent: Google-Extended`
`CCBot`	Common Crawl	Blocked	line 217 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 208 · `Disallow: /` under `User-agent: Applebot-Extended`
`GoogleOther`	Google	Blocked	line 205 · `Disallow: /` under `User-agent: GoogleOther`
`Meta-ExternalAgent`	Meta	Blocked	line 214 · `Disallow: /` under `User-agent: meta-externalagent`
`Bytespider`	ByteDance	Blocked	line 221 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Allowed	no rule matches the home page
`Claude-User`	Anthropic	Allowed	no rule matches the home page
`Perplexity-User`	Perplexity	Allowed	no rule matches the home page
`MistralAI-User`	Mistral	Allowed	no rule matches the home page
`Meta-ExternalFetcher`	Meta	Blocked	line 214 · `Disallow: /` under `User-agent: Meta-ExternalFetcher`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 309 · `Disallow: /` under `User-agent: AhrefsBot`
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Blocked	line 309 · `Disallow: /` under `User-agent: DotBot`

View yumpu.com/robots.txt · 318 lines

1# Served by CloudFront from the aws-infrastructure repository

2# (stacks/waf-www/robots.txt). Edit it there, not in the application.

4# Search + AI citation + user-triggered = allow. AI training = deny.

5# Groups do not merge (RFC 9309): a crawler matching its own group ignores "*"

6# and loses every Disallow. Allowed crawlers get NO group, with one exception:

7# bingbot has its own group for Crawl-delay, a full copy of the "*" rules.

8# Change a rule in "*" and make the same change in the bingbot group.

9# "/*/path/" covers language prefixes and /index.php/; both forms are needed.

11User-agent: *

13Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference

15Disallow: /document/forward/

16Disallow: /*/document/forward/

17Disallow: /document/forward_mail/

18Disallow: /*/document/forward_mail/

19Disallow: /document/inappropriate/

20Disallow: /*/document/inappropriate/

21Disallow: /document/embed_generator_code/

22Disallow: /*/document/embed_generator_code/

23Disallow: /document/embed_generator/

24Disallow: /*/document/embed_generator/

25Disallow: /document/readers/

26Disallow: /*/document/readers/

27Disallow: /document/rating/

28Disallow: /*/document/rating/

29Disallow: /document/search_text/

30Disallow: /*/document/search_text/

31Disallow: /document/search_words/

32Disallow: /*/document/search_words/

33Disallow: /document/search/

34Disallow: /*/document/search/

35Disallow: /document/tag/

36Disallow: /*/document/tag/

37Disallow: /document/progress/

38Disallow: /*/document/progress/

39Disallow: /document/create/

40Disallow: /*/document/create/

41Disallow: /document/edit/

42Disallow: /*/document/edit/

43Disallow: /document/delete/

44Disallow: /*/document/delete/

45Disallow: /document/view_more/

46Disallow: /*/document/view_more/

47Disallow: /document/views/

48Disallow: /*/document/views/

49Disallow: /document/comments/

50Disallow: /*/document/comments/

51Disallow: /embed/view/

52Disallow: /*/embed/view/

53Disallow: /utils/

54Disallow: /*/utils/

55Disallow: /bookshelf/

56Disallow: /*/bookshelf/

57Disallow: /browse/more_page/

58Disallow: /*/browse/more_page/

59Disallow: /magazine/get_comment/

60Disallow: /*/magazine/get_comment/

61Disallow: /upload/forgot_password/

62Disallow: /*/upload/forgot_password/

63Disallow: /upload/cloud/

64Disallow: /*/upload/cloud/

65Disallow: /upload/analyse/

66Disallow: /*/upload/analyse/

67Disallow: /document/download/

68Disallow: /*/document/download/

69Disallow: /user/connect/

70Disallow: /*/user/connect/

71Disallow: /user/social_auth/

72Disallow: /*/user/social_auth/

73Disallow: /services/oembed

74Disallow: /*/services/oembed

75Disallow: /tracking.php

76Disallow: /tracking/

79# --- BINGBOT: the "*" rules plus Crawl-delay (2026-10-08) -------------------

80# Throttled, not denied: 3.3 million requests a day against 1,373 visits sent.

81# Bing fetches at most one page per Crawl-delay window (seconds, 1 to 30), so

82# 1 caps it at 86,400 pages a day. This overrides Crawl Control in Bing

83# Webmaster Tools.

84# KEEP IN SYNC: everything below Crawl-delay is a copy of the "*" group above.

85# Bingbot reads only this group; a rule missing here is a rule Bing ignores.

87User-agent: bingbot

89Crawl-delay: 1

91Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference

93Disallow: /document/forward/

94Disallow: /*/document/forward/

95Disallow: /document/forward_mail/

96Disallow: /*/document/forward_mail/

97Disallow: /document/inappropriate/

98Disallow: /*/document/inappropriate/

99Disallow: /document/embed_generator_code/

100Disallow: /*/document/embed_generator_code/

101Disallow: /document/embed_generator/

102Disallow: /*/document/embed_generator/

103Disallow: /document/readers/

104Disallow: /*/document/readers/

105Disallow: /document/rating/

106Disallow: /*/document/rating/

107Disallow: /document/search_text/

108Disallow: /*/document/search_text/

109Disallow: /document/search_words/

110Disallow: /*/document/search_words/

111Disallow: /document/search/

112Disallow: /*/document/search/

113Disallow: /document/tag/

114Disallow: /*/document/tag/

115Disallow: /document/progress/

116Disallow: /*/document/progress/

117Disallow: /document/create/

118Disallow: /*/document/create/

119Disallow: /document/edit/

120Disallow: /*/document/edit/

121Disallow: /document/delete/

122Disallow: /*/document/delete/

123Disallow: /document/view_more/

124Disallow: /*/document/view_more/

125Disallow: /document/views/

126Disallow: /*/document/views/

127Disallow: /document/comments/

128Disallow: /*/document/comments/

129Disallow: /embed/view/

130Disallow: /*/embed/view/

131Disallow: /utils/

132Disallow: /*/utils/

133Disallow: /bookshelf/

134Disallow: /*/bookshelf/

135Disallow: /browse/more_page/

136Disallow: /*/browse/more_page/

137Disallow: /magazine/get_comment/

138Disallow: /*/magazine/get_comment/

139Disallow: /upload/forgot_password/

140Disallow: /*/upload/forgot_password/

141Disallow: /upload/cloud/

142Disallow: /*/upload/cloud/

143Disallow: /upload/analyse/

144Disallow: /*/upload/analyse/

145Disallow: /document/download/

146Disallow: /*/document/download/

147Disallow: /user/connect/

148Disallow: /*/user/connect/

149Disallow: /user/social_auth/

150Disallow: /*/user/social_auth/

151Disallow: /services/oembed

152Disallow: /*/services/oembed

153Disallow: /tracking.php

154Disallow: /tracking/

155

156

157# --- ALLOWED: inherit "*" above. Do not give any of these a group. ----------

158#

159# Search: Googlebot, Googlebot-Image, Googlebot-News, Googlebot-Video,

160# Google-InspectionTool, Storebot-Google, Applebot, DuckDuckBot,

161# Bravebot, Slurp, Yeti, Qwantbot, Mojeekbot,

162# PetalBot, Amzn-SearchBot

163# AI citation: OAI-SearchBot, Claude-SearchBot, DuckAssistBot,

164# ExaSearchBot, YouBot

165# User-triggered: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User,

166# Amzn-User, Kimi-User, TongyiBot, kagi-fetcher, Google-NotebookLM,

167# GoogleAgent-URLContext, Gemini-Deep-Research, LinerBot, PhindBot

168# Link previews: facebookexternalhit, Twitterbot, LinkedInBot, Slackbot,

169# WhatsApp, Discordbot, TelegramBot

170# Ads: AdsBot-Google, OAI-AdsBot, Meta-ExternalAds, Mediapartners-Google

171# Ad verification and contextual targeting (buyers exclude pages these cannot

172# classify): CriteoBot, peer39_crawler, proximic, SirdataBot, OGDWCtxCrawler,

173# DVbot, Leikibot, ias_crawler, ias_wombles, ADmantX

174# SEO tools: SemrushBot, SiteAuditBot

175

176

177# --- SEARCH ENGINES TAKEN OUT (2026-10-08): DENIED --------------------------

178# Out of proportion: requests fetched against visits sent. The web ACL blocks

179# the same names (lists/blocked-bots.yaml).

180

181User-agent: Yandex

182User-agent: YandexBot

183User-agent: SeznamBot

184User-agent: ShapBot

185Disallow: /

186

187

188# --- AI TRAINING: DENIED ----------------------------------------------------

189

190User-agent: GPTBot

191Disallow: /

192

193User-agent: anthropic-ai

194User-agent: ClaudeBot

195User-agent: Claude-Web

196Disallow: /

197

198User-agent: Google-Extended

199User-agent: Google-CloudVertexBot

200User-agent: CloudVertexBot

201User-agent: Google-Firebase

202User-agent: GoogleOther

203User-agent: GoogleOther-Image

204User-agent: GoogleOther-Video

205Disallow: /

206

207User-agent: Applebot-Extended

208Disallow: /

209

210User-agent: meta-externalagent

211User-agent: Meta-ExternalAgent

212User-agent: Meta-ExternalFetcher

213User-agent: FacebookBot

214Disallow: /

215

216User-agent: CCBot

217Disallow: /

218

219User-agent: Bytespider

220User-agent: TikTokSpider

221Disallow: /

222

223User-agent: YandexAdditional

224User-agent: YandexAdditionalBot

225Disallow: /

226

227User-agent: Amazonbot

228User-agent: bedrockbot

229User-agent: amazon-kendra

230User-agent: amazon-QBusiness

231Disallow: /

232

233User-agent: ImagesiftBot

234User-agent: img2dataset

235User-agent: imageSpider

236Disallow: /

237

238User-agent: LAIONDownloader

239User-agent: laion-huggingface-processor

240Disallow: /

241

242User-agent: AI2Bot

243User-agent: Ai2Bot-Dolma

244User-agent: aiHitBot

245User-agent: Brightbot 1.0

246User-agent: ChatGLM-Spider

247User-agent: cohere-ai

248User-agent: cohere-training-data-crawler

249User-agent: Cotoyogi

250User-agent: Datenbank Crawler

251User-agent: DeepSeekBot

252User-agent: Diffbot

253User-agent: Factset_spyderbot

254User-agent: FriendlyCrawler

255User-agent: iaskspider

256User-agent: iaskspider/2.0

257User-agent: ICC-Crawler

258User-agent: Kangaroo Bot

259User-agent: netEstate Imprint Crawler

260User-agent: omgili

261User-agent: omgilibot

262User-agent: PanguBot

263User-agent: Panscient

264User-agent: panscient.com

265User-agent: PerplexityBot

266User-agent: SBIntuitionsBot

267User-agent: Sidetrade indexer bot

268User-agent: Timpibot

269User-agent: VelenPublicWebCrawler

270User-agent: Webzio-Extended

271Disallow: /

272

273User-agent: ApifyBot

274User-agent: ApifyWebsiteContentCrawler

275User-agent: Crawl4AI

276User-agent: Crawlspace

277User-agent: FirecrawlAgent

278User-agent: Scrapy

279Disallow: /

280

281

282# --- DATA RESALE: DENIED ----------------------------------------------------

283

284User-agent: BLEXBot

285User-agent: DataProvider

286Disallow: /

287

288

289# --- BLOCKED BY THE FIREWALL: DENIED -----------------------------------------

290# Crawlers the web ACL blocks whatever they send (lists/blocked-bots.yaml) and

291# that no other section names. Telling them here spares both sides the requests.

292

293User-agent: meta-webindexer

294User-agent: MistralAI-Training

295User-agent: CohereBot

296Disallow: /

297

298User-agent: AhrefsBot

299User-agent: AhrefsSiteAudit

300User-agent: DotBot

301User-agent: rogerbot

302User-agent: MJ12bot

303User-agent: Barkrowler

304User-agent: BrightEdge

305User-agent: SISTRIX

306User-agent: SeobilityBot

307User-agent: DataForSeoBot

308User-agent: MetadataScraper

309Disallow: /

310

311User-agent: sleepbot

312Disallow: /

313

314

315Sitemap: https://www.yumpu.com/sitemap_premium.xml

316Sitemap: https://www.yumpu.com/sitemap_free.xml

317Sitemap: https://www.yumpu.com/news/sitemap_news.xml

318

What do the bots actually get?

This page reads robots.txt, which states what yumpu.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on yumpu.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does yumpu.com block GPTBot?

Yes. yumpu.com’s robots.txt disallows GPTBot from the whole site (line 191: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does yumpu.com block ClaudeBot?

Yes. yumpu.com’s robots.txt disallows ClaudeBot from the whole site (line 196: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does yumpu.com block Google-Extended?

Yes. yumpu.com’s robots.txt disallows Google-Extended from the whole site (line 205: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does yumpu.com block PerplexityBot?

Yes. yumpu.com’s robots.txt disallows PerplexityBot from the whole site (line 271: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find yumpu.com?

Yes. yumpu.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.