Who blocks AI · #1024 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does dzen.ru block AI crawlers?

No. dzen.ru’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotAllowed OAI-SearchBotAllowed ClaudeBotAllowed PerplexityBotAllowed Google-ExtendedAllowed CCBotAllowed

robots.txt: dzen.ru/robots.txt · 393 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Partly blocked	line 292 · `Disallow: /about?` line 293 · `Disallow: /away` and 46 more
`Bingbot`	Microsoft	Allowed	line 343 · `Allow: /` under `User-agent: *`
`DuckDuckBot`	DuckDuckGo	Allowed	line 343 · `Allow: /` under `User-agent: *`
`Applebot`	Apple	Partly blocked	line 292 · `Disallow: /about?` line 293 · `Disallow: /away` and 46 more
`PetalBot`	Huawei	Allowed	line 343 · `Allow: /` under `User-agent: *`
`YandexBot`	Yandex	Allowed	line 343 · `Allow: /` under `User-agent: *`
`Baiduspider`	Baidu	Allowed	line 343 · `Allow: /` under `User-agent: *`

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Allowed	line 343 · `Allow: /` under `User-agent: *`
`PerplexityBot`	Perplexity	Allowed	line 343 · `Allow: /` under `User-agent: *`
`Claude-SearchBot`	Anthropic	Allowed	line 343 · `Allow: /` under `User-agent: *`
`DuckAssistBot`	DuckDuckGo	Allowed	line 343 · `Allow: /` under `User-agent: *`
`Amazonbot`	Amazon	Allowed	line 343 · `Allow: /` under `User-agent: *`
`YouBot`	You.com	Allowed	line 343 · `Allow: /` under `User-agent: *`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Allowed	line 343 · `Allow: /` under `User-agent: *`
`ClaudeBot`	Anthropic	Allowed	line 343 · `Allow: /` under `User-agent: *`
`Google-Extended`	Google	Allowed	line 343 · `Allow: /` under `User-agent: *`
`CCBot`	Common Crawl	Allowed	line 343 · `Allow: /` under `User-agent: *`
`Applebot-Extended`	Apple	Allowed	line 343 · `Allow: /` under `User-agent: *`
`GoogleOther`	Google	Allowed	line 343 · `Allow: /` under `User-agent: *`
`Meta-ExternalAgent`	Meta	Allowed	line 343 · `Allow: /` under `User-agent: *`
`Bytespider`	ByteDance	Allowed	line 343 · `Allow: /` under `User-agent: *`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Allowed	line 343 · `Allow: /` under `User-agent: *`
`Claude-User`	Anthropic	Allowed	line 343 · `Allow: /` under `User-agent: *`
`Perplexity-User`	Perplexity	Allowed	line 343 · `Allow: /` under `User-agent: *`
`MistralAI-User`	Mistral	Allowed	line 343 · `Allow: /` under `User-agent: *`
`Meta-ExternalFetcher`	Meta	Allowed	line 343 · `Allow: /` under `User-agent: *`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	line 343 · `Allow: /` under `User-agent: *`
`SemrushBot`	Semrush	Allowed	line 343 · `Allow: /` under `User-agent: *`
`MJ12bot`	Majestic	Allowed	line 343 · `Allow: /` under `User-agent: *`
`DotBot`	Moz	Allowed	line 343 · `Allow: /` under `User-agent: *`

View dzen.ru/robots.txt · 393 lines

1User-agent: Yandex

2Allow: /

3Allow: /.well-known/assetlinks.json

4Allow: /*?utm_sourse=55

6Disallow: /about*?*

7Disallow: /away

8Disallow: /top$

9Disallow: /search

10Disallow: /money

11Disallow: /onboarding

12Disallow: /news/top

13Disallow: /news/smi

14Disallow: /news/search

15Disallow: /t/

16Disallow: /user/

17Disallow: /embed/

18Disallow: /*/url

19Disallow: /video/games/tags/

20Disallow: /*sso_failed=

21Disallow: /*erid=

22Disallow: /*?utm*

23Disallow: /games

24Disallow: /subscriptions-manager?channel=

26Disallow: /articles?

27Disallow: /video?

28Disallow: /discussions

30Clean-param: &_csrf

31Clean-param: &action

32Clean-param: &amp

33Clean-param: &ampplace

34Clean-param: &amupr_id

35Clean-param: &appsearch_header

36Clean-param: &autoplay

37Clean-param: &backoffice

38Clean-param: &branding

39Clean-param: &briefEditorPublicationId

40Clean-param: &bx_sender_conversion_id

41Clean-param: &clid

42Clean-param: &collection

43Clean-param: &comment-request

44Clean-param: &comments_data

45Clean-param: &content

46Clean-param: &contour

47Clean-param: &country_code

48Clean-param: &device_id

49Clean-param: &encoded_pulse_user_id

50Clean-param: &encoded_pulse_user_info

51Clean-param: &epik

52Clean-param: &etext=

53Clean-param: &experiment

54Clean-param: &f

55Clean-param: &family

56Clean-param: &fan

57Clean-param: &favid

58Clean-param: &feed_exp

59Clean-param: &feed_filter_source

60Clean-param: &firstchars

61Clean-param: &flight_id

62Clean-param: &from

63Clean-param: &from_block

64Clean-param: &from_cloud

65Clean-param: &from_floors_promo

66Clean-param: &from_page

67Clean-param: &from_parent_id

68Clean-param: &from_parent_type

69Clean-param: &from_site

70Clean-param: &ga_client_id

71Clean-param: &hide_interest_header

72Clean-param: &id

73Clean-param: &integration

74Clean-param: &interview_id

75Clean-param: &invite

76Clean-param: &item_id

77Clean-param: &lang

78Clean-param: &link

79Clean-param: &loop

80Clean-param: &lr

81Clean-param: &modal

82Clean-param: &mt_click_id

83Clean-param: &mt_link_id

84Clean-param: &mute

85Clean-param: &no_ad

86Clean-param: &noredir

87Clean-param: &nw

88Clean-param: &p

89Clean-param: &page

90Clean-param: &parent_rid

91Clean-param: &persistent_id

92Clean-param: &place

93Clean-param: &play_on_visible

94Clean-param: &pulse_user_id

95Clean-param: &query

96Clean-param: &ref

97Clean-param: &referrer

98Clean-param: &referrer_clid

99Clean-param: &referrer_place

100Clean-param: &repostId

101Clean-param: &rid

102Clean-param: &rubric

103Clean-param: &secdata

104Clean-param: &service

105Clean-param: &share_to

106Clean-param: &sid

107Clean-param: &sortby

108Clean-param: &source

109Clean-param: &sso_failed

110Clean-param: &stid

111Clean-param: &story

112Clean-param: &t

113Clean-param: &text

114Clean-param: &to

115Clean-param: &token

116Clean-param: &train

117Clean-param: &tst

118Clean-param: &tt

119Clean-param: &tv

120Clean-param: &utr

121Clean-param: &uuid

122Clean-param: &vertical

123Clean-param: &via

124Clean-param: &vid

125Clean-param: &video_id

126Clean-param: &wan

127Clean-param: &web_uid

128Clean-param: &web_yandexuid

129Clean-param: &wizard

130Clean-param: &yandexClientId

131

132User-agent: Googlebot

133Allow: /

134Allow: /.well-known/assetlinks.json

135Allow: /list/*

136

137Allow: /a/*_csrf

138Allow: /a/*action

139Allow: /a/*amp

140Allow: /a/*ampplace

141Allow: /a/*amupr_id

142Allow: /a/*appsearch_header=

143Allow: /a/*autoplay=

144Allow: /a/*backoffice=

145Allow: /a/*branding=

146Allow: /a/*briefEditorPublicationId=

147Allow: /a/*bx_sender_conversion_id=

148Allow: /a/*cachebust=

149Allow: /a/*clid=

150Allow: /a/*collection=

151Allow: /a/*country_code=

152Allow: /a/*erid=

153Allow: /a/*experiment=

154Allow: /a/*feed_exp=

155Allow: /a/*from_cloud=

156Allow: /a/*from_page=

157Allow: /a/*invite=

158Allow: /a/*loop=

159Allow: /a/*pulse_user_id=

160Allow: /a/*sso_failed=

161Allow: /a/*t=

162Allow: /a/*to=

163Allow: /a/*token=

164Allow: /a/*utm_referrer=

165Allow: /a/*utm_source=

166Allow: /a/*?etext=

167

168Allow: /b/*_csrf

169Allow: /b/*action

170Allow: /b/*amp

171Allow: /b/*ampplace

172Allow: /b/*amupr_id

173Allow: /b/*appsearch_header=

174Allow: /b/*autoplay=

175Allow: /b/*backoffice=

176Allow: /b/*branding=

177Allow: /b/*briefEditorPublicationId=

178Allow: /b/*bx_sender_conversion_id=

179Allow: /b/*cachebust=

180Allow: /b/*clid=

181Allow: /b/*collection=

182Allow: /b/*country_code=

183Allow: /b/*erid=

184Allow: /b/*experiment=

185Allow: /b/*feed_exp=

186Allow: /b/*from_cloud=

187Allow: /b/*from_page=

188Allow: /b/*invite=

189Allow: /b/*loop=

190Allow: /b/*pulse_user_id=

191Allow: /b/*sso_failed=

192Allow: /b/*t=

193Allow: /b/*to=

194Allow: /b/*token=

195Allow: /b/*utm_referrer=

196Allow: /b/*utm_source=

197Allow: /b/*?etext=

198

199Allow: /list/*_csrf

200Allow: /list/*action

201Allow: /list/*amp

202Allow: /list/*ampplace

203Allow: /list/*amupr_id

204Allow: /list/*appsearch_header=

205Allow: /list/*autoplay=

206Allow: /list/*backoffice=

207Allow: /list/*branding=

208Allow: /list/*briefEditorPublicationId=

209Allow: /list/*bx_sender_conversion_id=

210Allow: /list/*cachebust=

211Allow: /list/*clid=

212Allow: /list/*collection=

213Allow: /list/*country_code=

214Allow: /list/*erid=

215Allow: /list/*experiment=

216Allow: /list/*feed_exp=

217Allow: /list/*from_cloud=

218Allow: /list/*from_page=

219Allow: /list/*invite=

220Allow: /list/*loop=

221Allow: /list/*pulse_user_id=

222Allow: /list/*sso_failed=

223Allow: /list/*t=

224Allow: /list/*to=

225Allow: /list/*token=

226Allow: /list/*utm_referrer=

227Allow: /list/*utm_source=

228Allow: /list/*?etext=

229

230Allow: /video/watch/*_csrf

231Allow: /video/watch/*action

232Allow: /video/watch/*amp

233Allow: /video/watch/*ampplace

234Allow: /video/watch/*amupr_id

235Allow: /video/watch/*appsearch_header=

236Allow: /video/watch/*autoplay=

237Allow: /video/watch/*backoffice=

238Allow: /video/watch/*branding=

239Allow: /video/watch/*briefEditorPublicationId=

240Allow: /video/watch/*bx_sender_conversion_id=

241Allow: /video/watch/*cachebust=

242Allow: /video/watch/*clid=

243Allow: /video/watch/*collection=

244Allow: /video/watch/*country_code=

245Allow: /video/watch/*erid=

246Allow: /video/watch/*experiment=

247Allow: /video/watch/*feed_exp=

248Allow: /video/watch/*from_cloud=

249Allow: /video/watch/*from_page=

250Allow: /video/watch/*invite=

251Allow: /video/watch/*loop=

252Allow: /video/watch/*pulse_user_id=

253Allow: /video/watch/*sso_failed=

254Allow: /video/watch/*t=

255Allow: /video/watch/*to=

256Allow: /video/watch/*token=

257Allow: /video/watch/*utm_referrer=

258Allow: /video/watch/*utm_source=

259Allow: /video/watch/*?etext=

260

261Allow: /shorts/*_csrf

262Allow: /shorts/*action

263Allow: /shorts/*amp

264Allow: /shorts/*ampplace

265Allow: /shorts/*amupr_id

266Allow: /shorts/*appsearch_header=

267Allow: /shorts/*autoplay=

268Allow: /shorts/*backoffice=

269Allow: /shorts/*branding=

270Allow: /shorts/*briefEditorPublicationId=

271Allow: /shorts/*bx_sender_conversion_id=

272Allow: /shorts/*cachebust=

273Allow: /shorts/*clid=

274Allow: /shorts/*collection=

275Allow: /shorts/*country_code=

276Allow: /shorts/*erid=

277Allow: /shorts/*experiment=

278Allow: /shorts/*feed_exp=

279Allow: /shorts/*from_cloud=

280Allow: /shorts/*from_page=

281Allow: /shorts/*invite=

282Allow: /shorts/*loop=

283Allow: /shorts/*pulse_user_id=

284Allow: /shorts/*sso_failed=

285Allow: /shorts/*t=

286Allow: /shorts/*to=

287Allow: /shorts/*token=

288Allow: /shorts/*utm_referrer=

289Allow: /shorts/*utm_source=

290Allow: /shorts/*?etext=

291

292Disallow: /about*?*

293Disallow: /away

294Disallow: /top$

295Disallow: /search

296Disallow: /money

297Disallow: /onboarding

298Disallow: /news/top

299Disallow: /news/smi

300Disallow: /news/search

301Disallow: /t/

302Disallow: /user/

303Disallow: /embed/

304Disallow: /*/url

305Disallow: /video/games/tags/

306Disallow: /*_csrf

307Disallow: /*action

308Disallow: /*amp

309Disallow: /*ampplace

310Disallow: /*amupr_id

311Disallow: /*appsearch_header=

312Disallow: /*autoplay=

313Disallow: /*backoffice=

314Disallow: /*branding=

315Disallow: /*briefEditorPublicationId=

316Disallow: /*bx_sender_conversion_id=

317Disallow: /*cachebust=

318Disallow: /*clid=

319Disallow: /*collection=

320Disallow: /*country_code=

321Disallow: /*erid=

322Disallow: /*experiment=

323Disallow: /*feed_exp=

324Disallow: /*from_cloud=

325Disallow: /*from_page=

326Disallow: /*invite=

327Disallow: /*loop=

328Disallow: /*pulse_user_id=

329Disallow: /*sso_failed=

330Disallow: /*t=

331Disallow: /*to=

332Disallow: /*token=

333Disallow: /*utm_referrer=

334Disallow: /*utm_source=

335Disallow: /games

336Disallow: /*?etext=

337Disallow: /subscriptions-manager?channel=

338

339Disallow: /articles?

340Disallow: /video?

341

342User-Agent: *

343Allow: /

344Allow: /.well-known/assetlinks.json

345Disallow: /about*?*

346Disallow: /away

347Disallow: /embed/

348Disallow: /games

349Disallow: /money

350Disallow: /news/search

351Disallow: /news/smi

352Disallow: /news/top

353Disallow: /onboarding

354Disallow: /search

355Disallow: /subscriptions-manager?channel=

356Disallow: /t/

357Disallow: /top$

358Disallow: /user/

359Disallow: /*/url

360Disallow: /video/games/tags/

361Disallow: /*erid=

362Disallow: /*?etext=

363Disallow: /*sso_failed=

364Disallow: /*?sso_failed=

365Disallow: /*to=

366

367Sitemap: https://dzen.ru/sitemaps/brief_v2/sitemap.xml

368Sitemap: https://dzen.ru/sitemaps/channels_v2/sitemap.xml

369Sitemap: https://dzen.ru/sitemaps/gif_long_v2/sitemap.xml

370Sitemap: https://dzen.ru/sitemaps/gif_short_v2/sitemap.xml

371Sitemap: https://dzen.ru/sitemaps/image_v2/sitemap.xml

372Sitemap: https://dzen.ru/sitemaps/native_v2/sitemap.xml

373Sitemap: https://dzen.ru/sitemaps/slug-factory/sitemap.xml

374Sitemap: https://dzen.ru/sitemaps/suite_v2/sitemap.xml

375Sitemap: https://dzen.ru/sitemaps/tv/sitemap.xml

376Sitemap: https://dzen.ru/sitemaps/news-archive/sitemap.xml

377Sitemap: https://dzen.ru/sitemaps/news-region/sitemap.xml

378Sitemap: https://dzen.ru/sitemaps/news-rubric/sitemap.xml

379Sitemap: https://dzen.ru/sitemaps/meta-channel_v2/sitemap.xml

380

381Sitemap: https://dzen.ru/sitemaps/brief_v2/sitemap.xml.gz

382Sitemap: https://dzen.ru/sitemaps/channels_v2/sitemap.xml.gz

383Sitemap: https://dzen.ru/sitemaps/gif_long_v2/sitemap.xml.gz

384Sitemap: https://dzen.ru/sitemaps/gif_short_v2/sitemap.xml.gz

385Sitemap: https://dzen.ru/sitemaps/image_v2/sitemap.xml.gz

386Sitemap: https://dzen.ru/sitemaps/native_v2/sitemap.xml.gz

387Sitemap: https://dzen.ru/sitemaps/slug-factory/sitemap.xml.gz

388Sitemap: https://dzen.ru/sitemaps/suite_v2/sitemap.xml.gz

389Sitemap: https://dzen.ru/sitemaps/tv/sitemap.xml.gz

390Sitemap: https://dzen.ru/sitemaps/news-archive/sitemap.xml.gz

391Sitemap: https://dzen.ru/sitemaps/news-region/sitemap.xml.gz

392Sitemap: https://dzen.ru/sitemaps/news-rubric/sitemap.xml.gz

393Sitemap: https://dzen.ru/sitemaps/meta-channel_v2/sitemap.xml.gz

What do the bots actually get?

This page reads robots.txt, which states what dzen.ru asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on dzen.ru Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does dzen.ru block GPTBot?

No. dzen.ru’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does dzen.ru block ClaudeBot?

No. dzen.ru’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does dzen.ru block Google-Extended?

No. dzen.ru’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does dzen.ru block PerplexityBot?

No. dzen.ru’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find dzen.ru?

Yes. dzen.ru’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.