Who blocks AI · #986 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does zeit.de block AI crawlers?

Yes. zeit.de’s robots.txt blocks 18 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 15 more. It also blocks Baiduspider among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: www.zeit.de/robots.txt · 381 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Partly blocked	line 344 · `Disallow: /zeit/` line 345 · `Disallow: /suche/` and 10 more
`Bingbot`	Microsoft	Partly blocked	line 344 · `Disallow: /zeit/` line 345 · `Disallow: /suche/` and 10 more
`DuckDuckBot`	DuckDuckGo	Partly blocked	line 344 · `Disallow: /zeit/` line 345 · `Disallow: /suche/` and 10 more
`Applebot`	Apple	Partly blocked	line 344 · `Disallow: /zeit/` line 345 · `Disallow: /suche/` and 10 more
`PetalBot`	Huawei	Partly blocked	line 344 · `Disallow: /zeit/` line 345 · `Disallow: /suche/` and 10 more
`YandexBot`	Yandex	Partly blocked	line 344 · `Disallow: /zeit/` line 345 · `Disallow: /suche/` and 10 more
`Baiduspider`	Baidu	Blocked	line 369 · `Disallow: /` under `User-agent: *`

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`PerplexityBot`	Perplexity	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`Claude-SearchBot`	Anthropic	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`DuckAssistBot`	DuckDuckGo	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`Amazonbot`	Amazon	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`YouBot`	You.com	Blocked	line 369 · `Disallow: /` under `User-agent: *`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`ClaudeBot`	Anthropic	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`Google-Extended`	Google	Blocked	line 359 · `Disallow: /` under `User-agent: Google-Extended`
`CCBot`	Common Crawl	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`Applebot-Extended`	Apple	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`GoogleOther`	Google	Partly blocked	line 344 · `Disallow: /zeit/` line 345 · `Disallow: /suche/` and 10 more
`Meta-ExternalAgent`	Meta	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`Bytespider`	ByteDance	Blocked	line 369 · `Disallow: /` under `User-agent: *`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`Claude-User`	Anthropic	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`Perplexity-User`	Perplexity	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`MistralAI-User`	Mistral	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`Meta-ExternalFetcher`	Meta	Blocked	line 369 · `Disallow: /` under `User-agent: *`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`SemrushBot`	Semrush	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`MJ12bot`	Majestic	Blocked	line 369 · `Disallow: /` under `User-agent: *`
`DotBot`	Moz	Blocked	line 369 · `Disallow: /` under `User-agent: *`

View www.zeit.de/robots.txt · 381 lines

1User-agent: AASA-Bot

2User-agent: Accessible Web Bot

3User-agent: AccessibleWebBot

4User-agent: AdDefend

5User-agent: adidxbot

6User-agent: AdkernelTopicCrawler

7User-agent: AdsBot-Google

8User-agent: AdsBot-Google-Mobile

9User-agent: Adstxt Parser, Demandbase

10User-agent: AdsTxtCrawler

11User-agent: Akkoma

12User-agent: Amazon AdBot

13User-agent: Amazon-Advertising-ad-standards-bot

14User-agent: AmazonAdBot

15User-agent: APIs-Google

16User-agent: Apple AuthenticationServicesAgent

17User-agent: Apple Link Preview

18User-agent: Applebot

19User-agent: AppleNewsBot

20User-agent: ArachnysSpider

21User-agent: Assaybot

22User-agent: Automattic

23User-agent: Awario

24User-agent: AwarioBot

25User-agent: AwarioRssBot

26User-agent: AwarioSmartBot

27User-agent: BazQux

28User-agent: Bing Ads

29User-agent: Bing Preview

30User-agent: bingbot

31User-agent: bingbot/

32User-agent: BingPreview

33User-agent: bitlybot

34User-agent: Bluesky

35User-agent: botify

36User-agent: BrandVerity

37User-agent: brokenlinkcheck

38User-agent: Brusselsbot

39User-agent: Buck/

40User-agent: Buckbot

41User-agent: BufferLinkPreviewBot

42User-agent: captify-crawler

43User-agent: Chrome Privacy Preserving Prefetch Proxy

44User-agent: Chrome-Lighthouse

45User-agent: citoid

46User-agent: Citoid/WMF

47User-agent: CommaFeed

48User-agent: compatible; proximic

49User-agent: Country Feed Monitor

50User-agent: CriteoBot

51User-agent: CriteoBot/

52User-agent: Cxense

53User-agent: cXensebot

54User-agent: cXensebot/

55User-agent: deadlinkchecker

56User-agent: Discordbot

57User-agent: dlvr.it

58User-agent: dlvrit

59User-agent: DobbyNewsBot

60User-agent: Domains

61User-agent: domains-distillery

62User-agent: DoubleVerifyBot

63User-agent: DuckDuckBot

64User-agent: DVbot

65User-agent: Echobot Bot

66User-agent: Edge Health Probe

67User-agent: ElevenlabsBot

68User-agent: ElevenlabsCrawler

69User-agent: Embedly

70User-agent: emetriqContextualBot

71User-agent: EventRadar/1.0

72User-agent: ExaUser

73User-agent: ExteAdstxtCrawl

74User-agent: facebookcatalog

75User-agent: facebookexternalhit

76User-agent: facebookexternalua

77User-agent: facebookplatform

78User-agent: Facebot

79User-agent: FairOpenNewsBot

80User-agent: FAST-WebCrawler

81User-agent: Fastmail Bot

82User-agent: FastmailUA

83User-agent: Feedbin

84User-agent: feedbot

85User-agent: FeedBurner

86User-agent: Feeder /

87User-agent: feeder.co

88User-agent: FeedFetcher-Google

89User-agent: FeedFlow

90User-agent: Feedler/1.0

91User-agent: Feedly

92User-agent: Feedly/1

93User-agent: Feedsearch

94User-agent: Feedsearch-Crawler

95User-agent: FeedValidator

96User-agent: FleebsBot

97User-agent: FlipboardProxy

98User-agent: FlipboardRSS

99User-agent: FreshRSS

100User-agent: Friendica

101User-agent: Gettrbot

102User-agent: GlancelotBot

103User-agent: GnowitNewsbot

104User-agent: GoodNews-Crawler

105User-agent: Google AdsBot

106User-agent: Google AdSense

107User-agent: Google AppsViewer

108User-agent: Google Digital Asset Links

109User-agent: Google Favicon

110User-agent: Google Feed Fetcher

111User-agent: Google Image Proxy

112User-agent: Google Images

113User-agent: Google Inspection Tool

114User-agent: Google Messages Link Preview

115User-agent: Google Publisher Center

116User-agent: Google Schema Markup Testing Tool

117User-agent: Google Scholar

118User-agent: Google StoreBot

119User-agent: Google Videos

120User-agent: Google Web Preview

121User-agent: Google-Ads-Creatives-Assistant

122User-agent: Google-adstxt

123User-agent: Google-AdWords-Express

124User-agent: Google-Adwords-Instant

125User-agent: Google-BusinessLinkVerification

126User-agent: Google-Calendar-Importer

127User-agent: Google-Certificates-Bridge

128User-agent: Google-Display-Ads-Bot

129User-agent: Google-Flights-Search-Spackle

130User-agent: Google-InspectionTool

131User-agent: Google-NotebookLM

132User-agent: Google-PageRenderer

133User-agent: Google-PhysicalWeb

134User-agent: Google-Read-Aloud

135User-agent: Google-Safety

136User-agent: Google-Site-Verification

137User-agent: Google-Sitemaps

138User-agent: Google-Structured-Data-Testing

139User-agent: Google-Structured-Data-Testing-Tool

140User-agent: Google-TravelAds-Live

141User-agent: Google-Trust-Services

142User-agent: google-xrawler

143User-agent: google.com/feedfetcher

144User-agent: Google_InspectionTool

145User-agent: GoogleAssociationService

146User-agent: Googlebot

147User-agent: Googlebot-IA

148User-agent: Googlebot-IA/

149User-agent: Googlebot-Image

150User-agent: Googlebot-Mobile

151User-agent: Googlebot-News

152User-agent: Googlebot-Video

153User-agent: GoogleDocs

154User-agent: GoogleImageProxy

155User-agent: GoogleOther

156User-agent: GoogleOther-Image

157User-agent: GoogleOther-Video

158User-agent: GoogleProducer

159User-agent: GoogleStackdriverMonitoring-UptimeChecks

160User-agent: GoogleUserTriggeredFetcher

161User-agent: GrapeshotCrawler

162User-agent: Greppr

163User-agent: GumGum

164User-agent: Hootsuite-Authoring

165User-agent: HubSpot

166User-agent: HubSpot Crawler

167User-agent: IAB-Tech-Lab

168User-agent: ias-ie

169User-agent: ias-or

170User-agent: ias-va

171User-agent: ias_crawler

172User-agent: Iframely

173User-agent: IFTTT

174User-agent: Inoreader

175User-agent: inoreader.com

176User-agent: InteractiveAdvertisingBureau

177User-agent: IServ News Bot

178User-agent: Jetslide

179User-agent: JournalistAppBot

180User-agent: Leikibot

181User-agent: LetsEncrypt

182User-agent: LinkedInBot

183User-agent: Magus

184User-agent: Marginalia Search

185User-agent: Mastodon

186User-agent: Mediapartners-Google

187User-agent: meta-externalads

188User-agent: MetaInspector

189User-agent: MetaJobBot

190User-agent: Microsoft News Ingestion

191User-agent: Microsoft Preview

192User-agent: MicrosoftPreview

193User-agent: MicrosoftPreview/

194User-agent: Miniflux

195User-agent: MJ12bot

196User-agent: MochiImageCrawler

197User-agent: Monsidobot

198User-agent: MS Search 4.0 Robot

199User-agent: MS Search 6.0 Robot

200User-agent: MSMNewsBot

201User-agent: msnbot

202User-agent: msnbot-media

203User-agent: MuckRack

204User-agent: NanoInteractive

205User-agent: NetNewsWire

206User-agent: Netvibes

207User-agent: NewsBlur

208User-agent: NewsCrawler

209User-agent: Newslitbot

210User-agent: NewsNow

211User-agent: Newstral

212User-agent: NewsVergleichDE

213User-agent: NewsWallBot

214User-agent: NewzBot

215User-agent: NextCloud

216User-agent: Nextcloud Server Crawler

217User-agent: Nextcloud-Server-Crawler

218User-agent: NoahBot

219User-agent: Nooshub

220User-agent: OdklBot/

221User-agent: officestorebot

222User-agent: OpenGraph.io

223User-agent: OpenGraphBot

224User-agent: OpenindexSpider

225User-agent: outbrain

226User-agent: Overcast

227User-agent: Overcast/1.0 Podcast Sync

228User-agent: Ozon Web Grabber

229User-agent: Paqlebot

230User-agent: peer39_crawler

231User-agent: peer39_crawler/1.0

232User-agent: PetalBot

233User-agent: Pingdom

234User-agent: Pinterest

235User-agent: pinterest.com/

236User-agent: Pinterestbot

237User-agent: Pleroma

238User-agent: Pocket Casts Feed Parser

239User-agent: PocketImageCache

240User-agent: Podcatcher Bot

241User-agent: PodchaserParser

242User-agent: PostGooBot

243User-agent: PressEngine Bot

244User-agent: PressEngineBot

245User-agent: Pro Sitemaps

246User-agent: Pro-Sitemaps/

247User-agent: ProjectShield Url Check

248User-agent: ProjectShield-UrlCheck

249User-agent: Protopage

250User-agent: proximic

251User-agent: PubMatic

252User-agent: Qwantbot

253User-agent: Readybot.io

254User-agent: ReadYou

255User-agent: RecordedFuture Global Inventory Crawler

256User-agent: redditbot

257User-agent: retresco-tms

258User-agent: rss-monitor

259User-agent: rss2tg bot

260User-agent: scope3

261User-agent: Screaming Frog SEO Spider

262User-agent: search.marginalia.nu

263User-agent: Seeding Alliance ads.txt Check

264User-agent: Seekport

265User-agent: Seekport Crawler

266User-agent: SeekportBot

267User-agent: Sellers.Guide Crawler by Primis

268User-agent: semantic-visions

269User-agent: Seznam

270User-agent: SeznamBot

271User-agent: SimplePie

272User-agent: SinceraSyntheticUser

273User-agent: SISTRIX Crawler

274User-agent: SISTRIX Optimizer

275User-agent: Skype

276User-agent: SkypeUriPreview

277User-agent: Slack Image Proxy

278User-agent: Slack-ImgProxy

279User-agent: Slackbot

280User-agent: Slackbot 1.0

281User-agent: Slackbot-LinkExpanding

282User-agent: SlackbotLinkExpanding

283User-agent: SmartFeedBot

284User-agent: SmartologyBot

285User-agent: Snap URL Preview Service

286User-agent: SnapchatAds

287User-agent: SpaceCowboys RSS Reader

288User-agent: Speroll-AdsTxt-Crawler

289User-agent: StartmeBot

290User-agent: StatistikHessen

291User-agent: SteamChat

292User-agent: Storebot-Google

293User-agent: SummalyBot

294User-agent: Superfeedr

295User-agent: SurdotlyBot

296User-agent: Synapse

297User-agent: Telegram Bot

298User-agent: TelegramBot

299User-agent: theoldreader.com

300User-agent: Tickaroo-LinkPreview

301User-agent: TinEye-Web

302User-agent: Tiny Tiny RSS

303User-agent: tollbot

304User-agent: Trade Desk ads.txt

305User-agent: Trello Link Preview

306User-agent: TrueRelevanceBot

307User-agent: TTD-Content

308User-agent: Tumblr

309User-agent: Tumblr/

310User-agent: Twitterbot

311User-agent: Twitterbot/

312User-agent: uipbot

313User-agent: UrlDownloadabilityBot

314User-agent: Verity

315User-agent: Viber

316User-agent: ViennaTinyBot

317User-agent: ViralizeBot

318User-agent: vkShare

319User-agent: W3 Validator Services

320User-agent: W3C-checklink

321User-agent: w4mwnpbXf3MFAbxOkJRw

322User-agent: WebMoni/

323User-agent: WebsiteMetadataRetriever

324User-agent: WhatsApp

325User-agent: WMF Zotero Translation Server

326User-agent: WPMU DEV Broken Link Checker

327User-agent: Xing Bot

328User-agent: XingBot

329User-agent: yacybot

330User-agent: Yahoo Ad monitoring

331User-agent: Yahoo Link Preview

332User-agent: Yahoo! Slurp

333User-agent: YahooMailProxy

334User-agent: yandex

335User-agent: YandexBot

336User-agent: YandexImages

337User-agent: YandexMobileBot

338User-agent: YandexRenderResourcesBot

339User-agent: YandexVideo

340User-agent: ZetaGlobal-adstxt

341User-agent: ZoteroTranslationServer/WMF

342Allow: /

343

344Disallow: /zeit/

345Disallow: /suche/

346Disallow: /templates/

347Disallow: /hp_channels/

348Disallow: /send/

349Disallow: /rezepte/suche/

350Disallow: */comment-thread?

351Disallow: */liveblog-backend*

352Disallow: /framebuilder/

353Disallow: /campus/framebuilder/

354Disallow: /navigation-teasers*

355Disallow: *iqadcontroller.js

356Allow: /

357

358User-agent: Google-Extended

359Disallow: /

360Allow: /*-gxe$

361

362User-agent: Googlebot-News

363Disallow: /angebote/

364

365User-agent: GrapeshotCrawler

366crawl-delay: 3

367

368User-agent: *

369Disallow: /

370Allow:/llms.txt

371

372Sitemap: https://www.zeit.de/gsitemaps/index.xml

373

374# Legal notice: zeit.de expressly reserves the right to use its content

375# for commercial text and data mining (§ 44 b UrhG).

376# The use of robots or other automated means to access zeit.de or collect

377# or mine data without the express permission of zeit.de is strictly prohibited.

378# zeit.de may, in its discretion, permit certain automated access to certain

379# zeit.de pages.

380# If you would like to apply for permission to crawl zeit.de, collect or use

381# data, please email crawl-request@zeit.de.

What do the bots actually get?

This page reads robots.txt, which states what zeit.de asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on zeit.de Test a path in the robots.txt tester

History

19 October 2026: first checked.
9 November 2026: now blocks part of the site to Googlebot; now blocks part of the site to Bingbot; now blocks part of the site to DuckDuckBot; now blocks part of the site to Applebot; now blocks part of the site to PetalBot; now blocks part of the site to YandexBot; started blocking OAI-SearchBot; started blocking Claude-SearchBot; started blocking DuckAssistBot; started blocking Amazonbot; started blocking YouBot; now blocks part of the site to GoogleOther; started blocking ChatGPT-User; started blocking Claude-User; started blocking MistralAI-User; started blocking Meta-ExternalFetcher; started blocking AhrefsBot; started blocking SemrushBot; started blocking MJ12bot; started blocking DotBot.

Questions

Does zeit.de block GPTBot?

Yes. zeit.de’s robots.txt disallows GPTBot from the whole site (line 369: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does zeit.de block ClaudeBot?

Yes. zeit.de’s robots.txt disallows ClaudeBot from the whole site (line 369: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does zeit.de block Google-Extended?

Yes. zeit.de’s robots.txt disallows Google-Extended from the whole site (line 359: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does zeit.de block PerplexityBot?

Yes. zeit.de’s robots.txt disallows PerplexityBot from the whole site (line 369: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find zeit.de?

Not through its own crawler. zeit.de’s robots.txt disallows OAI-SearchBot from the whole site (line 369: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.