Who blocks AI · #404 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does zdnet.com block AI crawlers?

Yes. zdnet.com’s robots.txt blocks 17 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, PerplexityBot and 14 more. It also blocks Bingbot, Applebot and PetalBot among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedAllowed CCBotBlocked

robots.txt: www.zdnet.com/robots.txt · 228 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Blocked	line 199 · `Disallow: /` under `User-agent: msnbot`
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Blocked	line 199 · `Disallow: /` under `User-agent: Applebot`
`PetalBot`	Huawei	Blocked	line 199 · `Disallow: /` under `User-agent: PetalBot`
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 199 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 199 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Blocked	line 199 · `Disallow: /` under `User-agent: Claude-SearchBot`
`DuckAssistBot`	DuckDuckGo	Blocked	line 199 · `Disallow: /` under `User-agent: DuckAssistBot`
`Amazonbot`	Amazon	Blocked	line 199 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Blocked	line 199 · `Disallow: /` under `User-agent: YouBot`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 199 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 199 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Allowed	no rule matches the home page
`CCBot`	Common Crawl	Blocked	line 199 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 199 · `Disallow: /` under `User-agent: Applebot-extended`
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Blocked	line 199 · `Disallow: /` under `User-agent: meta-externalagent`
`Bytespider`	ByteDance	Blocked	line 199 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 199 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Blocked	line 199 · `Disallow: /` under `User-agent: Claude-User`
`Perplexity-User`	Perplexity	Blocked	line 199 · `Disallow: /` under `User-agent: Perplexity-User`
`MistralAI-User`	Mistral	Blocked	line 199 · `Disallow: /` under `User-agent: MistralAI-User`
`Meta-ExternalFetcher`	Meta	Blocked	line 199 · `Disallow: /` under `User-agent: meta-externalfetcher`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	no rule matches the home page
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View www.zdnet.com/robots.txt · 228 lines

1# www.robotstxt.org/

2# www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

4# Ziff Davis content is made available for your non-commercial use subject to our

5# Terms of Use here: https://www.ziffdavis.com/terms-of-use

6# Use of any robot, crawler, or other tool to scrape, harvest, extract, or retrieve any content on

7# this website using automated means is prohibited without written permission from Ziff Davis.

8# Prohibited uses include but are not limited to:

9# (1) text and data mining under Art. 4 of the EU Directive on Copyright in the Digital Single

10# Market;

11# (2) development or operation of artificial intelligence or machine learning software or

12# databases, including by training, fine-tuning, embedding, and retrieval-augmented generation;

13# (3) creating data sets containing our content or sharing it with others; and

14# (4) any commercial purposes.

15# Contact licensing@ziffdavis.com for assistance.

17User-agent: *

19Disallow: /user/*

20Disallow: /members/

21Disallow: /members/newsletters/

22Disallow: /members/alerts/add/

23Disallow: /search/

24Disallow: *Xhr*

25Disallow: */xhr*

26Disallow: */ajax/*

27Disallow: */fly/*/bundles/flyjs/*

28Disallow: */libs/*

29Disallow: */version!libs/*

30Disallow: /.well-known/*

31Disallow: /index.php/*

32Disallow: *?beta=*

33Disallow: *?ftag=*

35User-agent: msnbot

36Crawl-delay: 1

38# Ziff Davis bot list

39user-agent: AddSearchBot

40user-agent: AgentDataBot

41user-agent: AI2Bot

42user-agent: Ai2Bot-DeepResearchEval

43user-agent: Ai2Bot-Dolma

44user-agent: AIWebIndex

45user-agent: AIWebIndex-Agent

46user-agent: amazon-QBusiness

47user-agent: Amazonbot

48user-agent: AmazonBuyForMe

49user-agent: Amzn-SearchBot

50user-agent: Amzn-User

51user-agent: Anchor Browser

52user-agent: AndiAgent

53user-agent: anthropic-ai

54user-agent: ApifyBot

55user-agent: ApifyWebsiteContentCrawler

56user-agent: Applebot

57user-agent: Applebot-extended

58user-agent: atlassian-bot

59user-agent: AutoRAG

60user-agent: AwarioSmartBot

61user-agent: AzureAI-SearchBot

62user-agent: BeansLLM-CorpusBot

63user-agent: Big Sur AI

64user-agent: bigsur.ai

65user-agent: BixelBot

66user-agent: Brandwatch

67user-agent: Bravebot

68user-agent: Brightbot

69user-agent: Bytespider

70user-agent: CCBot

71user-agent: Channel3Bot

72user-agent: ChatGLM-Spider

73user-agent: ChatGPT-User

74user-agent: Claude-Code

75user-agent: Claude-SearchBot

76user-agent: Claude-User

77user-agent: Claude-Web

78user-agent: ClaudeBot

79user-agent: Cloudflare-AutoRAG

80user-agent: CloudflareBrowserRenderingCrawler

81user-agent: Code

82user-agent: cohere-ai

83user-agent: cohere-training-data-crawler

84user-agent: Cotoyogi

85user-agent: Crawl4AI

86user-agent: Cursor

87user-agent: Datenbank Crawler

88user-agent: Datenbank-Crawler

89user-agent: DeepSeekBot

90user-agent: DesearchBot

91user-agent: Devin

92user-agent: Diffbot

93user-agent: Direqt Anomura

94user-agent: Dola-SearchBot

95user-agent: DuckAssistBot

96user-agent: Echobot Bot

97user-agent: ExaBot

98user-agent: ExaSearchBot

99user-agent: FacebookBot

100user-agent: Factset_spyderbot

101user-agent: FirecrawlAgent

102user-agent: GeistHaus-PageFetcher

103user-agent: GPTBot

104user-agent: grok-agent

105user-agent: GrokBot

106user-agent: Hunyuan

107user-agent: iAsk

108user-agent: iAskBot

109user-agent: iaskspider

110user-agent: ICC Crawler

111user-agent: ICC-Crawler

112user-agent: ImageBot

113user-agent: ImageSiftBot

114user-agent: imageSpider

115user-agent: IsonSearchBot

116user-agent: kagi-fetcher

117user-agent: Kangaroo Bot

118user-agent: Kangaroo-Bot

119user-agent: Keenable-User

120user-agent: KeenableBot

121user-agent: Kimi-Agent

122user-agent: Kimi-User

123user-agent: KlaviyoAIBot

124user-agent: Kunato

125user-agent: KunatoCrawler

126user-agent: laion-huggingface-processor

127user-agent: LCC

128user-agent: LINER Bot

129user-agent: LinerBot

130user-agent: LinkupBot

131user-agent: Manus-User

132user-agent: meta-externalagent

133user-agent: meta-externalfetcher

134user-agent: meta-webindexer

135user-agent: mistral.ai

136user-agent: MistralAI-User

137user-agent: netEstate Imprint Crawler

138user-agent: NovaAct

139user-agent: Novellum

140user-agent: Novellum AI Crawl

141user-agent: OAI-SearchBot

142user-agent: omgili

143user-agent: opencode

144user-agent: PanguBot

145user-agent: PeopleInc-DCipher-Scraper/1.1.0

146user-agent: Perplexity-User

147user-agent: PerplexityBot

148user-agent: PetalBot

149user-agent: PhindBot

150user-agent: Poggio-Citations

151user-agent: qodercli

152user-agent: QualifiedBot

153user-agent: recraft-image-dataset

154user-agent: Reflectionbot

155user-agent: SBIntuitionsBot

156user-agent: ScryBot

157user-agent: SeekrBot

158user-agent: SemrushBot-OCOB

159user-agent: SemrushBotSwa

160user-agent: Shap-User

161user-agent: ShapBot

162user-agent: SofyaBot

163user-agent: SomethinggBot

164user-agent: Spider

165user-agent: TalarionFrontierCrawler

166user-agent: TalarionSearchBot

167user-agent: TavilyBot

168user-agent: TerraCotta

169user-agent: Timpibot

170user-agent: TongyiBot

171user-agent: Trae

172user-agent: TwinAgent

173user-agent: UseAI

174user-agent: Velen Crawler

175user-agent: VelenPublicWebCrawler

176user-agent: Webzio-Extended

177user-agent: Wrtn

178user-agent: WRTNBot

179user-agent: xAI-Grok

180user-agent: YandexAdditionalBot

181user-agent: YepBot

182user-agent: YiBot

183user-agent: YiyanBot

184user-agent: YouBot

185user-agent: ZanistaBot

186Allow: /article/how-ai-companies-are-secretly-collecting-training-data-from-the-web-and-why-it-matters/

187Allow: /article/how-proxy-servers-actually-work-and-why-theyre-so-valuable/

188Allow: /article/how-to-find-and-remove-your-personal-information-online/

189Allow: /article/how-to-print-checks-in-quickbooks-online/

190Allow: /article/how-to-remove-your-personal-information-from-whitepages-in-5-steps-and-why-you-should/

191Allow: /article/how-to-remove-yourself-from-whitepages-in-5-quick-steps-and-why-you-should/

192Allow: /article/how-to-undo-a-reconciliation-in-quickbooks-online-the-easy-way/

193Allow: /article/i-found-the-easiest-way-to-delete-myself-from-the-internet-and-why-you-shouldnt-wait-to-use-it-too/

194Allow: /article/incogni-review/

195Allow: /article/incogni-vs-deleteme/

196Allow: /article/this-proxy-provider-i-tested-is-the-best-for-web-scraping-and-its-not-iproyal-or-marsproxies/

197Allow: /article/xero-vs-quickbooks/

198Allow: /paid-content/article/new-tv-weak-sound-heres-how-to-fix-it-with-a-sonos-soundbar/

199Disallow: /

200

201#ZDnet restricted bots

202User-agent: "008"

203User-agent: AwarioRssBot

204User-agent: AwarioSmartBot

205User-agent: DDM-DCipher/1.0.7

206User-agent: HTTrack

207User-agent: Magpie-crawler

208User-agent: Nutch

209User-agent: Offline Explorer

210User-agent: Omgilibot

211User-agent: Peer39_crawler/1.0

212User-agent: Scrapy

213Disallow: /

214

215Sitemap: https://www.zdnet.com/sitemap-authors.xml

216

217Sitemap: https://www.zdnet.com/sitemap-google-news.xml

218

219Sitemap: https://www.zdnet.com/sitemaps/articles/index.xml

220Sitemap: https://www.zdnet.com/sitemaps/best/index.xml

221Sitemap: https://www.zdnet.com/sitemaps/categories/index.xml

222Sitemap: https://www.zdnet.com/sitemaps/deals/index.xml

223Sitemap: https://www.zdnet.com/sitemaps/galleries/index.xml

224Sitemap: https://www.zdnet.com/sitemaps/how-to/index.xml

225Sitemap: https://www.zdnet.com/sitemaps/news/index.xml

226Sitemap: https://www.zdnet.com/sitemaps/reviews/index.xml

227Sitemap: https://www.zdnet.com/sitemaps/video/index.xml

228

What do the bots actually get?

This page reads robots.txt, which states what zdnet.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on zdnet.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does zdnet.com block GPTBot?

Yes. zdnet.com’s robots.txt disallows GPTBot from the whole site (line 199: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does zdnet.com block ClaudeBot?

Yes. zdnet.com’s robots.txt disallows ClaudeBot from the whole site (line 199: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does zdnet.com block Google-Extended?

No. zdnet.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does zdnet.com block PerplexityBot?

Yes. zdnet.com’s robots.txt disallows PerplexityBot from the whole site (line 199: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find zdnet.com?

Not through its own crawler. zdnet.com’s robots.txt disallows OAI-SearchBot from the whole site (line 199: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.