Who blocks AI · #635 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does pcmag.com block AI crawlers?

Yes. pcmag.com’s robots.txt blocks 17 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, PerplexityBot and 14 more. It also blocks Applebot and PetalBot among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedAllowed CCBotBlocked

robots.txt: www.pcmag.com/robots.txt · 387 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Blocked	line 328 · `Disallow: /` under `User-agent: Applebot`
`PetalBot`	Huawei	Blocked	line 328 · `Disallow: /` under `User-agent: PetalBot`
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 328 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 328 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Blocked	line 328 · `Disallow: /` under `User-agent: Claude-SearchBot`
`DuckAssistBot`	DuckDuckGo	Blocked	line 328 · `Disallow: /` under `User-agent: DuckAssistBot`
`Amazonbot`	Amazon	Blocked	line 328 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Blocked	line 328 · `Disallow: /` under `User-agent: YouBot`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 328 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 328 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Allowed	no rule matches the home page
`CCBot`	Common Crawl	Blocked	line 328 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 328 · `Disallow: /` under `User-agent: Applebot-extended`
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Blocked	line 328 · `Disallow: /` under `User-agent: meta-externalagent`
`Bytespider`	ByteDance	Blocked	line 328 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 328 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Blocked	line 328 · `Disallow: /` under `User-agent: Claude-User`
`Perplexity-User`	Perplexity	Blocked	line 328 · `Disallow: /` under `User-agent: Perplexity-User`
`MistralAI-User`	Mistral	Blocked	line 328 · `Disallow: /` under `User-agent: MistralAI-User`
`Meta-ExternalFetcher`	Meta	Blocked	line 328 · `Disallow: /` under `User-agent: meta-externalfetcher`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	no rule matches the home page
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View www.pcmag.com/robots.txt · 387 lines

1# START Ziff Managed/Shared content

2# Bot Disallows

3# Block AI and other unwanted bots

4User-agent: AddSearchBot

5User-agent: AgentDataBot

6User-agent: AI2Bot

7User-agent: Ai2Bot-DeepResearchEval

8User-agent: Ai2Bot-Dolma

9User-agent: AIWebIndex

10User-agent: AIWebIndex-Agent

11User-agent: amazon-QBusiness

12User-agent: AmazonBuyForMe

13User-agent: Amazonbot

14User-agent: Amzn-SearchBot

15User-agent: Amzn-User

16User-agent: Anchor Browser

17User-agent: AndiAgent

18User-agent: anthropic-ai

19User-agent: ApifyBot

20User-agent: ApifyWebsiteContentCrawler

21User-agent: Applebot

22User-agent: Applebot-extended

23User-agent: atlassian-bot

24User-agent: AutoRAG

25User-agent: AwarioSmartBot

26User-agent: AzureAI-SearchBot

27User-agent: BeansLLM-CorpusBot

28User-agent: Big Sur AI

29User-agent: bigsur.ai

30User-agent: BixelBot

31User-agent: Brandwatch

32User-agent: Bravebot

33User-agent: Brightbot

34User-agent: Bytespider

35User-agent: CCBot

36User-agent: Channel3Bot

37User-agent: ChatGLM-Spider

38User-agent: ChatGPT-User

39User-agent: Claude-Code

40User-agent: Claude-SearchBot

41User-agent: Claude-User

42User-agent: Claude-Web

43User-agent: ClaudeBot

44User-agent: Cloudflare-AutoRAG

45User-agent: CloudflareBrowserRenderingCrawler

46User-agent: Code

47User-agent: cohere-ai

48User-agent: cohere-training-data-crawler

49User-agent: Cotoyogi

50User-agent: Crawl4AI

51User-agent: Cursor

52User-agent: Datenbank Crawler

53User-agent: Datenbank-Crawler

54User-agent: DeepSeekBot

55User-agent: DesearchBot

56User-agent: Devin

57User-agent: Diffbot

58User-agent: Direqt Anomura

59User-agent: Dola-SearchBot

60User-agent: DuckAssistBot

61User-agent: Echobot Bot

62User-agent: ExaBot

63User-agent: ExaSearchBot

64User-agent: FacebookBot

65User-agent: Factset_spyderbot

66User-agent: FirecrawlAgent

67User-agent: GeistHaus-PageFetcher

68User-agent: GPTBot

69User-agent: grok-agent

70User-agent: GrokBot

71User-agent: Hunyuan

72User-agent: iAsk

73User-agent: iAskBot

74User-agent: iaskspider

75User-agent: ICC Crawler

76User-agent: ICC-Crawler

77User-agent: ImageBot

78User-agent: ImageSiftBot

79User-agent: imageSpider

80User-agent: IsonSearchBot

81User-agent: kagi-fetcher

82User-agent: Kangaroo Bot

83User-agent: Kangaroo-Bot

84User-agent: Keenable-User

85User-agent: KeenableBot

86User-agent: Kimi-Agent

87User-agent: Kimi-User

88User-agent: KlaviyoAIBot

89User-agent: Kunato

90User-agent: KunatoCrawler

91User-agent: laion-huggingface-processor

92User-agent: LCC

93User-agent: LinerBot

94User-agent: LinkupBot

95User-agent: LINER Bot

96User-agent: Manus-User

97User-agent: meta-externalagent

98User-agent: meta-externalfetcher

99User-agent: meta-webindexer

100User-agent: mistral.ai

101User-agent: MistralAI-User

102User-agent: netEstate Imprint Crawler

103User-agent: Novellum AI Crawl

104User-agent: Novellum

105User-agent: NovaAct

106User-agent: OAI-SearchBot

107User-agent: omgili

108User-agent: opencode

109User-agent: PanguBot

110User-agent: Perplexity-User

111User-agent: PerplexityBot

112User-agent: PeopleInc-DCipher-Scraper/1.1.0

113User-agent: PetalBot

114User-agent: PhindBot

115User-agent: Poggio-Citations

116User-agent: qodercli

117User-agent: QualifiedBot

118User-agent: recraft-image-dataset

119User-agent: Reflectionbot

120User-agent: ScryBot

121User-agent: SeekrBot

122User-agent: SemrushBot-OCOB

123User-agent: SemrushBotSwa

124User-agent: Shap-User

125User-agent: ShapBot

126User-agent: SBIntuitionsBot

127User-agent: SofyaBot

128User-agent: SomethinggBot

129User-agent: Spider

130User-agent: TalarionFrontierCrawler

131User-agent: TalarionSearchBot

132User-agent: TerraCotta

133User-agent: Timpibot

134User-agent: TavilyBot

135User-agent: TongyiBot

136User-agent: Trae

137User-agent: TwinAgent

138User-agent: UseAI

139User-agent: Velen Crawler

140User-agent: VelenPublicWebCrawler

141User-agent: Webzio-Extended

142User-agent: Wrtn

143User-agent: WRTNBot

144User-agent: xAI-Grok

145User-agent: YandexAdditionalBot

146User-agent: YepBot

147User-agent: YiBot

148User-agent: YiyanBot

149User-agent: YouBot

150User-agent: ZanistaBot

151

152Allow: /articles/achieve-picture-perfect-monitor-settings-with-msi-uniform-luminance

153Allow: /articles/alienware-gaming-laptops-in-canada-2026-specs-prices-and-upgrade-paths

154Allow: /articles/can-you-share-a-cash-app-account-with-your-child-how-the-family-account

155Allow: /articles/consider-msis-newest-amd-motherboard-lineup-for-your-next-build

156Allow: /articles/headed-back-to-the-office-7-things-to-look-for-in-a-laptop-for-hybrid-work

157Allow: /articles/help-your-kids-build-good-money-habits-with-cash-app-families-accounts

158Allow: /articles/how-ai-app-builders-are-streamlining-the-path-to-ios-and-android

159Allow: /articles/how-base44-makes-building-a-website-as-easy-as-writing-an-email

160Allow: /articles/how-the-galaxy-s26-ultras-ai-camera-features-make-it-stand-out

161Allow: /articles/i-always-travel-with-a-vpn-heres-why-its-a-game-changer

162Allow: /articles/i-have-zero-programming-skills-my-first-attempt-at-vibe-coding-went-surprisingly

163Allow: /articles/i-used-a-vpn-to-hunt-for-cheap-flights-does-it-actually-work

164Allow: /articles/navigating-cryptocurrency-and-digital-asset-taxes-with-hr-block

165Allow: /articles/t-mobile-gets-down-to-business-with-supermobile-a-high-end-42month-phone

166Allow: /articles/t-mobile-sets-a-new-standard-for-business-plans-with-supermobile

167Allow: /articles/tax-breaks-you-might-miss-in-2026-and-how-to-find-them

168Allow: /articles/the-1-mistake-people-make-choosing-a-vpn-and-how-to-avoid-it

169Allow: /articles/meet-the-dell-xps-14-an-ai-ready-laptop-for-creatives-and-power-users

170Allow: /articles/the-latest-dell-xps-14-packs-big-creative-power-into-a-premium-frame

171Allow: /articles/the-power-of-plain-english-build-professional-apps-with-base44

172Allow: /articles/how-to-secure-your-digital-life-with-surfshark

173Allow: /articles/how-to-pick-the-right-dell-laptop-for-your-remote-work-setup

174Allow: /articles/you-use-multiple-platforms-every-day-but-are-they-all-actually-secure

175Allow: /articles/how-to-spot-ai-driven-scams-across-your-favorite-apps

176Allow: /articles/high-performance-anime-hardware-integrating-the-msi-frieren-limited-edition

177Allow: /articles/how-to-keep-your-multi-device-household-running-smoothly

178Allow: /articles/4-gaming-devices-at-walmart-that-serious-gamers-will-actually-want

179Allow: /articles/how-to-upgrade-your-summer-audio-setup-for-under-150-at-walmart

180Allow: /articles/sick-of-expensive-printer-ink-its-time-to-switch-to-hp-tank

181Allow: /articles/laser-printer-vs-inkjet-selecting-the-most-efficient-workspace-printing

182Allow: /articles/how-bitdefenders-digital-life-protection-can-secure-your-multi-device-world

183Allow: /articles/how-to-stop-ai-voice-scams-and-deepfakes-with-bitdefender-scam-protection

184Allow: /articles/galaxy-s26-ultra-features-what-you-need-to-know-before-you-buy

185Allow: /articles/what-are-directv-genre-packs-a-breakdown-of-every-pack-price-and-channel

186Allow: /articles/hyperx-cloud-iii-s-cloud-alpha-2-or-cloudx-flight-2-which-wireless-gaming

187Allow: /articles/which-square-setup-fits-your-business-from-solo-sellers-to-full-storefronts

188

189Allow: /comparisons/aura-vs-incogni-which-wins-the-battle-for-your-data-privacy

190Allow: /comparisons/bright-data-vs-decodo-which-proxy-solution-comes-out-on-top

191Allow: /comparisons/decodo-vs-iproyal-which-proxy-provider-takes-the-lead

192Allow: /comparisons/deleteme-vs-incogni-which-personal-data-removal-service-is-right-for-you

193Allow: /comparisons/deleteme-vs-optery-which-cleans-up-your-digital-footprint-better

194Allow: /comparisons/freshdesk-vs-zendesk-a-simple-guide-to-choosing-the-best-help-desk

195Allow: /comparisons/incogni-vs-optery-which-data-removal-service-comes-out-on-top

196Allow: /comparisons/incogni-vs-privacy-bee-which-really-scrubs-your-digital-footprint-clean

197Allow: /comparisons/ink-tank-vs-laser-which-type-of-printer-saves-you-more-money

198Allow: /comparisons/lastpass-vs-roboform-the-password-manager-you-actually-need

199Allow: /comparisons/quickbooks-online-vs-xero-which-will-make-your-books-a-breeze

200Allow: /comparisons/roboform-vs-keeper-which-should-you-trust-with-your-passwords

201Allow: /comparisons/tor-vs-vpn-the-battle-for-online-anonymity

202Allow: /comparisons/expressvpn-vs-proton-vpn-5-things-you-need-to-know-before-choosing

203Allow: /comparisons/proton-vs-tunnelbear-privacy-powerhouse-or-beginner-friendly-simplicity

204Allow: /comparisons/mullvad-vs-proton-the-vpn-privacy-showdown-you-cant-ignore

205Allow: /comparisons/1password-vs-roboform-which-password-manager-gives-you-more-for-less

206Allow: /comparisons/enpass-vs-roboform-battle-of-the-budget-friendly-password-managers

207Allow: /explainers/7-dangerous-vpn-myths-you-probably-believe

208Allow: /deals/protect-your-household-for-less-save-58-on-norton-360-deluxe-all-in-one

209Allow: /explainers/7-reasons-to-use-a-proxy-server-instead-of-a-vpn

210Allow: /explainers/10-payroll-features-you-need-and-10-youre-paying-too-much-for

211Allow: /explainers/are-you-dating-a-machine-5-warning-signs-your-perfect-match-is-actually

212Allow: /explainers/avoid-irs-fines-and-penalties-how-payroll-software-protects-your-business

213Allow: /explainers/data-brokers-have-a-file-on-you-heres-how-to-see-whats-inside

214Allow: /explainers/dont-be-fooled-7-vpn-myths-that-could-put-you-at-risk

215Allow: /explainers/doxxing-demystified-what-it-is-how-it-works-and-how-to-protect-yourself

216Allow: /explainers/free-vs-paid-antivirus-is-paying-for-protection-really-worth-it

217Allow: /explainers/freshman-fraud-101-the-digital-scams-every-student-and-parent-should-know

218Allow: /explainers/future-proof-your-workflow-9-ways-ai-pcs-can-transform-your-business

219Allow: /explainers/hackers-are-targeting-windows-10-do-this-one-thing-to-stay-safe

220Allow: /explainers/how-exposed-is-your-data-really-a-practical-guide-to-finding-out-and-fixing

221Allow: /explainers/how-to-vibe-code-5-simple-tips-for-using-ai-to-become-a-programmer

222Allow: /explainers/i-automatically-do-these-3-things-to-increase-my-internet-speed-when-using

223Allow: /explainers/i-use-a-vpn-with-chrome-here-are-3-smart-reasons-you-should-too

224Allow: /explainers/is-your-old-vpn-still-protecting-you-make-sure-it-has-these-6-features

225Allow: /explainers/is-your-vpn-leaking-data-heres-how-to-check-and-lock-it-down-in-seconds

226Allow: /explainers/is-your-vpn-really-protecting-you-heres-how-to-tell

227Allow: /explainers/privacy-pros-always-change-these-5-vpn-settings-do-you

228Allow: /explainers/run-payroll-like-a-pro-a-stress-free-setup-guide-for-small-businesses

229Allow: /explainers/scammer-got-your-email-heres-what-to-do-immediately

230Allow: /explainers/should-you-turn-off-vpn-heres-when-its-ok-how-to-do-it-safely

231Allow: /explainers/small-business-big-decision-10-expert-tips-to-help-you-pick-the-right-accounting

232Allow: /explainers/still-doing-payroll-manually-these-7-traps-could-cost-you-big-time

233Allow: /explainers/stop-these-5-antivirus-misconceptions-could-leave-you-unprotected

234Allow: /explainers/tax-time-is-open-season-for-scammers-7-tips-to-stay-safe

235Allow: /explainers/the-digital-black-market-how-your-data-is-bought-sold-and-traded-after

236Allow: /explainers/the-hidden-cost-of-free-vpns-how-to-tell-which-ones-are-safe

237Allow: /explainers/think-payroll-software-costs-too-much-the-math-says-otherwise

238Allow: /explainers/this-one-service-could-save-your-business-from-a-privacy-nightmare

239Allow: /explainers/turn-your-vpn-into-a-privacy-powerhouse-with-these-5-expert-level-tweaks

240Allow: /explainers/vpn-metadata-explained-what-your-provider-can-and-cant-see

241Allow: /explainers/want-better-cash-flow-and-happier-customers-this-simple-upgrade-can-help

242Allow: /explainers/what-is-split-tunneling-the-vpn-superpower-youre-probably-not-using

243Allow: /explainers/what-is-web-scraping-a-beginners-guide-to-data-extraction

244Allow: /explainers/yes-you-can-use-one-vpn-to-protect-all-of-your-devices-heres-how

245Allow: /explainers/your-data-is-already-out-there-can-a-vpn-do-anything-about-it

246Allow: /explainers/your-info-is-all-over-the-web-should-you-pay-someone-to-clean-it-up

247Allow: /explainers/your-online-shopping-cart-is-spying-on-you-and-selling-the-evidence

248Allow: /explainers/your-small-business-needs-a-vpn-heres-how-to-choose-the-right-one

249Allow: /explainers/your-vpn-has-a-hidden-safety-button-make-sure-to-turn-on-the-kill-switch

250Allow: /explainers/your-vpn-might-be-spying-on-you-heres-how-to-tell

251Allow: /explainers/avoid-payroll-pitfalls-10-tips-to-keep-your-small-business-running-smoothly

252Allow: /explainers/think-vpns-are-enough-heres-why-proxy-servers-still-matter

253Allow: /how-to/how-to-create-a-vpn-server-with-raspberry-pi

254Allow: /how-to/how-to-watch-2026-fifa-world-cup-live-without-cable-even-for-free

255Allow: /how-to/i-set-up-my-own-proxy-server-heres-why-and-how-you-can-do-it-yourself

256Allow: /how-to/yes-you-can-remove-your-personal-info-from-google-search-heres-how

257Allow: /explainers/5-everyday-situations-where-a-vpn-can-save-the-day

258Allow: /explainers/can-a-actually-protect-you-from-hackers-the-answer-is-complicated

259Allow: /explainers/5-reasons-encrypted-email-isnt-just-for-tech-experts-anymore

260Allow: /explainers/is-your-cloud-storage-actually-secure-heres-what-most-companies-dont-explain

261Allow: /comparisons/freshservice-vs-zendesk-for-employee-service-which-it-help-desk-is-right

262Allow: /articles/upgrade-your-setup-high-performance-dell-pcs-and-gaming-gear-up-to-1000

263Allow: /articles/does-mcafee-instantly-flag-ai-scams-and-deepfakes-yes-heres-how

264Allow: /articles/how-to-find-the-ideal-dell-setup-for-work-and-home

265Allow: /articles/t-mobiles-new-era-of-business-dedicated-satellite-backed-5g-internet-and

266Allow: /articles/which-msi-ai-laptops-are-best-for-gaming-and-creation

267Allow: /explainers/streaming-with-a-vpn-which-services-work-which-dont-and-how-to-get-around

268Allow: /explainers/using-a-vpn-for-online-banking-you-might-be-solving-the-wrong-problem

269Allow: /explainers/is-using-a-vpn-always-legal-a-country-by-country-reality-check

270Allow: /articles/how-to-choose-an-rgb-led-tv-for-your-home

271Allow: /articles/t-mobile-is-designing-dual-pathway-business-internet-for-hard-to-serve

272Allow: /articles/how-to-find-the-right-msi-gaming-desktop-for-your-setup

273Allow: /articles/how-to-upgrade-your-student-tech-setup-with-apple-deals-at-best-buy

274Allow: /articles/bark-is-a-safe-parent-controlled-smartphone-that-grows-with-your-child

275Allow: /articles/why-standard-security-cant-stop-ai-powered-phishing-and-digital-fraud

276Allow: /articles/is-that-invoice-real-how-ai-is-changing-small-business-payment-fraud

277Allow: /articles/no-it-department-how-to-secure-your-teams-devices-accounts-and-inboxes

278Allow: /articles/which-msi-z890-motherboard-fits-your-gaming-build

279Allow: /articles/compromised-account-dangers-and-how-to-fix

280Allow: /explainers/your-vpn-could-be-even-more-private-change-these-5-settings-now

281Allow: /explainers/3-surprising-ways-a-vpn-can-level-up-your-gaming-experience

282Allow: /how-to/how-to-watch-2026-fifa-world-cup-final-live-for-free-today

283Allow: /explainers/the-ultimate-business-stack-hack-7-reasons-to-unify-your-payroll-hr-and

284Allow: /explainers/is-your-payroll-system-starting-to-crack-7-signs-its-time-to-upgrade

285Allow: /explainers/9-year-end-payroll-mistakes-to-avoid-and-how-software-can-help

286Allow: /articles/how-dell-builds-storage-for-enterprise-ai-workloads

287Allow: /articles/object-storage-vs-enterprise-file-storage-for-ai-which-does-your-workload

288Allow: /articles/scaling-beyond-ai-pilots-how-your-storage-stack-changes-for-production

289Allow: /articles/5-signs-a-video-might-be-an-ai-deepfake-and-how-to-check-fast

290Allow: /articles/5-things-data-brokers-know-about-you-and-how-to-make-them-delete-your-info

291Allow: /articles/walmart-picks-for-under-50-16-things-actually-worth-buying-this-fall

292Allow: /articles/which-apple-products-should-you-buy-at-walmart-macbooks-airpods-ipads-and

293Allow: /articles/hp-elitebook-x-the-ai-powered-business-laptop-built-for-busy-professionals

294Allow: /articles/free-accounting-software-what-quickbooks-free-does-and-where-the-limits

295Allow: /articles/payroll-that-syncs-with-your-accounting-software-what-actually-connects

296Allow: /articles/cloud-accounting-for-small-business-7-questions-about-quickbooks-online

297Allow: /articles/roboform-in-2026-security-passkeys-and-small-business-features-to-know

298Allow: /articles/can-you-get-an-iphone-18-pro-on-t-mobile-when-you-switch

299Allow: /articles/symmetrical-upload-speeds-arent-just-a-fiber-thing-anymore-heres-what-you

300Allow: /articles/which-msi-laptop-is-right-for-gaming-productivity-or-campus-use

301Allow: /explainers/traveling-soon-i-never-board-a-plane-without-this-on-my-laptop-vpn

302Allow: /articles/i-used-vibe-coding-to-build-my-own-zillow-in-just-a-few-hours

303Allow: /explainers/i-used-a-vpn-to-find-cheap-flights-does-this-travel-hack-actually-work

304Allow: /explainers/freshman-fraud-101-digital-scams-college-students-how-to-stop-them

305Allow: /explainers/can-you-stay-windows-10-i-wouldnt-unless-you-follow-steps-to-protect-pc

306Allow: /explainers/vibe-coding-101-how-to-build-apps-and-more-with-ai

307Allow: /explainers/i-do-these-3-things-to-increase-my-internet-speed-when-using-a-vpn

308Allow: /explainers/chrome-users-using-a-vpn-can-boost-your-privacy

309Allow: /explainers/the-secret-vpn-trick-to-keep-your-speeds-fast-without-losing-protection

310Allow: /explainers/think-your-vpn-protects-you-from-data-breaches-think-again

311Allow: /explainers/vpn-slowing-you-down-a-proxy-could-be-the-faster-fix

312Allow: /how-to/still-paying-for-a-vpn-i-built-my-own-in-hours-with-raspberry-pi

313Allow: /how-to/psst-you-can-remove-your-personal-info-from-google-search-heres-how

314Allow: /explainers/can-a-vpn-actually-protect-you-from-hackers-the-answer-is-complicated

315Allow: /explainers/warning-this-popular-safety-step-step-might-flag-your-bank-account-as-hacked

316Allow: /articles/will-this-laptop-last-for-years-what-to-look-for-and-maintain-over-time

317Allow: /articles/should-you-buy-an-ai-laptop-now-or-wait

318Allow: /articles/which-of-these-alienware-laptops-matches-how-you-actually-play

319Allow: /articles/how-to-build-ransomware-defense-and-data-governance-into-your-ai-storage

320Allow: /articles/how-to-tell-hisenses-2026-tvs-apart-and-pick-the-right-one-u6-u7-ur8-or

321Allow: /articles/dell-xps-is-back-whats-new-in-the-2026-xps-13-14-and-16

322Allow: /articles/roku-select-vs-plus-vs-pro-which-smart-tv-belongs-in-your-room

323Allow: /articles/dell-16s-laptop-for-work-school-and-home

324Allow: /articles/which-sling-tv-plan-carries-the-channels-you-watch

325Allow: /articles/find-the-hp-omnibook-copilot-plus-pc-that-fits-your-lifestyle

326Allow: /articles/what-small-businesses-should-look-for-when-evaluating-new-pcs

327Allow: /articles/time-tracking-that-feeds-payroll-how-small-businesses-turn-hours-into-accurate

328Disallow: /

329

330# END Ziff Managed/Shared content

331

332# Ziff Davis content is made available for your non-commercial use subject to our

333# Terms of Use here: https://www.ziffdavis.com/terms-of-use

334# Use of any robot, crawler, or other tool to scrape, harvest, extract, or retrieve any content on

335# this website using automated means is prohibited without written permission from Ziff Davis.

336# Prohibited uses include but are not limited to:

337# (1) text and data mining under Art. 4 of the EU Directive on Copyright in the Digital Single

338# Market;

339# (2) development or operation of artificial intelligence or machine learning software or

340# databases, including by training, fine-tuning, embedding, and retrieval-augmented generation;

341# (3) creating data sets containing our content or sharing it with others; and

342# (4) any commercial purposes.

343# Contact licensing@ziffdavis.com for assistance.

344

345User-agent: *

346Disallow: /search/

347Disallow: /archive/

348Disallow: /otc/

349Disallow: /api/

350Disallow: /cdn-cgi/

351Disallow: /*?*test_uuid=

352Disallow: /*?*test_variant=

353

354User-agent: "008"

355User-agent: Ai2Bot-Dolma

356User-agent: Amazonbot

357User-agent: anthropic-ai

358User-agent: Applebot

359User-agent: Applebot-extended

360User-agent: Bytespider

361User-agent: CCBot

362User-agent: ChatGPT-User

363User-agent: Claude-Web

364User-agent: ClaudeBot

365User-agent: cohere-ai

366User-agent: DDM-DCipher/1.0.7

367User-agent: Diffbot

368User-agent: DuckAssistBot

369User-agent: FacebookBot

370User-agent: GPTBot

371User-agent: HTTrack

372User-agent: meta-externalagent

373User-agent: meta-externalfetcher

374User-agent: Nutch

375User-agent: OAI-SearchBot

376User-agent: Offline Explorer

377User-agent: omgili

378User-agent: Perplexity-User

379User-agent: PerplexityBot

380User-agent: Scrapy

381User-agent: Timpibot

382User-agent: YouBot

383Disallow: /

384

385Sitemap: https://www.pcmag.com/sitemap-index.xml

386Sitemap: https://www.pcmag.com/sitemap-google-news.xml

387

What do the bots actually get?

This page reads robots.txt, which states what pcmag.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on pcmag.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does pcmag.com block GPTBot?

Yes. pcmag.com’s robots.txt disallows GPTBot from the whole site (line 328: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does pcmag.com block ClaudeBot?

Yes. pcmag.com’s robots.txt disallows ClaudeBot from the whole site (line 328: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does pcmag.com block Google-Extended?

No. pcmag.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does pcmag.com block PerplexityBot?

Yes. pcmag.com’s robots.txt disallows PerplexityBot from the whole site (line 328: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find pcmag.com?

Not through its own crawler. pcmag.com’s robots.txt disallows OAI-SearchBot from the whole site (line 328: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.