Who blocks AI · #227 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does cnet.com block AI crawlers?
Yes. cnet.com’s robots.txt blocks 17 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, PerplexityBot and 14 more. It also blocks Applebot and PetalBot among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedAllowedCCBotBlocked- robots.txt
- www.cnet.com/robots.txt · 446 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Blocked | line 414 · Disallow: / under User-agent: Applebot |
PetalBot | Huawei | Blocked | line 414 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 414 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 414 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 414 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 414 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 414 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 414 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 414 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 414 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Blocked | line 414 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 414 · Disallow: / under User-agent: Applebot-extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 414 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 414 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 414 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 414 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 414 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 414 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Blocked | line 414 · Disallow: / under User-agent: meta-externalfetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.cnet.com/robots.txt · 446 lines
1# www.robotstxt.org/
2# www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
3#
4
5# Ziff Davis content is made available for your non-commercial use subject to our
6# Terms of Use here: https://www.ziffdavis.com/terms-of-use
7# Use of any robot, crawler, or other tool to scrape, harvest, extract, or retrieve any content on
8# this website using automated means is prohibited without written permission from Ziff Davis.
9# Prohibited uses include but are not limited to:
10# (1) text and data mining under Art. 4 of the EU Directive on Copyright in the Digital Single
11# Market;
12# (2) development or operation of artificial intelligence or machine learning software or
13# databases, including by training, fine-tuning, embedding, and retrieval-augmented generation;
14# (3) creating data sets containing our content or sharing it with others; and
15# (4) any commercial purposes.
16# Contact licensing@ziffdavis.com for assistance.
17
18User-agent: *
19
20Disallow: */*max-price=*
21Disallow: */*min-price=*
22Disallow: */Xhr*
23Disallow: */account-information/*
24Disallow: */ad/ad-cookie/*
25Disallow: */ajax/
26Disallow: */alert/*
27Disallow: */cnet_global_app_redirect.xml
28Disallow: */component/*
29Disallow: */edit/
30Disallow: */edit/change-email/*
31Disallow: */edit/change-password/*
32Disallow: */email/editor/*
33Disallow: */email/member/*
34Disallow: */email/newsletter/*
35Disallow: */error/*
36Disallow: */ext/make/ingest/*
37Disallow: */feature/*
38Disallow: */finder/*/*/?*filter=*_*_*_*_*
39Disallow: */finder/*/?*filter=*_*_*_*_*
40Disallow: */forums/*
41Disallow: */g00/*
42Disallow: */informacion-de-mi-cuenta/*
43Disallow: */members/*
44Disallow: */products/*?*filter=*
45Disallow: */roadshow/finder/*
46Disallow: */search/*
47Disallow: */user/*
48Disallow: */videos/manifest/*
49Disallow: */videos/secure/*
50Disallow: */videos/selector/*
51Disallow: */wrap-function*
52Disallow: */wrap-xhr*
53Disallow: */xhr*
54Disallow: *?sort=popularity*
55Disallow: *?sort=price*
56Disallow: *?sort=user-rating*
57Disallow: *hub-preview-version=*
58Disallow: /.well-known/*
59Disallow: /api/editionurlgenerator/
60Disallow: /apple-news/*
61Disallow: /au/*
62Disallow: /es/*
63Disallow: /html/story/*
64Disallow: /index.php/*
65Disallow: /products/*
66Disallow: /uk/*
67
68# Ziff Davis bot list
69user-agent: AddSearchBot
70user-agent: AgentDataBot
71user-agent: AI2Bot
72user-agent: Ai2Bot-DeepResearchEval
73user-agent: Ai2Bot-Dolma
74user-agent: AIWebIndex
75user-agent: AIWebIndex-Agent
76user-agent: amazon-QBusiness
77user-agent: Amazonbot
78user-agent: AmazonBuyForMe
79user-agent: Amzn-SearchBot
80user-agent: Amzn-User
81user-agent: Anchor Browser
82user-agent: AndiAgent
83user-agent: anthropic-ai
84user-agent: ApifyBot
85user-agent: ApifyWebsiteContentCrawler
86user-agent: Applebot
87user-agent: Applebot-extended
88user-agent: atlassian-bot
89user-agent: AutoRAG
90user-agent: AwarioSmartBot
91user-agent: AzureAI-SearchBot
92user-agent: BeansLLM-CorpusBot
93user-agent: Big Sur AI
94user-agent: bigsur.ai
95user-agent: BixelBot
96user-agent: Brandwatch
97user-agent: Bravebot
98user-agent: Brightbot
99user-agent: Bytespider
100user-agent: CCBot
101user-agent: Channel3Bot
102user-agent: ChatGLM-Spider
103user-agent: ChatGPT-User
104user-agent: Claude-Code
105user-agent: Claude-SearchBot
106user-agent: Claude-User
107user-agent: Claude-Web
108user-agent: ClaudeBot
109user-agent: Cloudflare-AutoRAG
110user-agent: CloudflareBrowserRenderingCrawler
111user-agent: Code
112user-agent: cohere-ai
113user-agent: cohere-training-data-crawler
114user-agent: Cotoyogi
115user-agent: Crawl4AI
116user-agent: Cursor
117user-agent: Datenbank Crawler
118user-agent: Datenbank-Crawler
119user-agent: DeepSeekBot
120user-agent: DesearchBot
121user-agent: Devin
122user-agent: Diffbot
123user-agent: Direqt Anomura
124user-agent: Dola-SearchBot
125user-agent: DuckAssistBot
126user-agent: Echobot Bot
127user-agent: ExaBot
128user-agent: ExaSearchBot
129user-agent: FacebookBot
130user-agent: Factset_spyderbot
131user-agent: FirecrawlAgent
132user-agent: GeistHaus-PageFetcher
133user-agent: GPTBot
134user-agent: grok-agent
135user-agent: GrokBot
136user-agent: Hunyuan
137user-agent: iAsk
138user-agent: iAskBot
139user-agent: iaskspider
140user-agent: ICC Crawler
141user-agent: ICC-Crawler
142user-agent: ImageBot
143user-agent: ImageSiftBot
144user-agent: imageSpider
145user-agent: IsonSearchBot
146user-agent: kagi-fetcher
147user-agent: Kangaroo Bot
148user-agent: Kangaroo-Bot
149user-agent: Keenable-User
150user-agent: KeenableBot
151user-agent: Kimi-Agent
152user-agent: Kimi-User
153user-agent: KlaviyoAIBot
154user-agent: Kunato
155user-agent: KunatoCrawler
156user-agent: laion-huggingface-processor
157user-agent: LCC
158user-agent: LINER Bot
159user-agent: LinerBot
160user-agent: LinkupBot
161user-agent: Manus-User
162user-agent: meta-externalagent
163user-agent: meta-externalfetcher
164user-agent: meta-webindexer
165user-agent: mistral.ai
166user-agent: MistralAI-User
167user-agent: netEstate Imprint Crawler
168user-agent: NovaAct
169user-agent: Novellum
170user-agent: Novellum AI Crawl
171user-agent: OAI-SearchBot
172user-agent: omgili
173user-agent: opencode
174user-agent: PanguBot
175user-agent: PeopleInc-DCipher-Scraper/1.1.0
176user-agent: Perplexity-User
177user-agent: PerplexityBot
178user-agent: PetalBot
179user-agent: PhindBot
180user-agent: Poggio-Citations
181user-agent: qodercli
182user-agent: QualifiedBot
183user-agent: recraft-image-dataset
184user-agent: Reflectionbot
185user-agent: SBIntuitionsBot
186user-agent: ScryBot
187user-agent: SeekrBot
188user-agent: SemrushBot-OCOB
189user-agent: SemrushBotSwa
190user-agent: Shap-User
191user-agent: ShapBot
192user-agent: SofyaBot
193user-agent: SomethinggBot
194user-agent: Spider
195user-agent: TalarionFrontierCrawler
196user-agent: TalarionSearchBot
197user-agent: TavilyBot
198user-agent: TerraCotta
199user-agent: Timpibot
200user-agent: TongyiBot
201user-agent: Trae
202user-agent: TwinAgent
203user-agent: UseAI
204user-agent: Velen Crawler
205user-agent: VelenPublicWebCrawler
206user-agent: Webzio-Extended
207user-agent: Wrtn
208user-agent: WRTNBot
209user-agent: xAI-Grok
210user-agent: YandexAdditionalBot
211user-agent: YepBot
212user-agent: YiBot
213user-agent: YiyanBot
214user-agent: YouBot
215user-agent: ZanistaBot
216Allow: /home/fiber-cable-or-5g-internet-how-to-choose/
217Allow: /home/internet/do-you-need-symmetrical-internet-speeds-what-to-know-before-you-upgrade/
218Allow: /home/internet/dont-buy-internet-until-you-check-the-cost-per-mbps/
219Allow: /home/internet/equipment-for-fast-home-internet/
220Allow: /home/internet/fix-wi-fi-dead-spots-how-to-get-internet-throughout-your-home/
221Allow: /home/internet/gigabit-internet-is-it-worth-splurging-for-a-faster-internet-plan/
222Allow: /home/internet/how-to-spot-hidden-fees-and-save-money/
223Allow: /home/internet/internet-data-caps-explained-avoid-extra-charges-and-make-most-of-your-internet-plan/
224Allow: /home/internet/is-your-internet-connection-delivering-on-the-speeds-you-pay-for-heres-how-to-tell/
225Allow: /home/internet/lag-free-internet-how-to-find-the-right-internet-speed-for-a-family-of-4-or-more/
226Allow: /home/internet/what-are-the-different-internet-connection-types/
227Allow: /home/internet/why-your-gigabit-internet-plan-might-be-overkill/
228Allow: /home/internet/why-your-upload-speed-is-ruining-your-work-calls-and-how-to-fix-it/
229Allow: /home/internet/xfinity-vs-att-which-internet-provider-is-best/
230Allow: /home/internet/xfinity-vs-verizon-which-internet-provider-is-best/
231Allow: /home/internet/your-internet-bill-might-be-booby-trapped-heres-what-to-do/
232Allow: /news/ai-automation-changing-how-we-file-taxes/
233Allow: /paid-content/air-fry-without-the-countertop-clutter-how-mideas-new-appliances-can-help-your-holiday-cooking/
234Allow: /paid-content/cash-app-families-guide-how-to-set-up-a-teen-or-child-account-step-by-step/
235Allow: /paid-content/cash-app-for-teens-safety-and-parental-controls/
236Allow: /paid-content/cash-apps-new-kids-debit-card-what-parents-need-to-know/
237Allow: /paid-content/can-you-use-an-alienware-laptop-for-both-gaming-and-professional-work/
238Allow: /paid-content/cricket-wireless-offers-reliable-prepaid-nationwide-5g/
239Allow: /paid-content/dell-xps-14-battery-life-a-full-workday-unplugged/
240Allow: /paid-content/dji-builds-on-12-years-of-handheld-imaging-innovation-with-the-dji-osmo-360-ii/
241Allow: /paid-content/dji-osmo-pocket-4p-is-a-powerful-tool-for-independent-filmmakers/
242Allow: /paid-content/do-you-actually-need-a-separate-antivirus-vpn-and-scam-protection-mcafees-all-in-one-plans-have-you-covered/
243Allow: /paid-content/enjoy-unlimited-wi-fi-and-predictable-pricing-with-on-your-home-internet/
244Allow: /paid-content/from-tacos-to-tvs-doordash-delivers-your-game-day-essentials/
245Allow: /paid-content/galaxy-s26-camera-system-the-biggest-questions-answered-before-you-buy/
246Allow: /paid-content/get-an-unlimited-5g-phone-plan-for-25-per-month-with-metro-by-t-mobile/
247Allow: /paid-content/get-everything-you-need-for-the-world-cup-delivered-with-doordash/
248Allow: /paid-content/h-r-blocks-local-tax-pros-help-you-navigate-complex-returns/
249Allow: /paid-content/host-the-holidays-with-less-work-sharkninja-gear-for-cooking-treats-and-cleanup/
250Allow: /paid-content/how-cash-app-builds-savings-habits-for-kids-and-teens/
251Allow: /paid-content/how-cash-app-sponsorship-works-adding-removing-and-managing-parental-controls/
252Allow: /paid-content/how-families-get-more-with-t-mobile/
253Allow: /paid-content/how-fast-should-your-home-internet-be-for-browsing-and-streaming-t-mobile/
254Allow: /paid-content/how-galaxy-ai-camera-tools-help-you-take-better-photos-automatically-on-the-s25/
255Allow: /paid-content/how-h-r-block-helps-freelancers-and-gig-workers-stay-ahead-of-tax-season/
256Allow: /paid-content/how-h-r-block-helps-you-maximize-freelance-and-gig-worker-write-offs/
257Allow: /paid-content/how-h-r-block-simplifies-taxes-for-first-time-filers/
258Allow: /paid-content/how-h-r-blocks-experts-can-help-you-avoid-common-filing-mistakes/
259Allow: /paid-content/how-samsungs-galaxy-ai-boosts-gaming-performance/
260Allow: /paid-content/how-sling-is-disrupting-the-old-live-tv-streaming-model/
261Allow: /paid-content/how-to-build-a-solo-video-setup-with-dji-the-camera-stabilizer-and-mic-you-actually-need/
262Allow: /paid-content/how-to-choose-a-reliable-home-internet-provider-in-2026/
263Allow: /paid-content/how-to-choose-the-right-dell-ai-laptop-for-your-workflow/
264Allow: /paid-content/how-to-get-fast-reliable-fiber-internet-at-a-predictable-transparent-price/
265Allow: /paid-content/how-to-get-your-backyard-ready-for-summer-with-lights-and-sound/
266Allow: /paid-content/how-to-stream-pro-and-college-football-in-2026-with-directv-mysports-genre-pack/
267Allow: /paid-content/how-to-time-your-next-major-appliance-purchase-for-maximum-savings/
268Allow: /paid-content/how-to-try-out-ai-glasses-from-meta-at-your-local-best-buy/
269Allow: /paid-content/how-to-watch-live-sports-and-avoid-wheres-the-game-chaos-with-xfinity/
270Allow: /paid-content/i-tried-internxt-an-encrypted-secure-cloud-storage-built-for-the-post-quantum-era/
271Allow: /paid-content/is-a-prepaid-phone-plan-really-cheaper-for-a-family-of-four/
272Allow: /paid-content/is-diy-tax-filing-right-for-you-heres-what-h-r-blocks-tools-make-possible/
273Allow: /paid-content/is-fiber-to-the-home-ftth-actually-better-than-xfinity-heres-what-the-evidence-says/
274Allow: /paid-content/is-it-time-to-upgrade-your-iphone-with-t-mobiles-trade-in-offers-the-answer-is-yes/
275Allow: /paid-content/is-the-dell-xps-13-a-good-windows-alternative-to-a-macbook/
276Allow: /paid-content/is-wi-fi-7-worth-the-upgrade-heres-how-msis-roamii-mesh-systems-fit-in/
277Allow: /paid-content/life-changes-and-new-laws-can-affect-your-tax-refund-how-to-estimate/
278Allow: /paid-content/maximize-your-refund-with-h-r-blocks-smart-tax-tools/
279Allow: /paid-content/maximize-your-wireless-savings-with-the-t-mobile-visa-credit-card/
280Allow: /paid-content/meet-the-new-hp-omnibook-5-an-oled-laptop-built-for-long-days-of-work-class-and-streaming/
281Allow: /paid-content/meta-ray-ban-wayfarer-ai-glasses-discount-may-2026/
282Allow: /paid-content/moving-soon-how-to-get-t-mobile-5g-home-internet-working-the-day-you-move-in/
283Allow: /paid-content/news/cash-app-families-guide-how-to-set-up-a-teen-or-child-account-step-by-step/
284Allow: /paid-content/news/cash-apps-new-kids-debit-card-what-parents-need-to-know/
285Allow: /paid-content/news/cricket-wireless-offers-reliable-prepaid-nationwide-5g/
286Allow: /paid-content/news/dji-osmo-pocket-4p-is-a-powerful-tool-for-independent-filmmakers/
287Allow: /paid-content/news/enjoy-unlimited-wi-fi-and-predictable-pricing-with-on-your-home-internet/
288Allow: /paid-content/news/from-tacos-to-tvs-doordash-delivers-your-game-day-essentials/
289Allow: /paid-content/news/galaxy-s26-camera-system-the-biggest-questions-answered-before-you-buy/
290Allow: /paid-content/news/get-an-unlimited-5g-phone-plan-for-25-per-month-with-metro-by-t-mobile/
291Allow: /paid-content/news/get-everything-you-need-for-the-world-cup-delivered-with-doordash/
292Allow: /paid-content/news/h-r-blocks-local-tax-pros-help-you-navigate-complex-returns/
293Allow: /paid-content/news/how-cash-app-builds-savings-habits-for-kids-and-teens/
294Allow: /paid-content/news/how-cash-app-sponsorship-works-adding-removing-and-managing-parental-controls/
295Allow: /paid-content/news/how-galaxy-ai-camera-tools-help-you-take-better-photos-automatically-on-the-s25/
296Allow: /paid-content/news/how-h-r-block-helps-you-maximize-freelance-and-gig-worker-write-offs/
297Allow: /paid-content/news/how-h-r-block-simplifies-taxes-for-first-time-filers/
298Allow: /paid-content/news/how-h-r-blocks-experts-can-help-you-avoid-common-filing-mistakes/
299Allow: /paid-content/news/how-samsungs-galaxy-ai-boosts-gaming-performance/
300Allow: /paid-content/news/how-sling-is-disrupting-the-old-live-tv-streaming-model/
301Allow: /paid-content/news/how-t-mobile-can-help-you-save-thousands-on-your-family-wireless-plan/
302Allow: /paid-content/news/how-to-choose-a-reliable-home-internet-provider-in-2026/
303Allow: /paid-content/news/how-to-choose-the-right-dell-ai-laptop-for-your-workflow/
304Allow: /paid-content/news/how-to-get-fast-reliable-fiber-internet-at-a-predictable-transparent-price/
305Allow: /paid-content/news/how-to-get-your-backyard-ready-for-summer-with-lights-and-sound/
306Allow: /paid-content/news/how-to-time-your-next-major-appliance-purchase-for-maximum-savings/
307Allow: /paid-content/news/how-to-watch-live-sports-and-avoid-wheres-the-game-chaos-with-xfinity/
308Allow: /paid-content/news/i-tried-internxt-an-encrypted-secure-cloud-storage-built-for-the-post-quantum-era/
309Allow: /paid-content/news/is-diy-tax-filing-right-for-you-heres-what-h-r-blocks-tools-make-possible/
310Allow: /paid-content/news/is-wi-fi-7-worth-the-upgrade-heres-how-msis-roamii-mesh-systems-fit-in/
311Allow: /paid-content/news/life-changes-and-new-laws-can-affect-your-tax-refund-how-to-estimate/
312Allow: /paid-content/news/maximize-your-refund-with-h-r-blocks-smart-tax-tools/
313Allow: /paid-content/news/maximize-your-wireless-savings-with-the-t-mobile-visa-credit-card/
314Allow: /paid-content/news/meta-ray-ban-wayfarer-ai-glasses-discount-may-2026/
315Allow: /paid-content/news/must-know-tax-season-tips-for-small-businesses-and-solopreneurs/
316Allow: /paid-content/news/nomad-esim-wants-to-give-you-more-than-just-data/
317Allow: /paid-content/news/scale-your-small-business-with-fast-funding-via-loanbuilder/
318Allow: /paid-content/news/selling-stuff-online-what-you-could-be-on-the-hook-for-tax-wise/
319Allow: /paid-content/news/shopping-for-a-major-appliance-let-best-buy-do-the-heavy-lifting/
320Allow: /paid-content/news/streamline-your-weeknight-cooking-routine-with-a-multi-speed-air-fryer/
321Allow: /paid-content/news/switch-to-t-mobile-in-15-minutes-and-get-up-to-800-dollars/
322Allow: /paid-content/news/t-mobile-5g-home-internet-offers-a-solution-to-contracts-cords-and-hidden-fees/
323Allow: /paid-content/news/tempos-fresh-chef-crafted-meals-make-your-meal-prep-smarter-and-tastier/
324Allow: /paid-content/news/the-dell-xps-13-is-subtle-luxury-in-laptop-form/
325Allow: /paid-content/news/the-difference-between-fiber-to-the-home-and-fiber-powered-internet/
326Allow: /paid-content/news/upgrade-your-home-on-a-budget-with-these-summer-appliance-rollbacks-from-walmart/
327Allow: /paid-content/news/upgrade-your-summer-content-with-a-portable-digital-camera/
328Allow: /paid-content/news/what-americans-pay-for-internet-in-2026-and-how-xfinity-helps-you-save/
329Allow: /paid-content/news/why-a-reliable-fiber-powered-network-from-xfinity-is-the-ultimate-solution-for-home-wi-fi/
330Allow: /paid-content/news/why-tax-filers-are-choosing-h-r-block-this-season/
331Allow: /paid-content/news/why-you-need-a-cutting-edge-vpn-to-keep-you-safe-online/
332Allow: /paid-content/news/why-you-need-an-internet-provider-with-no-data-caps/
333Allow: /paid-content/nomad-esim-wants-to-give-you-more-than-just-data/
334Allow: /paid-content/nordictrack-just-out-engineered-its-own-best-treadmill-heres-everything-thats-new-on-the-ultra-1/
335Allow: /paid-content/nordictrack-rowing-machine-or-exercise-bike-which-one-will-you-actually-stick-with/
336Allow: /paid-content/oled-or-mini-led-how-to-pick-the-right-lg-tv-for-your-room-lighting-and-viewing-habits/
337Allow: /paid-content/ready-to-take-better-photos-how-to-pick-the-right-canon-camera-for-beginners/
338Allow: /paid-content/samsung-micro-rgb-tvs-show-you-more-colors-than-ever-before/
339Allow: /paid-content/scale-your-small-business-with-fast-funding-via-loanbuilder/
340Allow: /paid-content/selling-stuff-online-what-you-could-be-on-the-hook-for-tax-wise/
341Allow: /paid-content/shop-now-save-now-12-walmart-tech-picks-at-great-prices-no-promo-code-needed/
342Allow: /paid-content/shopping-for-a-major-appliance-let-best-buy-do-the-heavy-lifting/
343Allow: /paid-content/should-you-buy-a-frigidaire-refrigerator-at-best-buy-what-to-consider/
344Allow: /paid-content/should-you-get-the-iphone-18-now-or-wait-what-to-know-about-preorder-and-carrier-deals/
345Allow: /paid-content/stop-comparing-home-internet-providers-by-fiber-vs-cable-use-this-checklist-instead/
346Allow: /paid-content/streamline-your-weeknight-cooking-routine-with-a-multi-speed-air-fryer/
347Allow: /paid-content/studio-pilates-vs-a-home-reformer-the-math-on-how-much-youd-actually-save/
348Allow: /paid-content/what-can-a-roborock-robot-vacuum-and-mop-do-5-common-scenarios-plus-models-to-shop/
349Allow: /paid-content/roboform-for-everyday-use-ease-of-use-cost-and-getting-set-up/
350Allow: /paid-content/switch-to-t-mobile-in-15-minutes-and-get-up-to-800-dollars/
351Allow: /paid-content/t-mobile-5g-home-internet-gets-you-online-fast-with-no-technicians-or-contracts/
352Allow: /paid-content/t-mobile-5g-home-internet-offers-a-solution-to-contracts-cords-and-hidden-fees/
353Allow: /paid-content/t-mobile-fiber-vs-cable-internet-why-ultra-fast-internet-is-worth-the-upgrade/
354Allow: /paid-content/tempos-fresh-chef-crafted-meals-make-your-meal-prep-smarter-and-tastier/
355Allow: /paid-content/the-dell-xps-13-is-subtle-luxury-in-laptop-form/
356Allow: /paid-content/the-dell-14-plus-is-a-copilot-pc-heres-what-that-actually-means/
357Allow: /paid-content/the-difference-between-fiber-to-the-home-and-fiber-powered-internet/
358Allow: /paid-content/the-nordictrack-commercial-2450-combines-big-speed-with-smart-adjustment-features/
359Allow: /paid-content/the-nordictrack-treadmill-you-actually-need-for-the-12-3-30-workout/
360Allow: /paid-content/upgrade-your-home-on-a-budget-with-these-summer-appliance-rollbacks-from-walmart/
361Allow: /paid-content/upgrade-your-summer-content-with-a-portable-digital-camera/
362Allow: /paid-content/what-americans-pay-for-internet-in-2026-and-how-xfinity-helps-you-save/
363Allow: /paid-content/what-is-a-wearable-exoskeleton-and-who-actually-needs-one/
364Allow: /paid-content/what-is-an-all-in-one-printer-and-do-you-need-one/
365Allow: /paid-content/whats-a-good-casetify-case-for-my-next-iphone-a-complete-guide-to-practical-protection/
366Allow: /paid-content/when-can-kids-get-a-debit-card-heres-how-cash-app-handles-every-age/
367Allow: /paid-content/how-fast-should-your-home-internet-be/
368Allow: /paid-content/what-can-ray-ban-meta-ai-glasses-do-5-use-cases-for-your-everyday-routine/
369Allow: /paid-content/what-to-include-in-a-heart-health-daily-regimen-from-supplements-to-home-monitors/
370Allow: /paid-content/what-does-an-npu-actually-do-and-does-your-laptop-need-one/
371Allow: /paid-content/which-dell-laptops-are-actually-ai-ready-in-canada/
372Allow: /paid-content/why-a-reliable-fiber-powered-network-from-xfinity-is-the-ultimate-solution-for-home-wi-fi/
373Allow: /paid-content/why-tax-filers-are-choosing-h-r-block-this-season/
374Allow: /paid-content/why-the-samsung-galaxy-s26-ultra-feels-like-the-first-phone-truly-built-for-ai/
375Allow: /paid-content/how-to-check-if-t-mobile-fiber-is-available-at-your-address/
376Allow: /paid-content/why-you-need-a-cutting-edge-vpn-to-keep-you-safe-online/
377Allow: /paid-content/why-you-need-an-internet-provider-with-no-data-caps/
378Allow: /tech/mobile/apple-iphone-17-event-september-2025-what-to-expect/
379Allow: /tech/mobile/best-android-phone/
380Allow: /tech/mobile/best-cheap-phone/
381Allow: /tech/mobile/best-phone/
382Allow: /tech/mobile/i-underestimated-workout-buddy-apples-playing-the-long-game-for-ai-coaching/
383Allow: /tech/mobile/new-iphone-buying-guide-carrier-vs-apple-unlocked/
384Allow: /tech/mobile/phone-deals-carrier-bill-credits-installment-plan-buying-guide/
385Allow: /tech/mobile/phone-plan-home-internet-bundle/
386Allow: /tech/mobile/samsung-galaxy-s26-and-s26-plus-hands-on-first-take/
387Allow: /tech/mobile/samsung-galaxy-s26-ultra-vs-galaxy-s26-and-s26-plus-flagship-specs-comparison/
388Allow: /tech/mobile/tomorrow-we-could-see-the-iphone-17-debut-with-a-long-overdue-update/
389Allow: /tech/services-and-software/a-guide-to-data-removal-services-should-you-pay-for-privacy/
390Allow: /tech/services-and-software/ai-automation-changing-how-we-file-taxes/
391Allow: /tech/services-and-software/complex-income-tax-filing-rules-2026/
392Allow: /tech/services-and-software/crypto-new-tax-form-rules/
393Allow: /tech/services-and-software/data-brokers-how-your-personal-data-becomes-business/
394Allow: /tech/services-and-software/end-year-tax-checklist-maximize-tax-refund/
395Allow: /tech/services-and-software/free-tax-software-cost-upgrades/
396Allow: /tech/services-and-software/how-h-r-block-helps-freelancers-and-gig-workers-stay-ahead-of-tax-season/
397Allow: /tech/services-and-software/how-to-prevent-irs-penalties-and-delays-this-tax-season/
398Allow: /tech/services-and-software/ios-27-is-almost-here-but-you-should-check-out-all-these-ios-26-features-first/
399Allow: /tech/services-and-software/married-single-or-self-employed-how-your-filing-status-changes-everything/
400Allow: /tech/services-and-software/must-know-tax-season-tips-for-small-businesses-and-solopreneurs/
401Allow: /tech/services-and-software/proxies-web-scraping-ultimate-guide/
402Allow: /tech/services-and-software/qualified-tips-overtime-tax-deduction-rules/
403Allow: /tech/services-and-software/self-employment-tax-guide-1099-contractor/
404Allow: /tech/services-and-software/w2-vs-1099-difference/
405Allow: /tech/services-and-software/what-is-vibe-coding-how-anyone-can-make-apps-with-the-help-of-ai/
406Allow: /tech/services-and-software/what-online-retailers-actually-do-with-your-data-after-you-check-out/
407Allow: /tech/services-and-software/what-someone-can-do-with-your-phone-number-and-how-to-protect-yourself/
408Allow: /tech/services-and-software/why-use-proxy-server-benefits-drawbacks-common-use-cases/
409Allow: /tech/services-and-software/why-would-a-business-need-to-use-a-proxy-server/
410Allow: /videos/2026-tax-season-checklist-what-you-need-before-you-file/
411Allow: /videos/5-finance-and-tax-tips-for-freelancers/
412Allow: /videos/navigating-your-tax-filing-options-simple-vs-complex-situations/
413Allow: /videos/tax-breaks-you-might-miss-in-2026-and-how-to-find-them/
414Disallow: /
415
416#CNET restricted bots
417User-agent: "008"
418User-agent: AwarioRssBot
419User-agent: DDM-DCipher/1.0.7
420User-agent: HTTrack
421User-agent: Magpie-crawler
422User-agent: Nutch
423User-agent: Offline Explorer
424User-agent: Omgilibot
425User-agent: Peer39_crawler/1.0
426User-agent: Scrapy
427Disallow: /
428
429Sitemap: https://www.cnet.com/sitemap-authors.xml
430
431Sitemap: https://www.cnet.com/sitemap-google-news.xml
432
433Sitemap: https://www.cnet.com/sitemaps/article/index.xml
434Sitemap: https://www.cnet.com/sitemaps/best/index.xml
435Sitemap: https://www.cnet.com/sitemaps/categories/index.xml
436Sitemap: https://www.cnet.com/sitemaps/collections/index.xml
437Sitemap: https://www.cnet.com/sitemaps/deals/index.xml
438Sitemap: https://www.cnet.com/sitemaps/gallery/index.xml
439Sitemap: https://www.cnet.com/sitemaps/how-to/index.xml
440Sitemap: https://www.cnet.com/sitemaps/updating-story/index.xml
441Sitemap: https://www.cnet.com/sitemaps/news/index.xml
442Sitemap: https://www.cnet.com/sitemaps/pages/index.xml
443Sitemap: https://www.cnet.com/sitemaps/review/index.xml
444Sitemap: https://www.cnet.com/sitemaps/versus/index.xml
445Sitemap: https://www.cnet.com/sitemaps/video/index.xml
446
What do the bots actually get?
This page reads robots.txt, which states what cnet.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on cnet.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does cnet.com block GPTBot?
Yes. cnet.com’s robots.txt disallows GPTBot from the whole site (line 414: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does cnet.com block ClaudeBot?
Yes. cnet.com’s robots.txt disallows ClaudeBot from the whole site (line 414: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does cnet.com block Google-Extended?
No. cnet.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does cnet.com block PerplexityBot?
Yes. cnet.com’s robots.txt disallows PerplexityBot from the whole site (line 414: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find cnet.com?
Not through its own crawler. cnet.com’s robots.txt disallows OAI-SearchBot from the whole site (line 414: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.