Who blocks AI · #78 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does forbes.com block AI crawlers?

Yes. forbes.com’s robots.txt blocks 8 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, PerplexityBot and 5 more. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotBlockedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedAllowedCCBotBlocked
robots.txt
www.forbes.com/robots.txt · 1,133 lines
llms.txt
none
Checked
9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGoogleAllowedno rule matches the home page
BingbotMicrosoftAllowedno rule matches the home page
DuckDuckBotDuckDuckGoAllowedno rule matches the home page
ApplebotAppleAllowedno rule matches the home page
PetalBotHuaweiAllowedno rule matches the home page
YandexBotYandexAllowedno rule matches the home page
BaiduspiderBaiduAllowedno rule matches the home page

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIAllowedline 361 · Allow: / under User-agent: OAI-SearchBot
PerplexityBotPerplexityBlockedline 370 · Disallow: / under User-agent: PerplexityBot
Claude-SearchBotAnthropicAllowedline 378 · Allow: / under User-agent: Claude-SearchBot
DuckAssistBotDuckDuckGoAllowedno rule matches the home page
AmazonbotAmazonBlockedline 319 · Disallow: / under User-agent: Amazonbot
YouBotYou.comAllowedno rule matches the home page

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIBlockedline 351 · Disallow: / under User-agent: GPTBot
ClaudeBotAnthropicBlockedline 335 · Disallow: / under User-agent: ClaudeBot
Google-ExtendedGoogleAllowedno rule matches the home page
CCBotCommon CrawlBlockedline 332 · Disallow: / under User-agent: CCBot
Applebot-ExtendedAppleBlockedline 325 · Disallow: / under User-agent: Applebot-Extended
GoogleOtherGoogleAllowedno rule matches the home page
Meta-ExternalAgentMetaBlockedline 358 · Disallow: / under User-agent: meta-externalagent
BytespiderByteDanceBlockedline 328 · Disallow: / under User-agent: Bytespider

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIAllowedno rule matches the home page
Claude-UserAnthropicAllowedline 378 · Allow: / under User-agent: Claude-User
Perplexity-UserPerplexityAllowedno rule matches the home page
MistralAI-UserMistralAllowedno rule matches the home page
Meta-ExternalFetcherMetaAllowedno rule matches the home page

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsAllowedline 378 · Allow: / under User-agent: AhrefsBot
SemrushBotSemrushAllowedno rule matches the home page
MJ12botMajesticAllowedno rule matches the home page
DotBotMozAllowedno rule matches the home page
View www.forbes.com/robots.txt · 1,133 lines
1User-agent: *
2Allow: /ajax/follow_info*
3Allow: /ajax/getUniregUserData*
4
5Disallow: /following/
6Disallow: /search/
7Disallow: /follow/
8Disallow: /ajax/
9Disallow: /author/following/
10Disallow: /blog/following/
11Disallow: /json/
12Disallow: /header/channels/
13Disallow: /find-more/
14Disallow: /preview/
15Disallow: *wp-admin*
16Disallow: /typeahead/
17Disallow: /pepe/
18Disallow: /media-manager/
19Disallow: /coupons/visit/*
20Disallow: /test/
21Disallow: /advisor/*zx1fa*
22Disallow: /advisor/journey/
23Disallow: /wui/
24Disallow: /research-survey
25
26User-agent: googlebot-news
27Disallow: /sites/adobe/
28Disallow: /sites/arizonacommerce/
29Disallow: /sites/bmoharrisbank/
30Disallow: /sites/businessaviation/
31Disallow: /sites/capitalonespark/
32Disallow: /sites/cdw/
33Disallow: /sites/centurylink/
34Disallow: /sites/dell/
35Disallow: /sites/emc/
36Disallow: /sites/ey/
37Disallow: /sites/fedex/
38Disallow: /sites/fidelity/
39Disallow: /sites/gyro/
40Disallow: /sites/hsbc/
41Disallow: /sites/houlihanlawrence/
42Disallow: /sites/hiscox/
43Disallow: /sites/ibm/
44Disallow: /sites/infosys/
45Disallow: /sites/mdanderson/
46Disallow: /sites/medidata/
47Disallow: /sites/microsoft/
48Disallow: /sites/microsoftdynamics/
49Disallow: /sites/netapp/
50Disallow: /sites/northwesternmutual/
51Disallow: /sites/oppenheimerfunds/
52Disallow: /sites/oracle/
53Disallow: /sites/pace/
54Disallow: /sites/raymondjames/
55Disallow: /sites/sage/
56Disallow: /sites/samsung/
57Disallow: /sites/sap/
58Disallow: /sites/sungardas/
59Disallow: /sites/taboola/
60Disallow: /sites/ups/
61Disallow: /sites/delta/
62Disallow: /sites/airbus/
63Disallow: /sites/symantec/
64Disallow: /sites/paypal/
65Disallow: /sites/olddominion/
66Disallow: /sites/salesforce/
67Disallow: /sites/gapinternational/
68Disallow: /sites/sprintbusiness/
69Disallow: /sites/transamerica/
70Disallow: /sites/constantcontact/
71Disallow: /sites/robertocoin/
72Disallow: /sites/canon/
73Disallow: /sites/samsungbusiness/
74Disallow: /sites/att/
75Disallow: /sites/lasvegas/
76Disallow: /sites/iowa/
77Disallow: /sites/teradata/
78Disallow: /sites/thehartford/
79Disallow: /sites/statoil/
80Disallow: /sites/cit/
81Disallow: /sites/deloitte/
82Disallow: /sites/huawei/
83Disallow: /sites/relateiq/
84Disallow: /sites/cftc/
85Disallow: /sites/celebritycruises/
86Disallow: /sites/universityofcalifornia/
87Disallow: /sites/coxbusiness/
88Disallow: /sites/pega/
89Disallow: /sites/athenahealth/
90Disallow: /sites/etrade/
91Disallow: /sites/comcastbusiness/
92Disallow: /sites/optum/
93Disallow: /sites/seagate/
94Disallow: /sites/rbcwealthmanagement/
95Disallow: /sites/avigilon/
96Disallow: /sites/citi/
97Disallow: /sites/castlight/
98Disallow: /sites/hyatt/
99Disallow: /sites/johnhancock/
100Disallow: /sites/koreanair/
101Disallow: /sites/merrilllynch/
102Disallow: /sites/ptc/
103Disallow: /sites/toyota/
104Disallow: /sites/unify/
105Disallow: /sites/xerox/
106Disallow: /sites/united/
107Disallow: /sites/zenithwatches/
108Disallow: /sites/jpmorganchase/
109Disallow: /sites/ramcommercial/
110Disallow: /sites/puremichigan/
111Disallow: /sites/tmfgroup/
112Disallow: /sites/usaa/
113Disallow: /sites/emerson/
114Disallow: /sites/aberdeen/
115Disallow: /sites/kellogg/
116Disallow: /sites/zurich/
117Disallow: /sites/freeenterprise/
118Disallow: /sites/allclearid/
119Disallow: /sites/accenture/
120Disallow: /sites/mercedesbenz/
121Disallow: /sites/nice/
122Disallow: /sites/riverbed/
123Disallow: /sites/pimco/
124Disallow: /sites/netjets/
125Disallow: /sites/vmware/
126Disallow: /sites/visa/
127Disallow: /sites/statefarm/
128Disallow: /sites/philips/
129Disallow: /sites/adp/
130Disallow: /sites/zenithwatches/
131Disallow: /sites/citrix/
132Disallow: /sites/gwcic/
133Disallow: /sites/cargill/
134Disallow: /sites/bankofsingapore/
135Disallow: /sites/dbsbank/
136Disallow: /sites/gradsoflife/
137Disallow: /sites/internationaliotcouncil/
138Disallow: /pega/customers-for-life
139Disallow: /celebritycruises/guide-to-the-globe
140Disallow: /sites/disneyinstitute/
141Disallow: /sites/concertocloud/
142Disallow: /sites/capitalonefutureedge/
143Disallow: /sites/mahindracomviva/
144Disallow: /sites/workday/
145Disallow: /sites/colorado/
146Disallow: /sites/charlesschwab/
147Disallow: /sites/chicagobooth/
148Disallow: /sites/morganstanley/
149Disallow: /sites/opentext/
150Disallow: /sites/icims/
151Disallow: /sites/thriventmutualfunds/
152Disallow: /sites/cbre/
153Disallow: /sites/universityofphoenix/#57934e047260
154Disallow: /sites/ppg/
155Disallow: /sites/americanfunds/
156Disallow: /sites/lilly/
157Disallow: /sites/norton/
158Disallow: /sites/qualityassurance/
159Disallow: /sites/qlik/
160Disallow: /whats-next/planes-trains-and-analytics
161Disallow: /top-50-wealth-managers
162Disallow: /toyota/fueled-by-skepticism
163Disallow: /toyota/coming-soon
164Disallow: /dell/cloud-future-of-medicine
165Disallow: /cit/renewable-energy
166Disallow: /retail
167Disallow: /mobile/cities-transformed
168Disallow: /cit/small-business-mergers-2
169Disallow: /cit/small-business-mergers
170Disallow: /sites/kpmg/
171Disallow: /dell/cities-transformed
172Disallow: /sites/neustar
173Disallow: /sites/ubs
174Disallow: /sites/northerntrust
175Disallow: /sites/infiniti
176Disallow: /sites/honeywell
177Disallow: /sites/charlesschwabadvisor
178Disallow: /sites/frisco
179Disallow: /sites/colehaan
180Disallow: /sites/gapincthiswayahead
181Disallow: /sites/discoverpersonalloans
182Disallow: /sites/delltechnologies
183Disallow: /colehaan/daring-to-step-forward
184Disallow: /johnhancock/
185Disallow: /johnhancock/in-a-lifetime-healthcare
186Disallow: /johnhancock/in-a-lifetime-work
187Disallow: /oppenheimerfunds/the-upside
188Disallow: /sites/hpeenterprisenxt
189Disallow: /sites/rackspace
190Disallow: /sites/themacallan
191Disallow: /themacallan/worlds-together
192Disallow: /sites/inteliotpartnership
193Disallow: /oppenheimerfunds/the-upside
194Disallow: /partners/
195Disallow: /johnhancock/in-a-lifetime-sustainability
196Disallow: /sites/domperignon
197Disallow: /sites/shinola
198Disallow: /sites/hpinc
199Disallow: /sites/hp
200Disallow: /sites/alfresco
201Disallow: /cbre/built-for-performance
202Disallow: /sites/mmedric
203Disallow: /sites/indiana
204Disallow: /sites/siemenshealthineers
205Disallow: /sites/rackspace
206Disallow: /rackspace/elevating-performance
207Disallow: /sites/suntrust
208Disallow: /sites/unicefusa
209Disallow: /sites/mitsubishiheavyindustries
210Disallow: /best-countries-for-business/list
211Disallow: /sites/paloaltonetworks
212Disallow: /sites/f5
213Disallow: /sites/samsungtv
214Disallow: /sites/qa
215Disallow: /sites/paycom
216Disallow: /sites/testbrandvoice
217Disallow: /business/blahblah
218Disallow: /sites/spdr
219Disallow: /sites/lincoln
220Disallow: /sites/abb
221Disallow: /sites/eycybersecurity
222Disallow: /sites/civicnation
223Disallow: /sites/bcg
224Disallow: /sites/vanguard
225Disallow: /sites/akamai
226Disallow: /sites/kyocera
227Disallow: /sites/socialimpact
228Disallow: /sites/emirates
229Disallow: /sites/hpincemea
230Disallow: /sites/ibmperformancecomputing
231Disallow: /sites/pfizer
232Disallow: /sites/ibmpowersystems
233Disallow: /sites/vanguardcharitable
234Disallow: /sites/suntrustprivatewealth
235Disallow: /sites/capitalone
236Disallow: /sites/juniper
237Disallow: /sites/wellsfargo
238Disallow: /sites/bnymellon
239Disallow: /sites/braintree
240Disallow: /sites/bnymellonwealthmanagement
241Disallow: /sites/epson
242Disallow: /sites/thomsonreuters
243Disallow: /sites/paysafe
244Disallow: /sites/josabank
245Disallow: /sites/hennessyparadisimperial
246Disallow: /sites/cartier
247Disallow: /sites/anabluewing
248Disallow: /sites/eidoo
249Disallow: /sites/cmstesting
250Disallow: /sites/adobepartnership
251Disallow: /sites/cornelljohnson
252Disallow: /sites/quantcast
253Disallow: /sites/japan
254Disallow: /sites/impactpartners
255Disallow: /sites/nvidia
256Disallow: /sites/personalcapital
257Disallow: /sites/microsoft365
258Disallow: /sites/providencedigitalinnovation
259Disallow: /sites/whittiertrust
260Disallow: /sites/kinetica
261Disallow: /sites/paulevans
262Disallow: /sites/ritz-carltonleadershipcenter
263Disallow: /sites/icoengine
264Disallow: /sites/tesaro
265Disallow: /sites/theglenlivet
266Disallow: /feature/sf-test
267Disallow: /sites/courtyardmarriott
268Disallow: /sites/nuveen
269Disallow: /sites/intelai
270Disallow: /sites/secret
271Disallow: /sites/hec
272Disallow: /retail/old
273Disallow: /sites/alliantgroup
274Disallow: /sites/intuitivesurgical
275Disallow: /sites/sunovion
276Disallow: /sites/ofx
277Disallow: /sites/firstrepublicbank
278Disallow: /sites/1850byfolgers
279Disallow: /sites/audi
280Disallow: /sites/nuveencef
281Disallow: /sites/1850brandcoffee
282Disallow: /sites/cisco
283Disallow: /sites/blackstarcapitalltd
284Disallow: /sites/usbankaltitudereserve
285Disallow: /sites/dassaultsystemes
286Disallow: /sites/urogen
287Disallow: /sites/smartsheet
288Disallow: /sites/iwc
289Disallow: /sites/datingcom
290Disallow: /sites/intuitquickbooks
291Disallow: /sites/oticon
292Disallow: /sites/servicenow
293Disallow: /sites/deloitteriskandfinancialadvisory
294Disallow: /sites/dbsprivatebank
295Disallow: /sites/americanexpress
296Disallow: /sites/tanotofoundation
297Disallow: /sites/citizensaccess
298Disallow: /sites/salesforceessentials
299Disallow: /sites/drexeluniversity
300Disallow: /sites/microsoftinnovation
301Disallow: /sites/mypos
302Disallow: /sites/harvardpilgrimhealthcare
303Disallow: /sites/appdynamics
304Disallow: /sites/ministryoftourismindia
305Disallow: /sites/orlando
306Disallow: /sites/extrahop
307Disallow: /sites/ascm
308Disallow: /sites/crowe
309Disallow: /sites/peerfit
310Disallow: /sites/theleukemialymphomasociety
311Disallow: /sites/hewlettpackardenterprise
312Disallow: /sites/montblanc
313Disallow: /sites/meero
314Disallow: /sites/braze
315Disallow: /sites/publicissapient
316Disallow: /sites/flexport
317
318User-agent: Amazonbot
319Disallow: /
320
321User-agent: anthropic-ai
322Disallow: /
323
324User-agent: Applebot-Extended
325Disallow: /
326
327User-agent: Bytespider
328Disallow: /
329
330User-agent: CCBot
331User-agent: CCbot
332Disallow: /
333
334User-agent: ClaudeBot
335Disallow: /
336
337User-agent: cohere-ai
338User-agent: Cohere-AI
339Disallow: /
340
341User-agent: Diffbot
342Disallow: /
343
344User-agent: FacebookBot
345Disallow: /
346
347User-agent: FriendlyCrawler
348Disallow: /
349
350User-agent: GPTBot
351Disallow: /
352
353User-agent: ImagesiftBot
354Disallow: /
355
356User-agent: meta-externalagent
357User-agent: Meta-ExternalAgent
358Disallow: /
359
360User-Agent: OAI-SearchBot
361Allow: /
362
363User-agent: omgili
364Disallow: /
365
366User-agent: omgilibot
367Disallow: /
368
369User-Agent: PerplexityBot
370Disallow: /
371
372User-agent: AhrefsBot
373User-agent: AhrefsSiteAudit
374Crawl-delay: 2
375
376User-Agent: Claude-User
377User-Agent: Claude-SearchBot
378Allow: /
379
380User-agent: AmazonAdBot
381Allow: /following/
382Allow: /search/
383Allow: /follow/
384Allow: /author/following/
385Allow: /blog/following/
386Allow: /header/channels/
387Allow: /find-more/
388Allow: /preview/
389Allow: /typeahead/
390Allow: /pepe/
391Allow: /media-manager/
392Allow: /coupons/visit/*
393Allow: /advisor/*zx1fa*
394Allow: /wui/
395
396User-agent: APSMeridianBot
397Disallow: /advisor
398Disallow: /betting/
399Disallow: /councils/
400Disallow: /financial-services/
401Disallow: /sites/_glenfiddich/
402Disallow: /sites/1850brandcoffee/
403Disallow: /sites/84-lumber/
404Disallow: /sites/aarp/
405Disallow: /sites/ab-inbev/
406Disallow: /sites/abb/
407Disallow: /sites/aberdeen/
408Disallow: /sites/accenture-india/
409Disallow: /sites/accenture/
410Disallow: /sites/acoustic/
411Disallow: /sites/acronis/
412Disallow: /sites/adobe/
413Disallow: /sites/adobepartnership/
414Disallow: /sites/adp/
415Disallow: /sites/adyen/
416Disallow: /sites/airbus/
417Disallow: /sites/akamai/
418Disallow: /sites/alfresco/
419Disallow: /sites/alibaba/
420Disallow: /sites/alight/
421Disallow: /sites/allclearid/
422Disallow: /sites/alliantgroup/
423Disallow: /sites/almanac/
424Disallow: /sites/alpine/
425Disallow: /sites/alteryx/
426Disallow: /sites/amazon-ads/
427Disallow: /sites/amazon-black-business-accelerator/
428Disallow: /sites/amazon-web-services-asean/
429Disallow: /sites/amazonwebservices/
430Disallow: /sites/amd/
431Disallow: /sites/american-cancer-society/
432Disallow: /sites/american-heart-association/
433Disallow: /sites/american-heart-associations-leaders-of-impact/
434Disallow: /sites/american-tire-distributors/
435Disallow: /sites/americanexpress/
436Disallow: /sites/amp-energy/
437Disallow: /sites/amtd-group/
438Disallow: /sites/anabluewing/
439Disallow: /sites/anaplan/
440Disallow: /sites/ant-international/
441Disallow: /sites/aptpay/
442Disallow: /sites/arizona-commerce-authority/
443Disallow: /sites/arizonacommerce/
444Disallow: /sites/arm/
445Disallow: /sites/ascm/
446Disallow: /sites/astrazeneca/
447Disallow: /sites/athenahealth/
448Disallow: /sites/athene/
449Disallow: /sites/atlassian/
450Disallow: /sites/att/
451Disallow: /sites/audemars-piguet/
452Disallow: /sites/audemarspiguet/
453Disallow: /sites/audi/
454Disallow: /sites/avanade/
455Disallow: /sites/avigilon/
456Disallow: /sites/aviram-family-foundation/
457Disallow: /sites/aws-and-pega/
458Disallow: /sites/aws-business-transformation/
459Disallow: /sites/aws-enterprise-transformation/
460Disallow: /sites/aws-transformation/
461Disallow: /sites/awscloudmigration/
462Disallow: /sites/awsinfrastructuresolutions/
463Disallow: /sites/awsmediaandentertainment/
464Disallow: /sites/awsstartups/
465Disallow: /sites/bacardi/
466Disallow: /sites/bankofsingapore/
467Disallow: /sites/baume_mercier/
468Disallow: /sites/bayer/
469Disallow: /sites/bbtsuntrustnowtruist/
470Disallow: /sites/bcbsassociation/
471Disallow: /sites/bcg/
472Disallow: /sites/benbridgejeweler/
473Disallow: /sites/beximco-group/
474Disallow: /sites/blackstarcapitalltd/
475Disallow: /sites/block-advisors/
476Disallow: /sites/bloodcancerunited/
477Disallow: /sites/bluejeans-by-verizon/
478Disallow: /sites/bluetti-power-inc/
479Disallow: /sites/bmc/
480Disallow: /sites/bmoharrisbank/
481Disallow: /sites/bnymellonwealthmanagement/
482Disallow: /sites/bold-by-veuve-clicquot/
483Disallow: /sites/boston-university-questrom-school-of-business/
484Disallow: /sites/bowlero/
485Disallow: /sites/braintree/
486Disallow: /sites/brandstorytelling/
487Disallow: /sites/braze/
488Disallow: /sites/bristol-myers-squibb/
489Disallow: /sites/bt/
490Disallow: /sites/bushmills/
491Disallow: /sites/businessaviation/
492Disallow: /sites/businessreporter/
493Disallow: /sites/cadence/
494Disallow: /sites/calm/
495Disallow: /sites/cambium-learning-group/
496Disallow: /sites/canon-inc/
497Disallow: /sites/canon/
498Disallow: /sites/canva/
499Disallow: /sites/capgemini/
500Disallow: /sites/capital-group-american-funds/
501Disallow: /sites/capitalone/
502Disallow: /sites/capitalonefutureedge/
503Disallow: /sites/capitalonespark/
504Disallow: /sites/carecredit/
505Disallow: /sites/cargill/
506Disallow: /sites/cartier/
507Disallow: /sites/castlight/
508Disallow: /sites/cathay-pacific/
509Disallow: /sites/cbre/
510Disallow: /sites/cdw/
511Disallow: /sites/celebritycruises/
512Disallow: /sites/celonis/
513Disallow: /sites/centurylink/
514Disallow: /sites/cerity-partners/
515Disallow: /sites/cftc/
516Disallow: /sites/chainalysis/
517Disallow: /sites/charlesschwabadvisor/
518Disallow: /sites/chase-ink-business-credit-cards/
519Disallow: /sites/chase-ink/
520Disallow: /sites/chase-sapphire/
521Disallow: /sites/chasehomelending/
522Disallow: /sites/checkmarx/
523Disallow: /sites/chicagobooth/
524Disallow: /sites/cigna/
525Disallow: /sites/cirocvs/
526Disallow: /sites/cisco-webex/
527Disallow: /sites/cisco/
528Disallow: /sites/ciscoappdynamics/
529Disallow: /sites/cit/
530Disallow: /sites/citi/
531Disallow: /sites/citizensaccess/
532Disallow: /sites/citrix/
533Disallow: /sites/city-of-hope/
534Disallow: /sites/civicnation/
535Disallow: /sites/cle-de-peau-beaute/
536Disallow: /sites/cloudera/
537Disallow: /sites/cloudflare/
538Disallow: /sites/cme-group/
539Disallow: /sites/coalition-for-faith-and-media/
540Disallow: /sites/coconut-software/
541Disallow: /sites/cognizant/
542Disallow: /sites/coldwellbanker/
543Disallow: /sites/colehaan/
544Disallow: /sites/colorado/
545Disallow: /sites/comarch/
546Disallow: /sites/comcastbusiness/
547Disallow: /sites/commure/
548Disallow: /sites/commvault/
549Disallow: /sites/company-of-good/
550Disallow: /sites/concertocloud/
551Disallow: /sites/connection/
552Disallow: /sites/connectly/
553Disallow: /sites/constantcontact/
554Disallow: /sites/cornelljohnson/
555Disallow: /sites/courtyard/
556Disallow: /sites/coxbusiness/
557Disallow: /sites/crowe/
558Disallow: /sites/ctia-wireless-foundation/
559Disallow: /sites/dassaultsystemes/
560Disallow: /sites/dataiku/
561Disallow: /sites/dbs-wealth-management/
562Disallow: /sites/dbsbank/
563Disallow: /sites/dbsprivatebank/
564Disallow: /sites/deep-instinct/
565Disallow: /sites/dell-technologies/
566Disallow: /sites/dell/
567Disallow: /sites/delltechnologies/
568Disallow: /sites/deloitte-microsoft-technology-services-practice/
569Disallow: /sites/deloitte/
570Disallow: /sites/deloitteriskandfinancialadvisory/
571Disallow: /sites/delta/
572Disallow: /sites/dhl-supply-chain/
573Disallow: /sites/diageo-rare--exceptional/
574Disallow: /sites/dimopoulos-injury-law/
575Disallow: /sites/diners-club-international/
576Disallow: /sites/discoverpersonalloans/
577Disallow: /sites/disneyinstitute/
578Disallow: /sites/dockers/
579Disallow: /sites/doctors-without-borders/
580Disallow: /sites/docusign/
581Disallow: /sites/dollar-general/
582Disallow: /sites/dominican-republic-ministry-of-tourism/
583Disallow: /sites/domperignon/
584Disallow: /sites/dpworld/
585Disallow: /sites/drexeluniversity/
586Disallow: /sites/dubai-tourism/
587Disallow: /sites/dxc-technology/
588Disallow: /sites/ebay/
589Disallow: /sites/ecolab/
590Disallow: /sites/ehealth/
591Disallow: /sites/eidoo/
592Disallow: /sites/elevance-health/
593Disallow: /sites/emc/
594Disallow: /sites/emirates/
595Disallow: /sites/empower/
596Disallow: /sites/energy-asia/
597Disallow: /sites/epos/
598Disallow: /sites/epson/
599Disallow: /sites/epsoncorporate/
600Disallow: /sites/eset/
601Disallow: /sites/etrade/
602Disallow: /sites/everbridge/
603Disallow: /sites/exl/
604Disallow: /sites/exness/
605Disallow: /sites/extrahop/
606Disallow: /sites/ey-global-government--infrastructure/
607Disallow: /sites/ey/
608Disallow: /sites/eycybersecurity/
609Disallow: /sites/f45training/
610Disallow: /sites/f5/
611Disallow: /sites/facebook/
612Disallow: /sites/familyofficehk/
613Disallow: /sites/fannie-mae/
614Disallow: /sites/fedex-express/
615Disallow: /sites/fedex/
616Disallow: /sites/ferrari-roma/
617Disallow: /sites/fidelity/
618Disallow: /sites/first-horizon-bank/
619Disallow: /sites/first-tee/
620Disallow: /sites/firstrepublicbank/
621Disallow: /sites/fis-global/
622Disallow: /sites/flexport/
623Disallow: /sites/forbeseq/
624Disallow: /sites/forcepoint/
625Disallow: /sites/fortinet/
626Disallow: /sites/freeenterprise/
627Disallow: /sites/frisco/
628Disallow: /sites/frog/
629Disallow: /sites/fujitsu-limited/
630Disallow: /sites/funnyfuzzy/
631Disallow: /sites/gainbridge/
632Disallow: /sites/gamuda-berhad/
633Disallow: /sites/ganz/
634Disallow: /sites/gapincthiswayahead/
635Disallow: /sites/gapinternational/
636Disallow: /sites/genentech/
637Disallow: /sites/genesyscontactcentercloud/
638Disallow: /sites/gerber-life-insurance/
639Disallow: /sites/gfk/
640Disallow: /sites/gigabyte/
641Disallow: /sites/gladly/
642Disallow: /sites/glenfiddich/
643Disallow: /sites/globalizationpartners/
644Disallow: /sites/goodrx/
645Disallow: /sites/google-ads/
646Disallow: /sites/google-maps/
647Disallow: /sites/googlecloud/
648Disallow: /sites/googlecloudaisolutions/
649Disallow: /sites/gradsoflife/
650Disallow: /sites/gran-coramino/
651Disallow: /sites/guild/
652Disallow: /sites/guocoland/
653Disallow: /sites/gwcic/
654Disallow: /sites/gyro/
655Disallow: /sites/hanwha-brandvoice/
656Disallow: /sites/harman/
657Disallow: /sites/harvard-division-of-continuing-education/
658Disallow: /sites/harvardpilgrimhealthcare/
659Disallow: /sites/havas-media-network/
660Disallow: /sites/hec/
661Disallow: /sites/henley--partners/
662Disallow: /sites/hennessyparadisimperial/
663Disallow: /sites/hewlettpackardenterprise/
664Disallow: /sites/hiscox/
665Disallow: /sites/hitachi-vantara-perspectives/
666Disallow: /sites/hitachienergy/
667Disallow: /sites/hola-prime/
668Disallow: /sites/hondajet/
669Disallow: /sites/honeywell/
670Disallow: /sites/hong-kong-talent-engage/
671Disallow: /sites/hongkong/
672Disallow: /sites/houlihanlawrence/
673Disallow: /sites/hp-inc-and-wwf/
674Disallow: /sites/hp/
675Disallow: /sites/hpeenterprisenxt/
676Disallow: /sites/hpincemea/
677Disallow: /sites/hsbc/
678Disallow: /sites/huawei/
679Disallow: /sites/hubinternational/
680Disallow: /sites/hyatt/
681Disallow: /sites/ibm/
682Disallow: /sites/ibmai/
683Disallow: /sites/ibmpowersystems/
684Disallow: /sites/ibmsecurity/
685Disallow: /sites/icims/
686Disallow: /sites/icoengine/
687Disallow: /sites/iedcagtechplus/
688Disallow: /sites/ifast-global-bank/
689Disallow: /sites/iintoo/
690Disallow: /sites/impactpartners/
691Disallow: /sites/indiana/
692Disallow: /sites/infiniti/
693Disallow: /sites/infocomm-media-development-authority/
694Disallow: /sites/informatica/
695Disallow: /sites/infosys/
696Disallow: /sites/insights-acxiom/
697Disallow: /sites/insights-adobe-microsoft/
698Disallow: /sites/insights-adobe/
699Disallow: /sites/insights-amazon-web-services-public-sector/
700Disallow: /sites/insights-amazon-web-services/
701Disallow: /sites/insights-american-express/
702Disallow: /sites/insights-arculus-by-composecure/
703Disallow: /sites/insights-asqexcellence/
704Disallow: /sites/insights-bcg/
705Disallow: /sites/insights-bmc/
706Disallow: /sites/insights-capitalone/
707Disallow: /sites/insights-cisco/
708Disallow: /sites/insights-cliffordchance/
709Disallow: /sites/insights-cloudera/
710Disallow: /sites/insights-delltechnologies/
711Disallow: /sites/insights-docusign/
712Disallow: /sites/insights-ecolab/
713Disallow: /sites/insights-engine-by-gen/
714Disallow: /sites/insights-ey/
715Disallow: /sites/insights-fortinet/
716Disallow: /sites/insights-glassbox/
717Disallow: /sites/insights-greenhousecom/
718Disallow: /sites/insights-hcl-software/
719Disallow: /sites/insights-hitachi/
720Disallow: /sites/insights-hsbc/
721Disallow: /sites/insights-ibmai/
722Disallow: /sites/insights-ibmresiliency/
723Disallow: /sites/insights-ibmservices/
724Disallow: /sites/insights-idex/
725Disallow: /sites/insights-intel/
726Disallow: /sites/insights-intelai/
727Disallow: /sites/insights-inteldatacenter/
728Disallow: /sites/insights-inteliot/
729Disallow: /sites/insights-jp-morgan-payments-commerce-solutions/
730Disallow: /sites/insights-jpmorgan/
731Disallow: /sites/insights-klgates/
732Disallow: /sites/insights-kpmg/
733Disallow: /sites/insights-kyriba/
734Disallow: /sites/insights-michiganeconomicdevelopmentcorporation/
735Disallow: /sites/insights-microsoft/
736Disallow: /sites/insights-microsoftai/
737Disallow: /sites/insights-mmgnet/
738Disallow: /sites/insights-northerntrust/
739Disallow: /sites/insights-northwestern-mutual/
740Disallow: /sites/insights-nuveen/
741Disallow: /sites/insights-ofx/
742Disallow: /sites/insights-pega/
743Disallow: /sites/insights-penske/
744Disallow: /sites/insights-pmi/
745Disallow: /sites/insights-roostify/
746Disallow: /sites/insights-sailpoint/
747Disallow: /sites/insights-sap-concur/
748Disallow: /sites/insights-sap/
749Disallow: /sites/insights-scrumalliance/
750Disallow: /sites/insights-sentry-insurance/
751Disallow: /sites/insights-silkroad/
752Disallow: /sites/insights-smartsheet/
753Disallow: /sites/insights-sovos/
754Disallow: /sites/insights-synchrony-bank/
755Disallow: /sites/insights-teamviewer/
756Disallow: /sites/insights-temenos/
757Disallow: /sites/insights-teradata/
758Disallow: /sites/insights-thomsonreuters/
759Disallow: /sites/insights-thoughtworks/
760Disallow: /sites/insights-treasuredata/
761Disallow: /sites/insights-vertiv/
762Disallow: /sites/insights-vmwaresecurity/
763Disallow: /sites/insights-whittiertrust/
764Disallow: /sites/insights-zoom/
765Disallow: /sites/intact-insurance-specialty-solutions/
766Disallow: /sites/integra-connect/
767Disallow: /sites/intel/
768Disallow: /sites/intelai/
769Disallow: /sites/inteliot/
770Disallow: /sites/inteliotpartnership/
771Disallow: /sites/intuit-expert/
772Disallow: /sites/intuitivesurgical/
773Disallow: /sites/intuitquickbooks/
774Disallow: /sites/invesco-qqq/
775Disallow: /sites/invest-puerto-rico/
776Disallow: /sites/ioi-properties/
777Disallow: /sites/iowa/
778Disallow: /sites/iris-powered-by-generali/
779Disallow: /sites/ishares/
780Disallow: /sites/itrustcapital/
781Disallow: /sites/iwc/
782Disallow: /sites/japan/
783Disallow: /sites/japanesevisionarywomen/
784Disallow: /sites/jbs-usa/
785Disallow: /sites/johnhancock/
786Disallow: /sites/johnson--johnson/
787Disallow: /sites/josabank/
788Disallow: /sites/jp-morgan-payments/
789Disallow: /sites/jpmorgan-chase-women-on-the-move/
790Disallow: /sites/jpmorganchase/
791Disallow: /sites/jumio/
792Disallow: /sites/jumpcloud/
793Disallow: /sites/juniper/
794Disallow: /sites/jurin-ai/
795Disallow: /sites/kaiserpermanente/
796Disallow: /sites/kantar/
797Disallow: /sites/kao/
798Disallow: /sites/kellogg/
799Disallow: /sites/kenvue/
800Disallow: /sites/keybanc-capital-markets/
801Disallow: /sites/kinaxis/
802Disallow: /sites/kinetica/
803Disallow: /sites/kpmg-international/
804Disallow: /sites/kpmg/
805Disallow: /sites/ksac/
806Disallow: /sites/kyndryl/
807Disallow: /sites/kyocera/
808Disallow: /sites/lasvegas/
809Disallow: /sites/lenovo-nvidia/
810Disallow: /sites/lexus/
811Disallow: /sites/lg/
812Disallow: /sites/lgt-private-banking-asia-pacific/
813Disallow: /sites/lincoln/
814Disallow: /sites/linkedin-apac/
815Disallow: /sites/logitech/
816Disallow: /sites/lopesan-costa-bavaro-resort-spa--casino/
817Disallow: /sites/loreal/
818Disallow: /sites/lowes-creator/
819Disallow: /sites/lumen/
820Disallow: /sites/magnit/
821Disallow: /sites/mahindra-group/
822Disallow: /sites/mahindracomviva/
823Disallow: /sites/malaysia-digital-economy-corporation/
824Disallow: /sites/manhattan-associates/
825Disallow: /sites/manulife-singapore/
826Disallow: /sites/manulife/
827Disallow: /sites/marcus-by-goldman-sachs/
828Disallow: /sites/mars/
829Disallow: /sites/maserati/
830Disallow: /sites/mastercard/
831Disallow: /sites/maxartechnologies/
832Disallow: /sites/mcdonalds/
833Disallow: /sites/medallia/
834Disallow: /sites/medidata/
835Disallow: /sites/meero/
836Disallow: /sites/meltwater/
837Disallow: /sites/mercedesbenz/
838Disallow: /sites/merrilllynch/
839Disallow: /sites/meta/
840Disallow: /sites/metronome/
841Disallow: /sites/micron-technology/
842Disallow: /sites/microsoft_/
843Disallow: /sites/microsoft365/
844Disallow: /sites/microsoftdynamics/
845Disallow: /sites/midfirst-bank/
846Disallow: /sites/migombank/
847Disallow: /sites/ministryoftourismindia/
848Disallow: /sites/mitsubishi-electric/
849Disallow: /sites/mitsubishiheavyindustries/
850Disallow: /sites/monogram/
851Disallow: /sites/montblanc/
852Disallow: /sites/moveworks/
853Disallow: /sites/mr-sancho-pancho/
854Disallow: /sites/mural/
855Disallow: /sites/mypos/
856Disallow: /sites/nacc-thailand/
857Disallow: /sites/netapp/
858Disallow: /sites/netjets/
859Disallow: /sites/neustar/
860Disallow: /sites/new-cies/
861Disallow: /sites/newyorklifeinvestments/
862Disallow: /sites/ni/
863Disallow: /sites/nice/
864Disallow: /sites/nokia-industry-40/
865Disallow: /sites/nomos-glashutte/
866Disallow: /sites/none/
867Disallow: /sites/northerntrust/
868Disallow: /sites/northwesternmutual/
869Disallow: /sites/novavax/
870Disallow: /sites/ntt-data/
871Disallow: /sites/nuskin/
872Disallow: /sites/nutanix/
873Disallow: /sites/nutrien/
874Disallow: /sites/nutro/
875Disallow: /sites/nuveen/
876Disallow: /sites/nvidia/
877Disallow: /sites/nx-group/
878Disallow: /sites/nyu-langone-health/
879Disallow: /sites/officedepotofficemax/
880Disallow: /sites/ofx/
881Disallow: /sites/olddominion/
882Disallow: /sites/omnia-partners/
883Disallow: /sites/opentext/
884Disallow: /sites/oppenheimerfunds/
885Disallow: /sites/oppo/
886Disallow: /sites/optum/
887Disallow: /sites/oracle/
888Disallow: /sites/oraclecloud/
889Disallow: /sites/orlando/
890Disallow: /sites/osim/
891Disallow: /sites/oticon/
892Disallow: /sites/otsuka/
893Disallow: /sites/outshift-by-cisco/
894Disallow: /sites/pace/
895Disallow: /sites/paloaltonetworks/
896Disallow: /sites/panasonic-energy/
897Disallow: /sites/parloa/
898Disallow: /sites/pastebywetransfer/
899Disallow: /sites/paulevans/
900Disallow: /sites/paycom/
901Disallow: /sites/paypal/
902Disallow: /sites/paysafe/
903Disallow: /sites/peerfit/
904Disallow: /sites/pega/
905Disallow: /sites/peopleconnect/
906Disallow: /sites/pepsico/
907Disallow: /sites/perennial-holdings/
908Disallow: /sites/personalcapital/
909Disallow: /sites/pete--gerrys/
910Disallow: /sites/peter-millar/
911Disallow: /sites/petronas/
912Disallow: /sites/pfizer/
913Disallow: /sites/pharmamar/
914Disallow: /sites/philanthropy-asia-alliance/
915Disallow: /sites/philip-morris-international/
916Disallow: /sites/philips/
917Disallow: /sites/pimco/
918Disallow: /sites/plusgrade/
919Disallow: /sites/pmi/
920Disallow: /sites/polestar/
921Disallow: /sites/post-university/
922Disallow: /sites/ppg/
923Disallow: /sites/premier/
924Disallow: /sites/proctergamble/
925Disallow: /sites/pros/
926Disallow: /sites/providencedigitalinnovation/
927Disallow: /sites/ptc/
928Disallow: /sites/publicissapient/
929Disallow: /sites/purdue-university/
930Disallow: /sites/puremichigan/
931Disallow: /sites/pwc-and-aws-switch-on-possibility/
932Disallow: /sites/pwc-cloud-and-digital-transformation/
933Disallow: /sites/pwc-consulting-llc/
934Disallow: /sites/pwc-sustainability/
935Disallow: /sites/pwc/
936Disallow: /sites/qatarfoundation/
937Disallow: /sites/qlik/
938Disallow: /sites/quad/
939Disallow: /sites/qualityassurance/
940Disallow: /sites/quantcast/
941Disallow: /sites/quest-software/
942Disallow: /sites/quest/
943Disallow: /sites/quid/
944Disallow: /sites/quility/
945Disallow: /sites/rackspace/
946Disallow: /sites/ramcommercial/
947Disallow: /sites/raymondjames/
948Disallow: /sites/rbcwealthmanagement/
949Disallow: /sites/realme/
950Disallow: /sites/recruit-holdings/
951Disallow: /sites/refinitiv/
952Disallow: /sites/regeneron/
953Disallow: /sites/relateiq/
954Disallow: /sites/reliaquest/
955Disallow: /sites/revolut-business/
956Disallow: /sites/richard-mille/
957Disallow: /sites/ricoh/
958Disallow: /sites/ricoheurope/
959Disallow: /sites/riskified/
960Disallow: /sites/ritz-carltonleadershipcenter/
961Disallow: /sites/riverbed/
962Disallow: /sites/robertocoin/
963Disallow: /sites/rocketmortgage/
964Disallow: /sites/rocky-mountaineer/
965Disallow: /sites/rokt/
966Disallow: /sites/ronesans-holding/
967Disallow: /sites/ryder/
968Disallow: /sites/sage/
969Disallow: /sites/sailpoint/
970Disallow: /sites/salesforce/
971Disallow: /sites/salesforceessentials/
972Disallow: /sites/samsung-galaxy-for-business/
973Disallow: /sites/samsung/
974Disallow: /sites/samsungbusiness/
975Disallow: /sites/samsungmobilecomputing/
976Disallow: /sites/samsungsds/
977Disallow: /sites/samsungtv/
978Disallow: /sites/sap/
979Disallow: /sites/sapconcur/
980Disallow: /sites/saskatchewan/
981Disallow: /sites/schmidt-futures/
982Disallow: /sites/schneiderelectric/
983Disallow: /sites/seagate/
984Disallow: /sites/securian-financial/
985Disallow: /sites/seoul/
986Disallow: /sites/servicenow/
987Disallow: /sites/shinola/
988Disallow: /sites/shoplazza/
989Disallow: /sites/siemens-smart-infrastructure/
990Disallow: /sites/siemenshealthineers/
991Disallow: /sites/sisense/
992Disallow: /sites/sk/
993Disallow: /sites/smartsheet/
994Disallow: /sites/snowflake/
995Disallow: /sites/socialimpact/
996Disallow: /sites/softserve/
997Disallow: /sites/sopra-banking-software/
998Disallow: /sites/spdr/
999Disallow: /sites/spectrum-business/
1000Disallow: /sites/splunk/
1001Disallow: /sites/sprinklr/
1002Disallow: /sites/sprintbusiness/
1003Disallow: /sites/square/
1004Disallow: /sites/stack-overflow-for-teams/
1005Disallow: /sites/stand-together/
1006Disallow: /sites/standard-chartered-bank/
1007Disallow: /sites/staples-business/
1008Disallow: /sites/starbucks-card-corporate-sales/
1009Disallow: /sites/starkey/
1010Disallow: /sites/state-of-maryland/
1011Disallow: /sites/statefarm/
1012Disallow: /sites/statoil/
1013Disallow: /sites/stewardshipasiacentre/
1014Disallow: /sites/stjude/
1015Disallow: /sites/sungardas/
1016Disallow: /sites/sunovion/
1017Disallow: /sites/suntrust/
1018Disallow: /sites/suntrustprivatewealth/
1019Disallow: /sites/surveymonkey/
1020Disallow: /sites/suse/
1021Disallow: /sites/svb/
1022Disallow: /sites/symantec/
1023Disallow: /sites/t-rowe-price/
1024Disallow: /sites/tableau/
1025Disallow: /sites/tableauapac/
1026Disallow: /sites/taboola/
1027Disallow: /sites/take-me-fishing/
1028Disallow: /sites/takeda/
1029Disallow: /sites/tanium/
1030Disallow: /sites/tanotofoundation/
1031Disallow: /sites/tanqueray-gin/
1032Disallow: /sites/taskus/
1033Disallow: /sites/tcs/
1034Disallow: /sites/td-bank/
1035Disallow: /sites/teladoc-health/
1036Disallow: /sites/ten-x/
1037Disallow: /sites/tencent-cloud/
1038Disallow: /sites/tequilaherradura/
1039Disallow: /sites/teradata/
1040Disallow: /sites/tesaro/
1041Disallow: /sites/textronaviation/
1042Disallow: /sites/the-allstate-foundation/
1043Disallow: /sites/the-ascott-limited/
1044Disallow: /sites/the-guild/
1045Disallow: /sites/the-macallan/
1046Disallow: /sites/the-uks-department-for-business-and-trade/
1047Disallow: /sites/thebalvenieandglenfiddich/
1048Disallow: /sites/theglenlivet/
1049Disallow: /sites/thehartford/
1050Disallow: /sites/theleukemialymphomasociety/
1051Disallow: /sites/themacallan/
1052Disallow: /sites/thepowermba/
1053Disallow: /sites/think-with-google/
1054Disallow: /sites/thomsonreuters/
1055Disallow: /sites/threatlocker/
1056Disallow: /sites/thrivent/
1057Disallow: /sites/tiaa/
1058Disallow: /sites/tiffany/
1059Disallow: /sites/tiktok/
1060Disallow: /sites/tm-global/
1061Disallow: /sites/tmfgroup/
1062Disallow: /sites/tmobile/
1063Disallow: /sites/todaysmilitary/
1064Disallow: /sites/totalwireless/
1065Disallow: /sites/toyota-prius/
1066Disallow: /sites/toyota/
1067Disallow: /sites/transamerica/
1068Disallow: /sites/trello/
1069Disallow: /sites/turkish-airlines/
1070Disallow: /sites/twilio/
1071Disallow: /sites/ubs/
1072Disallow: /sites/unicefusa/
1073Disallow: /sites/unicycive-therapeutics/
1074Disallow: /sites/unify/
1075Disallow: /sites/unilever-food-solutions/
1076Disallow: /sites/unique-properties/
1077Disallow: /sites/unisys/
1078Disallow: /sites/unit4/
1079Disallow: /sites/united-cards-from-chase/
1080Disallow: /sites/united-healthcare/
1081Disallow: /sites/united/
1082Disallow: /sites/universityofcalifornia/
1083Disallow: /sites/universityofphoenix/
1084Disallow: /sites/ups/
1085Disallow: /sites/urogen/
1086Disallow: /sites/us-bank-wealth-management/
1087Disallow: /sites/usaa/
1088Disallow: /sites/usbank/
1089Disallow: /sites/usbankaltitudereserve/
1090Disallow: /sites/vaneck/
1091Disallow: /sites/vanguard/
1092Disallow: /sites/vanguardcharitable/
1093Disallow: /sites/varicent/
1094Disallow: /sites/veeam/
1095Disallow: /sites/verisign/
1096Disallow: /sites/verizon-media/
1097Disallow: /sites/visa-infinite-asia-pacific/
1098Disallow: /sites/visa/
1099Disallow: /sites/visier/
1100Disallow: /sites/vision-by-protiviti/
1101Disallow: /sites/vmware/
1102Disallow: /sites/walmart-business/
1103Disallow: /sites/walmart-marketplace/
1104Disallow: /sites/webroot/
1105Disallow: /sites/wellington-management/
1106Disallow: /sites/wellsfargo/
1107Disallow: /sites/whittiertrust/
1108Disallow: /sites/windriver/
1109Disallow: /sites/wm/
1110Disallow: /sites/workday/
1111Disallow: /sites/workfront/
1112Disallow: /sites/xerox/
1113Disallow: /sites/xfinity/
1114Disallow: /sites/your-impact-with-unicef/
1115Disallow: /sites/zendesk/
1116Disallow: /sites/zenithwatches/
1117Disallow: /sites/zircongroup/
1118Disallow: /sites/zscaler/
1119Disallow: /sites/zurich/
1120Disallow: /sites/zycus/
1121Disallow: /uk/advisor/
1122Disallow: /wheels/
1123
1124
1125Sitemap: https://www.forbes.com/news_sitemap.xml
1126Sitemap: https://www.forbes.com/video_sitemap.xml
1127Sitemap: https://www.forbes.com/companies_sitemap.xml
1128Sitemap: https://www.forbes.com/health/sitemap.xml
1129Sitemap: https://www.forbes.com/connect/events-sitemap.xml
1130Sitemap: https://www.forbes.com/advisor/sitemap.xml
1131Sitemap: https://www.forbes.com/home-improvement/sitemap.xml
1132Sitemap: https://www.forbes.com/digital-assets/sitemap.xml
1133

What do the bots actually get?

This page reads robots.txt, which states what forbes.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on forbes.com Test a path in the robots.txt tester

History

Questions

Does forbes.com block GPTBot?

Yes. forbes.com’s robots.txt disallows GPTBot from the whole site (line 351: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does forbes.com block ClaudeBot?

Yes. forbes.com’s robots.txt disallows ClaudeBot from the whole site (line 335: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does forbes.com block Google-Extended?

No. forbes.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does forbes.com block PerplexityBot?

Yes. forbes.com’s robots.txt disallows PerplexityBot from the whole site (line 370: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find forbes.com?

Yes. forbes.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.