Who blocks AI · #1093 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does thestar.com block AI crawlers?
Yes. thestar.com’s robots.txt blocks all 19 AI crawlers we track, from GPTBot and ClaudeBot to PerplexityBot. It also blocks PetalBot among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- thestar.com/robots.txt · 815 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 56 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 50 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 53 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 113 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 104 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 101 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 71 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 35 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 23 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 38 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 14 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 8 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Blocked | line 155 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Blocked | line 47 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 11 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 17 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 116 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 110 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 134 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Blocked | line 107 · Disallow: / under User-agent: Meta-ExternalFetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View thestar.com/robots.txt · 815 lines
1User-agent: AI2Bot
2Disallow: /
3
4User-agent: Ai2Bot-Dolma
5Disallow: /
6
7User-agent: Applebot-Extended
8Disallow: /
9
10User-agent: Bytespider
11Disallow: /
12
13User-agent: CCBot
14Disallow: /
15
16User-agent: ChatGPT-User
17Disallow: /
18
19User-agent: Claude-Web
20Disallow: /
21
22User-agent: ClaudeBot
23Disallow: /
24
25User-agent: Diffbot
26Disallow: /
27
28User-agent: FacebookBot
29Disallow: /
30
31User-agent: FriendlyCrawler
32Disallow: /
33
34User-agent: GPTBot
35Disallow: /
36
37User-agent: Google-Extended
38Disallow: /
39
40User-agent: ICC-Crawler
41Disallow: /
42
43User-agent: ImagesiftBot
44Disallow: /
45
46User-agent: Meta-ExternalAgent
47Disallow: /
48
49User-agent: OAI-SearchBot
50Disallow: /
51
52User-agent: PerplexityBot
53Disallow: /
54
55User-agent: PetalBot
56Disallow: /
57
58User-agent: Scrapy
59Disallow: /
60
61User-agent: Timpibot
62Disallow: /
63
64User-agent: VelenPublicWebCrawler
65Disallow: /
66
67User-agent: Webzio-Extended
68Disallow: /
69
70User-agent: YouBot
71Disallow: /
72
73User-agent: anthropic-ai
74Disallow: /
75
76User-agent: cohere-ai
77Disallow: /
78
79User-agent: iaskspider/2.0
80Disallow: /
81
82User-agent: img2dataset
83Disallow: /
84
85User-agent: omgili
86Disallow: /
87
88User-agent: omgilibot
89Disallow: /
90
91User-agent: cohere-training-data-crawler
92Disallow: /
93
94User-agent: Kangaroo Bot
95Disallow: /
96
97User-agent: PanguBot
98Disallow: /
99
100User-agent: Amazonbot
101Disallow: /
102
103User-agent: DuckAssistBot
104Disallow: /
105
106User-agent: Meta-ExternalFetcher
107Disallow: /
108
109User-agent: Perplexity-User
110Disallow: /
111
112User-agent: Claude-SearchBot
113Disallow: /
114
115User-agent: Claude-User
116Disallow: /
117
118User-agent: Cotoyogi
119Disallow: /
120
121User-agent: Datenbank
122Disallow: /
123
124User-agent: Devin
125Disallow: /
126
127User-agent: Factset_spyderbot
128Disallow: /
129
130User-agent: Gemini-Deep-Research
131Disallow: /
132
133User-agent: MistralAI-User
134Disallow: /
135
136User-agent: netEstate
137Disallow: /
138
139User-agent: Imprint
140Disallow: /
141
142User-agent: SemrushBot-OCOB
143Disallow: /
144
145User-agent: AddSearchBot
146Disallow: /
147
148User-agent: bigsur.ai
149Disallow: /
150
151User-agent: CloudVertexBot
152Disallow: /
153
154User-agent: GoogleOther
155Disallow: /
156
157User-agent: LinerBot
158Disallow: /
159
160User-agent: meta-externalagent
161Disallow: /
162
163User-agent: QualifiedBot
164Disallow: /
165
166User-agent: Ai2Bot-DeepResearchEval
167Disallow: /
168
169User-agent: ChatGLM-Spider
170Disallow: /
171
172User-agent: DeepSeekBot
173Disallow: /
174
175User-agent: Google-NotebookLM
176Disallow: /
177
178User-agent: iAskBot
179Disallow: /
180
181User-agent: KunatoCrawler
182Disallow: /
183
184User-agent: meta-externalfetcher
185Disallow: /
186
187User-agent: meta-webindexer
188Disallow: /
189
190User-agent: webzio-extended
191Disallow: /
192
193User-agent: WRTNBot
194Disallow: /
195
196User-agent: ZanistaBot
197Disallow: /
198
199User-agent: AI2Bot-DeepResearchEval
200Disallow: /
201
202User-agent: Anomura
203Disallow: /
204
205User-agent: atlassian-bot
206Disallow: /
207
208User-agent: Bravebot
209Disallow: /
210
211User-agent: Channel3Bot
212Disallow: /
213
214User-agent: Cloudflare-AutoRAG
215Disallow: /
216
217User-agent: Crawl4AI
218Disallow: /
219
220User-agent: Google-CloudVertexBot
221Disallow: /
222
223User-agent: imageSpider
224Disallow: /
225
226User-agent: KlaviyoAIBot
227Disallow: /
228
229User-agent: laion-huggingface-processor
230Disallow: /
231
232User-agent: LCC
233Disallow: /
234
235User-agent: LinkupBot
236Disallow: /
237
238User-agent: PhindBot
239Disallow: /
240
241User-agent: Poggio-Citations
242Disallow: /
243
244User-agent: SBIntuitionsBot
245Disallow: /
246
247User-agent: TavilyBot
248Disallow: /
249
250User-agent: Amzn-SearchBot
251Disallow: /
252
253User-agent: Amzn-User
254Disallow: /
255
256User-agent: ApifyBot
257Disallow: /
258
259User-agent: ApifyWebsiteContentCrawler
260Disallow: /
261
262User-agent: Aranet-SearchBot
263Disallow: /
264
265User-agent: AzureAI-SearchBot
266Disallow: /
267
268User-agent: ExaBot
269Disallow: /
270
271User-agent: FirecrawlAgent
272Disallow: /
273
274User-agent: kagi-fetcher
275Disallow: /
276
277User-agent: TerraCotta
278Disallow: /
279
280User-agent: ScraperAPI
281Disallow: /
282
283User-agent: Scraper-API
284Disallow: /
285
286User-agent: ScrapingBee
287Disallow: /
288
289User-agent: ScrapingBeeBot
290Disallow: /
291
292User-agent: ZenRows
293Disallow: /
294
295User-agent: ZenRowsBot
296Disallow: /
297
298User-agent: ScrapeGraphAI
299Disallow: /
300
301User-agent: scrapegraph
302Disallow: /
303
304User-agent: Scrapfly
305Disallow: /
306
307User-agent: ScrapflyBot
308Disallow: /
309
310User-agent: ScrapflyClient
311Disallow: /
312
313User-agent: Zyte
314Disallow: /
315
316User-agent: ZyteBot
317Disallow: /
318
319User-agent: Scrapinghub
320Disallow: /
321
322User-agent: SerpApi
323Disallow: /
324
325User-agent: SerpApiBot
326Disallow: /
327
328User-agent: WebScraping.AI
329Disallow: /
330
331User-agent: WebScrapingAI
332Disallow: /
333
334User-agent: webscraping-ai
335Disallow: /
336
337User-agent: Scrapeless
338Disallow: /
339
340User-agent: ScrapelessBot
341Disallow: /
342
343User-agent: Scrapingdog
344Disallow: /
345
346User-agent: ScrapingdogBot
347Disallow: /
348
349User-agent: Scrapestack
350Disallow: /
351
352User-agent: ScrapestackBot
353Disallow: /
354
355User-agent: ScaleSERP
356Disallow: /
357
358User-agent: SerpWow
359Disallow: /
360
361User-agent: Rainforest-API
362Disallow: /
363
364User-agent: BlueCart-API
365Disallow: /
366
367User-agent: BrightData
368Disallow: /
369
370User-agent: Bright Data
371Disallow: /
372
373User-agent: BrightBot
374Disallow: /
375
376User-agent: Smartproxy
377Disallow: /
378
379User-agent: Oxylabs
380Disallow: /
381
382User-agent: OxylabsBot
383Disallow: /
384
385User-agent: Soax
386Disallow: /
387
388User-agent: SoaxBot
389Disallow: /
390
391User-agent: SOAX
392Disallow: /
393
394User-agent: Nimbleway
395Disallow: /
396
397User-agent: NimbleCrawler
398Disallow: /
399
400User-agent: NimblewayBot
401Disallow: /
402
403User-agent: Froxy
404Disallow: /
405
406User-agent: FroxyBot
407Disallow: /
408
409User-agent: BrowseAI
410Disallow: /
411
412User-agent: Browse-AI
413Disallow: /
414
415User-agent: BrowseAIBot
416Disallow: /
417
418User-agent: Octoparse
419Disallow: /
420
421User-agent: OctoparseBot
422Disallow: /
423
424User-agent: Import.io
425Disallow: /
426
427User-agent: ImportIO
428Disallow: /
429
430User-agent: importio
431Disallow: /
432
433User-agent: Kadoa
434Disallow: /
435
436User-agent: Kadoabot
437Disallow: /
438
439User-agent: KadoaBot
440Disallow: /
441
442User-agent: ScrapeHero
443Disallow: /
444
445User-agent: Scrape-Hero
446Disallow: /
447
448User-agent: ScrapeHeroBot
449Disallow: /
450
451User-agent: Grepsr
452Disallow: /
453
454User-agent: GrepsrBot
455Disallow: /
456
457User-agent: GrepsrCrawler
458Disallow: /
459
460User-agent: MrScraper
461Disallow: /
462
463User-agent: Mr-Scraper
464Disallow: /
465
466User-agent: MrScraperBot
467Disallow: /
468
469User-agent: PromptCloud
470Disallow: /
471
472User-agent: PromptCloudBot
473Disallow: /
474
475User-agent: CrawlNow
476Disallow: /
477
478User-agent: CrawlNowBot
479Disallow: /
480
481User-agent: ForageAI
482Disallow: /
483
484User-agent: Forage-AI
485Disallow: /
486
487User-agent: ForageBot
488Disallow: /
489
490User-agent: DumplingAI
491Disallow: /
492
493User-agent: Dumpling-AI
494Disallow: /
495
496User-agent: DumplingBot
497Disallow: /
498
499User-agent: spider-cloud
500Disallow: /
501
502User-agent: SpiderCloud
503Disallow: /
504
505User-agent: Hyperbrowser
506Disallow: /
507
508User-agent: HyperbrowserBot
509Disallow: /
510
511User-agent: HyperAgent
512Disallow: /
513
514User-agent: skrape.ai
515Disallow: /
516
517User-agent: SkrapeAI
518Disallow: /
519
520User-agent: SkrapeBot
521Disallow: /
522
523User-agent: CloudflareBrowserRenderingCrawler
524Disallow: /
525
526User-agent: Apify-HTTP-Client
527Disallow: /
528
529User-agent: amazon-kendra
530Disallow: /
531
532User-agent: amazon-QBusiness
533Disallow: /
534
535User-agent: bedrockbo
536Disallow: /
537
538User-agent: CohereBot
539Disallow: /
540
541User-agent: Doubaobot
542Disallow: /
543
544User-agent: ERNIEBot
545Disallow: /
546
547User-agent: GeistHaus-PageFetche
548Disallow: /
549
550User-agent: GoogleAgent-URLContext
551Disallow: /
552
553User-agent: Kimi-SearchBot
554Disallow: /
555
556User-agent: Kimi-User
557Disallow: /
558
559User-agent: KimiBot
560Disallow: /
561
562User-agent: KimiCrawler
563Disallow: /
564
565User-agent: micro-crawl
566Disallow: /
567
568User-agent: MistralAI-Index
569Disallow: /
570
571User-agent: MistralBot
572Disallow: /
573
574User-agent: MoonshotBot
575Disallow: /
576
577User-agent: MoonSpider
578Disallow: /
579
580User-agent: QwenBot
581Disallow: /
582
583User-agent: SalamandraVLM
584Disallow: /
585
586User-agent: Shap-User
587Disallow: /
588
589User-agent: TongyiBot
590Disallow: /
591
592User-agent: UseAI
593Disallow: /
594
595User-agent: xAI-SearchBot
596Disallow: /
597
598User-agent: YandexAdditional
599Disallow: /
600
601User-agent: YandexAdditionalBot
602Disallow: /
603
604User-agent: YiyanBot
605Disallow: /
606
607User-agent: ZenphonyBot
608Disallow: /
609
610User-agent: bedrockbot
611Disallow: /
612
613User-agent: GeistHaus-PageFetcher
614Disallow: /
615
616User-agent: AIWebIndex
617Disallow: /
618
619User-agent: AIWebIndex-Agent
620Disallow: /
621
622User-agent: CragCrawler
623Disallow: /
624
625User-agent: Crawlspace
626Disallow: /
627
628User-agent: Datenbank Crawler
629Disallow: /
630
631User-agent: ExaSearchBot
632Disallow: /
633
634User-agent: HenkBot
635Disallow: /
636
637User-agent: Mozilla-Tabstack
638Disallow: /
639
640User-agent: netEstate Imprint Crawler
641Disallow: /
642
643User-agent: Querit-SearchBot
644Disallow: /
645
646User-agent: QueritBot
647Disallow: /
648
649User-agent: Reflectionbot
650Disallow: /
651
652User-agent: ShapBot
653Disallow: /
654
655User-agent: TalarionSearchBot
656Disallow: /
657
658User-agent: Terra Cotta
659Disallow: /
660
661User-agent: Spider
662Disallow: /
663
664User-agent: AgentDataBot
665Disallow: /
666
667User-agent: BeansLLM-CorpusBot
668Disallow: /
669
670User-agent: BixelBot
671Disallow: /
672
673User-agent: DesearchBot
674Disallow: /
675
676User-agent: Hunyuan
677Disallow: /
678
679User-agent: ImageBot
680Disallow: /
681
682User-agent: Keenable-User
683Disallow: /
684
685User-agent: KeenableBot
686Disallow: /
687
688User-agent: recraft-image-dataset
689Disallow: /
690
691User-agent: ScryBot
692Disallow: /
693
694User-agent: SofyaBot
695Disallow: /
696
697User-agent: TalarionFrontierCrawler
698Disallow: /
699
700User-agent: YiBot
701Disallow: /
702
703User-agent: artsdata-crawler
704Disallow: /
705
706User-agent: ElevenlabsCrawler
707Disallow: /
708
709User-agent: Quartr Crawler
710Disallow: /
711
712User-agent: *
713Disallow: /tncms/tracking/
714Disallow: /_services/
715Disallow: /buycanadianbriefing/
716Disallow: /newsletters/books/
717Disallow: /newsletters/business/
718Disallow: /newsletters/evening-headlines/
719Disallow: /newsletters/long-reads/
720Disallow: /newsletters/news/
721Disallow: /newsletters/opinion/
722Disallow: /newsletters/real-estate/
723Disallow: /newsletters/sports/
724Disallow: /newsletters/storyalerts/
725Disallow: /newsletters/success/
726Disallow: /newsletters/unsubscribe/
727Disallow: /newspapers/header/
728Disallow: /registration/
729Disallow: /registration/success/
730Disallow: /search/
731Disallow: /static/
732Disallow: /subscribe/3forfree/
733Disallow: /subscribe/amplified/
734Disallow: /subscribe/canoo/
735Disallow: /subscribe/promo/
736Disallow: /subscribe/registrant-onboarding-25/
737Disallow: /subscribe/registrant-onboarding-40/
738Disallow: /subscribe/registrant-onboarding-50/
739Disallow: /subscribe/special/
740Disallow: /subscribe/trial-purchase-success/
741Disallow: /subscribe/tryfree/
742Disallow: /subscribe/wb-auto/
743Disallow: /t/
744Disallow: /tapp/
745Disallow: /test/
746Disallow: /test_aplus/
747Disallow: /tncms/counter/
748Disallow: /tncms/block/
749Disallow: /tncms/messaging/
750Disallow: /tncms/admin/
751Disallow: /tncms/openweb/*
752Disallow: /tncms/track/
753Disallow: /tncms/webservice/
754Disallow: /tncms/dmp/
755Disallow: /tncms/tax/
756Disallow: /tncms/csrf/
757Disallow: /tncms/gtm/
758Disallow: /tncms/search/
759Disallow: /tncms/auth/
760Disallow: /tncms/media/
761Disallow: /tncms/weather/
762Disallow: /tncms/openid2/
763Disallow: /content/tncms/
764Disallow: /marketplace/*action=srch
765Disallow: /tncms/disqus/
766Disallow: /tncms/user/
767Disallow: /users/admin/
768Disallow: /users/*/?
769Disallow: /tncms/calendar/
770Disallow: /content/tncms/assets/v3/form/
771Disallow: /content/tncms/form/
772Disallow: /tncms/asset/editorial/http*
773Disallow: /*%7b%7burl%7d%7d
774Disallow: /*%7B%7Burl%7D%7D
775Disallow: /*{{url}}
776Disallow: /content/tncms/live/
777Disallow: /assets/fonts/
778Disallow: /content/dam/
779Disallow: /*li_medium=
780Disallow: /*li_source=
781
782Sitemap: https://www.thestar.com/sitemap.xml
783Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml
784Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1979
785Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1980
786Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1984
787Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1985
788Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1992
789Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1993
790Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1995
791Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1998
792Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1999
793Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2002
794Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2003
795Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2005
796Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2007
797Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2008
798Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2009
799Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2010
800Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2011
801Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2012
802Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2013
803Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2014
804Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2015
805Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2016
806Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2017
807Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2018
808Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2019
809Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2020
810Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2021
811Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2022
812Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2023
813Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2024
814Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2025
815
What do the bots actually get?
This page reads robots.txt, which states what thestar.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on thestar.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does thestar.com block GPTBot?
Yes. thestar.com’s robots.txt disallows GPTBot from the whole site (line 35: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does thestar.com block ClaudeBot?
Yes. thestar.com’s robots.txt disallows ClaudeBot from the whole site (line 23: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does thestar.com block Google-Extended?
Yes. thestar.com’s robots.txt disallows Google-Extended from the whole site (line 38: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does thestar.com block PerplexityBot?
Yes. thestar.com’s robots.txt disallows PerplexityBot from the whole site (line 53: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find thestar.com?
Not through its own crawler. thestar.com’s robots.txt disallows OAI-SearchBot from the whole site (line 50: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.