Who blocks AI · #1305 in the Majestic Million

robots.txt · checked 16 Nov 2026

Does business-standard.com block AI crawlers?

We couldn’t check business-standard.com: our request for www.business-standard.com/robots.txt was refused (403), most likely by bot protection, so we can’t say what it tells crawlers.

GPTBotUnknownOAI-SearchBotUnknownClaudeBotUnknownPerplexityBotUnknownGoogle-ExtendedUnknownCCBotUnknown
robots.txt
www.business-standard.com/robots.txt · our request for www.business-standard.com/robots.txt was refused (403), most likely by bot protection, so we can’t say what it tells crawlers
llms.txt
not checked
Checked
16 November 2026, as canaicrawl. We check again every week.

What do the bots actually get?

This page reads robots.txt, which states what business-standard.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on business-standard.com Test a path in the robots.txt tester

History