Who blocks AI · #246 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does sedo.com block AI crawlers?
We couldn’t check sedo.com: our request for sedo.com/robots.txt was refused (503), most likely by bot protection, so we can’t say what it tells crawlers.
GPTBotUnknownOAI-SearchBotUnknownClaudeBotUnknownPerplexityBotUnknownGoogle-ExtendedUnknownCCBotUnknown- robots.txt
- sedo.com/robots.txt · our request for sedo.com/robots.txt was refused (503), most likely by bot protection, so we can’t say what it tells crawlers
- llms.txt
- not checked
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
What do the bots actually get?
This page reads robots.txt, which states what sedo.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on sedo.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
- 2 November 2026: robots.txt couldn’t be read.