canaicrawl Get early access

Who blocks AI · #652 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does ucl.ac.uk block AI crawlers?

We couldn’t check ucl.ac.uk: our request for www.ucl.ac.uk/robots.txt was refused (403), most likely by bot protection, so we can’t say what it tells crawlers.

GPTBotUnknownOAI-SearchBotUnknownClaudeBotUnknownPerplexityBotUnknownGoogle-ExtendedUnknownCCBotUnknown
robots.txt
www.ucl.ac.uk/robots.txt · our request for www.ucl.ac.uk/robots.txt was refused (403), most likely by bot protection, so we can’t say what it tells crawlers
llms.txt
not checked
Checked
9 November 2026, as canaicrawl. We check again every week.

What do the bots actually get?

This page reads robots.txt, which states what ucl.ac.uk asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on ucl.ac.uk Test a path in the robots.txt tester

History