Who blocks AI · #1206 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does cuny.edu block AI crawlers?

Yes. cuny.edu’s robots.txt blocks all 19 AI crawlers we track, from GPTBot and ClaudeBot to PerplexityBot. It also blocks PetalBot, YandexBot and Baiduspider among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: cuny.edu/robots.txt · 45 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Partly blocked	line 12 · `Disallow: /global-components/` line 13 · `Disallow: /d-i/` and 17 more
`Bingbot`	Microsoft	Partly blocked	line 12 · `Disallow: /global-components/` line 13 · `Disallow: /d-i/` and 17 more
`DuckDuckBot`	DuckDuckGo	Partly blocked	line 12 · `Disallow: /global-components/` line 13 · `Disallow: /d-i/` and 17 more
`Applebot`	Apple	Partly blocked	line 12 · `Disallow: /global-components/` line 13 · `Disallow: /d-i/` and 17 more
`PetalBot`	Huawei	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`YandexBot`	Yandex	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Baiduspider`	Baidu	Blocked	line 43 · `Disallow: /` under `User-agent: *`

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`PerplexityBot`	Perplexity	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Claude-SearchBot`	Anthropic	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`DuckAssistBot`	DuckDuckGo	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Amazonbot`	Amazon	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`YouBot`	You.com	Blocked	line 43 · `Disallow: /` under `User-agent: *`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`ClaudeBot`	Anthropic	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Google-Extended`	Google	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`CCBot`	Common Crawl	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Applebot-Extended`	Apple	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`GoogleOther`	Google	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Meta-ExternalAgent`	Meta	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Bytespider`	ByteDance	Blocked	line 43 · `Disallow: /` under `User-agent: *`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Claude-User`	Anthropic	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Perplexity-User`	Perplexity	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`MistralAI-User`	Mistral	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Meta-ExternalFetcher`	Meta	Blocked	line 43 · `Disallow: /` under `User-agent: *`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`SemrushBot`	Semrush	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`MJ12bot`	Majestic	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`DotBot`	Moz	Blocked	line 43 · `Disallow: /` under `User-agent: *`

View cuny.edu/robots.txt · 45 lines

1User-Agent: googlebot

2User-Agent: bingbot

3User-Agent: msnbot

4User-Agent: Twitterbot

5User-Agent: Slurp

6User-Agent: LinkedInBot

7User-Agent: DuckDuckBot

8User-agent: SiteimproveBot-Crawler

9Crawl-delay: 30

10sitemap: https://www.cuny.edu/sitemap.xml

11sitemap: https://policy.cuny.edu/sitemap.xml

12Disallow: /global-components/

13Disallow: /d-i/

14Disallow: /cgi-bin/

15Disallow: /search/

16Disallow: /wp-admin/

17Disallow: /wp-includes/

18Disallow: /wp-content/plugins/

19Disallow: /wp-content/themes/

20Disallow: /alumni-test/

21Disallow: /role-template/

22Disallow: /home-preview/

23Disallow: /about/administration/offices/sa/

24Disallow: /homepage/digital-displays/

25Disallow: /employment/search-jobs/

26Disallow: /old-version/

27Disallow: /policyimport/

28Disallow: /global-assets/

29Disallow: /*includes/

30# Added June 15, 2024

31Disallow: /*?utm=*

33User-agent: SemanticScholarBot

34Disallow: /

36User-agent: facebookexternalhit

37Disallow:

39User-agent: Go-http-client

40Disallow: /

42User-agent: *

43Disallow: /

What do the bots actually get?

This page reads robots.txt, which states what cuny.edu asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on cuny.edu Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does cuny.edu block GPTBot?

Yes. cuny.edu’s robots.txt disallows GPTBot from the whole site (line 43: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does cuny.edu block ClaudeBot?

Yes. cuny.edu’s robots.txt disallows ClaudeBot from the whole site (line 43: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does cuny.edu block Google-Extended?

Yes. cuny.edu’s robots.txt disallows Google-Extended from the whole site (line 43: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does cuny.edu block PerplexityBot?

Yes. cuny.edu’s robots.txt disallows PerplexityBot from the whole site (line 43: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find cuny.edu?

Not through its own crawler. cuny.edu’s robots.txt disallows OAI-SearchBot from the whole site (line 43: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.