canaicrawl Get early access

Who blocks AI · #247 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does cambridge.org block AI crawlers?

Yes. cambridge.org’s robots.txt blocks 1 of the 19 AI crawlers we track, including ChatGPT-User. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed
robots.txt
www.cambridge.org/robots.txt · 119 lines
llms.txt
none
Checked
9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGoogleAllowedno rule matches the home page
BingbotMicrosoftAllowedno rule matches the home page
DuckDuckBotDuckDuckGoAllowedno rule matches the home page
ApplebotAppleAllowedno rule matches the home page
PetalBotHuaweiAllowedno rule matches the home page
YandexBotYandexAllowedno rule matches the home page
BaiduspiderBaiduAllowedno rule matches the home page

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIAllowedno rule matches the home page
PerplexityBotPerplexityAllowedno rule matches the home page
Claude-SearchBotAnthropicAllowedno rule matches the home page
DuckAssistBotDuckDuckGoAllowedno rule matches the home page
AmazonbotAmazonAllowedno rule matches the home page
YouBotYou.comAllowedno rule matches the home page

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIAllowedno rule matches the home page
ClaudeBotAnthropicAllowedno rule matches the home page
Google-ExtendedGoogleAllowedno rule matches the home page
CCBotCommon CrawlAllowedno rule matches the home page
Applebot-ExtendedAppleAllowedno rule matches the home page
GoogleOtherGoogleAllowedno rule matches the home page
Meta-ExternalAgentMetaAllowedno rule matches the home page
BytespiderByteDanceAllowedno rule matches the home page

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIBlockedline 5 · Disallow: / under User-agent: ChatGPT-User
Claude-UserAnthropicAllowedno rule matches the home page
Perplexity-UserPerplexityAllowedno rule matches the home page
MistralAI-UserMistralAllowedno rule matches the home page
Meta-ExternalFetcherMetaAllowedno rule matches the home page

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsAllowedno rule matches the home page
SemrushBotSemrushAllowedno rule matches the home page
MJ12botMajesticAllowedno rule matches the home page
DotBotMozAllowedno rule matches the home page
View www.cambridge.org/robots.txt · 119 lines
1User-agent: Yandex
2Disallow: /
3
4User-agent: ChatGPT-User
5Disallow: /
6
7User-agent: ChatGPT
8Disallow: /
9
10User-agent: *
11Allow: /concrete/*.css
12Allow: /concrete/*.js
13Allow: /concrete/*.jpg
14Allow: /concrete/*.svg
15Allow: /concrete/*.png
16Allow: /concrete/*.gif
17Allow: /packages/*.css
18Allow: /packages/*.js
19Allow: /packages/*.jpg
20Allow: /packages/*.svg
21Allow: /packages/*.png
22Allow: /packages/*.gif
23Allow: /blocks/*.css
24Allow: /blocks/*.js
25Allow: /blocks/*.jpg
26Allow: /blocks/*.svg
27Allow: /blocks/*.png
28Allow: /blocks/*.gif
29Allow: /tools/packages/cambridge_themes/getSocialImage
30Allow: /tools/packages/cambridge_themes/miniCart*
31Disallow: /aca/authorinformation/
32Disallow: /blocks
33Disallow: /concrete
34Disallow: /config
35Disallow: /controllers
36Disallow: /css
37Disallow: /elements
38Disallow: /helpers
39Disallow: /jobs
40Disallow: /js
41Disallow: /languages
42Disallow: /libraries
43Disallow: /mail
44Disallow: /models
45Disallow: /packages
46Disallow: /single_pages
47Disallow: /themes
48Disallow: /tools
49Disallow: /updates
50Disallow: /branding
51Disallow: /wm-ecommerce-web/academic/addtocart
52Disallow: /wm-ecommerce-web/english/addtocart
53Disallow: /wm-ecommerce-web/academic/cart
54Disallow: /wm-ecommerce-web/english/cart
55Disallow: /wm-ecommerce-web/education/cart
56Disallow: */c5catalogue/
57Disallow: /resources
58Disallow: /core/dashboard
59Disallow: /core/my-core
60Disallow: /core/services/aop-*
61Allow: /core/services/aop-cambridge-core/content/view/
62Allow: /core/services/aop-cambridge-core/altmetric
63Allow: /core/services/aop-file-manager/file
64Allow: /aus/interchange
65Allow: /files/*.jpg
66Allow: /files/*.jpeg
67Allow: /files/*.png
68Allow: /files/*.gif
69Disallow: /files
70Disallow: /series/
71Disallow: /titles/
72Disallow: /uk
73Allow: /uk/bookshop
74Allow: /uk/education
75Allow: /uk/jobs
76Allow: /uk/linguistics
77Allow: /uk/chemistry
78Allow: /uk/stm
79Allow: /uk/resources
80Disallow: /engage/*/submit-process
81Disallow: /engage/*/author-dashboard
82Disallow: /engage/*/administration-dashboard
83Disallow: /engage/_nuxt
84Disallow: */storefront/
85Disallow: /engage/demo
86Disallow: /engage/*/retract-process
87Disallow: /engage/*/test
88Disallow: /highereducation/_nuxt
89Disallow: /highereducation/my-account
90Disallow: /package.json
91Disallow: /api/ecommerce/bag
92Disallow: /node
93Disallow: /*?*site_view=
94Disallow: /*?*nestedTab=
95Disallow: /*?*ccm_paging_p_b=
96Disallow: /*?*layout=
97Disallow: /*?*f%5B*%5D=
98Disallow: /*?*options[]=
99Disallow: /*?*options%5B%5D=
100Disallow: /*?f[*
101Disallow: /*&f[*
102Disallow: /*element_properties.*
103Disallow: /*form_defaults.*
104Disallow: /*property_group_options.*
105Disallow: /*style_builder.*
106Disallow: /checkout/
107Disallow: /signin
108Disallow: /register
109
110
111
112User-agent: Twitterbot
113Disallow: *
114Disallow: /download_file/
115Disallow: /manifests
116Disallow: /infographic
117
118
119Sitemap: https://www.cambridge.org/highereducation/sitemap_index.xml

What do the bots actually get?

This page reads robots.txt, which states what cambridge.org asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on cambridge.org Test a path in the robots.txt tester

History

Questions

Does cambridge.org block GPTBot?

No. cambridge.org’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does cambridge.org block ClaudeBot?

No. cambridge.org’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does cambridge.org block Google-Extended?

No. cambridge.org’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does cambridge.org block PerplexityBot?

No. cambridge.org’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find cambridge.org?

Yes. cambridge.org’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.