Who blocks AI · #752 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does calameo.com block AI crawlers?
Yes. calameo.com’s robots.txt blocks 1 of the 19 AI crawlers we track, including GoogleOther. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.calameo.com/robots.txt · 182 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Partly blocked | line 93 · Disallow: /account$ line 94 · Disallow: /account/ and 24 more |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Blocked | line 8 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.calameo.com/robots.txt · 182 lines
1User-agent: Linguee
2Disallow: /
3
4User-agent: MJ12bot
5Disallow: /
6
7User-agent: GoogleOther
8Disallow: /
9
10User-agent: facebookexternalhit
11
12Disallow: /account$
13Disallow: /account/
14Disallow: /download$
15Disallow: /download/
16Disallow: /errors$
17Disallow: /errors/
18Disallow: /healthcheck$
19Disallow: /healthcheck/
20Disallow: /oauth$
21Disallow: /oauth/
22Disallow: /orders$
23Disallow: /orders/
24Disallow: /profile$
25Disallow: /profile/
26Disallow: /search$
27Disallow: /search/
28Disallow: /services$
29Disallow: /services/
30Disallow: /upload$
31Disallow: /upload/
32Disallow: /zendesk$
33Disallow: /zendesk/
34Disallow: /pagegangster$
35Disallow: /pagegangster
36
37User-agent: Twitterbot
38
39Disallow: /account$
40Disallow: /account/
41Disallow: /download$
42Disallow: /download/
43Disallow: /errors$
44Disallow: /errors/
45Disallow: /healthcheck$
46Disallow: /healthcheck/
47Disallow: /oauth$
48Disallow: /oauth/
49Disallow: /orders$
50Disallow: /orders/
51Disallow: /profile$
52Disallow: /profile/
53Disallow: /search$
54Disallow: /search/
55Disallow: /services$
56Disallow: /services/
57Disallow: /upload$
58Disallow: /upload/
59Disallow: /zendesk$
60Disallow: /zendesk/
61Disallow: /pagegangster$
62Disallow: /pagegangster
63
64User-agent: LinkedInBot
65
66Disallow: /account$
67Disallow: /account/
68Disallow: /download$
69Disallow: /download/
70Disallow: /errors$
71Disallow: /errors/
72Disallow: /healthcheck$
73Disallow: /healthcheck/
74Disallow: /oauth$
75Disallow: /oauth/
76Disallow: /orders$
77Disallow: /orders/
78Disallow: /profile$
79Disallow: /profile/
80Disallow: /search$
81Disallow: /search/
82Disallow: /services$
83Disallow: /services/
84Disallow: /upload$
85Disallow: /upload/
86Disallow: /zendesk$
87Disallow: /zendesk/
88Disallow: /pagegangster$
89Disallow: /pagegangster
90
91User-agent: Amazonbot
92
93Disallow: /account$
94Disallow: /account/
95Disallow: /books/social$
96Disallow: /books/social/
97Disallow: /download$
98Disallow: /download/
99Disallow: /errors$
100Disallow: /errors/
101Disallow: /healthcheck$
102Disallow: /healthcheck/
103Disallow: /oauth$
104Disallow: /oauth/
105Disallow: /orders$
106Disallow: /orders/
107Disallow: /profile$
108Disallow: /profile/
109Disallow: /search$
110Disallow: /search/
111Disallow: /services$
112Disallow: /services/
113Disallow: /upload$
114Disallow: /upload/
115Disallow: /zendesk$
116Disallow: /zendesk/
117Disallow: /pagegangster$
118Disallow: /pagegangster
119
120User-agent: Mediapartners-Google
121
122Disallow: /account$
123Disallow: /account/
124Disallow: /books/social$
125Disallow: /books/social/
126Disallow: /download$
127Disallow: /download/
128Disallow: /errors$
129Disallow: /errors/
130Disallow: /healthcheck$
131Disallow: /healthcheck/
132Disallow: /oauth$
133Disallow: /oauth/
134Disallow: /orders$
135Disallow: /orders/
136Disallow: /profile$
137Disallow: /profile/
138Disallow: /search$
139Disallow: /search/
140Disallow: /services$
141Disallow: /services/
142Disallow: /upload$
143Disallow: /upload/
144Disallow: /zendesk$
145Disallow: /zendesk/
146Disallow: /pagegangster$
147Disallow: /pagegangster
148
149User-agent: *
150Crawl-delay: 1
151
152Disallow: /account$
153Disallow: /account/
154Disallow: /books/social$
155Disallow: /books/social/
156Disallow: /download$
157Disallow: /download/
158Disallow: /read$
159Disallow: /read/
160Disallow: /errors$
161Disallow: /errors/
162Disallow: /healthcheck$
163Disallow: /healthcheck/
164Disallow: /oauth$
165Disallow: /oauth/
166Disallow: /orders$
167Disallow: /orders/
168Disallow: /profile$
169Disallow: /profile/
170Disallow: /search$
171Disallow: /search/
172Disallow: /services$
173Disallow: /services/
174Disallow: /upload$
175Disallow: /upload/
176Disallow: /zendesk$
177Disallow: /zendesk/
178Disallow: /pagegangster$
179Disallow: /pagegangster
180
181Sitemap: https://www.calameo.com/sitemaps/incremental
182Sitemap: https://www.calameo.com/sitemaps/index
What do the bots actually get?
This page reads robots.txt, which states what calameo.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on calameo.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does calameo.com block GPTBot?
No. calameo.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does calameo.com block ClaudeBot?
No. calameo.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does calameo.com block Google-Extended?
No. calameo.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does calameo.com block PerplexityBot?
No. calameo.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find calameo.com?
Yes. calameo.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.