Who blocks AI · #410 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does coursera.org block AI crawlers?
Yes. coursera.org’s robots.txt blocks 1 of the 19 AI crawlers we track, including Meta-ExternalAgent. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotPartly blockedOAI-SearchBotPartly blockedClaudeBotPartly blockedPerplexityBotPartly blockedGoogle-ExtendedPartly blockedCCBotPartly blocked- robots.txt
- www.coursera.org/robots.txt · 218 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Partly blocked | line 117 · Disallow: /maestro/api/ line 118 · Disallow: /api/ and 23 more |
PerplexityBot | Perplexity | Partly blocked | line 65 · Disallow: /lecture/ line 66 · Disallow: /maestro/api/ and 24 more |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Partly blocked | line 65 · Disallow: /lecture/ line 66 · Disallow: /maestro/api/ and 24 more |
ClaudeBot | Anthropic | Partly blocked | line 65 · Disallow: /lecture/ line 66 · Disallow: /maestro/api/ and 24 more |
Google-Extended | Partly blocked | line 65 · Disallow: /lecture/ line 66 · Disallow: /maestro/api/ and 24 more | |
CCBot | Common Crawl | Partly blocked | line 65 · Disallow: /lecture/ line 66 · Disallow: /maestro/api/ and 24 more |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 216 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Partly blocked | line 117 · Disallow: /maestro/api/ line 118 · Disallow: /api/ and 23 more |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Partly blocked | line 65 · Disallow: /lecture/ line 66 · Disallow: /maestro/api/ and 24 more |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.coursera.org/robots.txt · 218 lines
1User-agent: *
2Allow: /api/utilities/v1/imageproxy
3Allow: /llms.txt
4Disallow: /maestro/api/
5Disallow: /api/
6Disallow: /maestro/
7Disallow: /ui/
8Disallow: /signature/voucher/
9Disallow: /account/
10Disallow: /acclaimbadge/
11Disallow: /voucher/
12Disallow: /search
13Disallow: /ent-website/
14Disallow: /learn-perf/
15Disallow: /specializations-perf/
16Disallow: /professional-certificates-perf/
17Disallow: /learn-noperf/
18Disallow: /specializations-noperf/
19Disallow: /professional-certificates-noperf/
20Disallow: /business/xmlrpc.php
21Disallow: /business/wp-content/uploads/
22Disallow: /business/search
23Disallow: /business/teams/search
24Disallow: /organizations/
25Disallow: /programs/
26Disallow: /5JKLVK/
27
28# /courses pagination: block all, allow pages 1-5
29Disallow: /courses?page=
30Disallow: /courses?*&page=
31Allow: /courses?page=1$
32Allow: /courses?page=1&
33Allow: /courses?page=2$
34Allow: /courses?page=2&
35Allow: /courses?page=3$
36Allow: /courses?page=3&
37Allow: /courses?page=4$
38Allow: /courses?page=4&
39Allow: /courses?page=5$
40Allow: /courses?page=5&
41Allow: /courses?*&page=1$
42Allow: /courses?*&page=1&
43Allow: /courses?*&page=2$
44Allow: /courses?*&page=2&
45Allow: /courses?*&page=3$
46Allow: /courses?*&page=3&
47Allow: /courses?*&page=4$
48Allow: /courses?*&page=4&
49Allow: /courses?*&page=5$
50Allow: /courses?*&page=5&
51
52
53User-agent: AdsBot-Google
54Allow: /
55
56
57User-agent: CCBot
58User-agent: GPTBot
59User-agent: ClaudeBot
60User-agent: MistralAI-User
61User-agent: Google-Extended
62User-agent: PerplexityBot
63Allow: /api/utilities/v1/imageproxy
64Allow: /llms.txt
65Disallow: /lecture/
66Disallow: /maestro/api/
67Disallow: /api/
68Disallow: /maestro/
69Disallow: /ui/
70Disallow: /signature/voucher/
71Disallow: /account/
72Disallow: /acclaimbadge/
73Disallow: /voucher/
74Disallow: /search
75Disallow: /ent-website/
76Disallow: /learn-perf/
77Disallow: /specializations-perf/
78Disallow: /professional-certificates-perf/
79Disallow: /learn-noperf/
80Disallow: /specializations-noperf/
81Disallow: /professional-certificates-noperf/
82Disallow: /business/xmlrpc.php
83Disallow: /business/wp-content/uploads/
84Disallow: /business/search
85Disallow: /business/teams/search
86Disallow: /organizations/
87Disallow: /programs/
88Disallow: /5JKLVK/
89Disallow: /courses?page=
90Disallow: /courses?*&page=
91Allow: /courses?page=1$
92Allow: /courses?page=1&
93Allow: /courses?page=2$
94Allow: /courses?page=2&
95Allow: /courses?page=3$
96Allow: /courses?page=3&
97Allow: /courses?page=4$
98Allow: /courses?page=4&
99Allow: /courses?page=5$
100Allow: /courses?page=5&
101Allow: /courses?*&page=1$
102Allow: /courses?*&page=1&
103Allow: /courses?*&page=2$
104Allow: /courses?*&page=2&
105Allow: /courses?*&page=3$
106Allow: /courses?*&page=3&
107Allow: /courses?*&page=4$
108Allow: /courses?*&page=4&
109Allow: /courses?*&page=5$
110Allow: /courses?*&page=5&
111
112
113User-agent: OAI-SearchBot
114User-agent: ChatGPT-User
115Allow: /api/utilities/v1/imageproxy
116Allow: /llms.txt
117Disallow: /maestro/api/
118Disallow: /api/
119Disallow: /maestro/
120Disallow: /ui/
121Disallow: /signature/voucher/
122Disallow: /account/
123Disallow: /acclaimbadge/
124Disallow: /voucher/
125Disallow: /search
126Disallow: /ent-website/
127Disallow: /learn-perf/
128Disallow: /specializations-perf/
129Disallow: /professional-certificates-perf/
130Disallow: /learn-noperf/
131Disallow: /specializations-noperf/
132Disallow: /professional-certificates-noperf/
133Disallow: /business/xmlrpc.php
134Disallow: /business/wp-content/uploads/
135Disallow: /business/search
136Disallow: /business/teams/search
137Disallow: /organizations/
138Disallow: /programs/
139Disallow: /5JKLVK/
140Disallow: /courses?page=
141Disallow: /courses?*&page=
142Allow: /courses?page=1$
143Allow: /courses?page=1&
144Allow: /courses?page=2$
145Allow: /courses?page=2&
146Allow: /courses?page=3$
147Allow: /courses?page=3&
148Allow: /courses?page=4$
149Allow: /courses?page=4&
150Allow: /courses?page=5$
151Allow: /courses?page=5&
152Allow: /courses?*&page=1$
153Allow: /courses?*&page=1&
154Allow: /courses?*&page=2$
155Allow: /courses?*&page=2&
156Allow: /courses?*&page=3$
157Allow: /courses?*&page=3&
158Allow: /courses?*&page=4$
159Allow: /courses?*&page=4&
160Allow: /courses?*&page=5$
161Allow: /courses?*&page=5&
162
163
164User-agent: LinkedInBot
165User-agent: Twitterbot
166Allow: /account/accomplishments/
167Allow: /api/utilities/v1/imageproxy
168Disallow: /maestro/api/
169Disallow: /api/
170Disallow: /maestro/
171Disallow: /ui/
172Disallow: /signature/voucher/
173Disallow: /account/
174Disallow: /acclaimbadge/
175Disallow: /voucher/
176Disallow: /search
177Disallow: /ent-website/
178Disallow: /learn-perf/
179Disallow: /specializations-perf/
180Disallow: /professional-certificates-perf/
181Disallow: /learn-noperf/
182Disallow: /specializations-noperf/
183Disallow: /professional-certificates-noperf/
184Disallow: /business/xmlrpc.php
185Disallow: /business/wp-content/uploads/
186Disallow: /business/search
187Disallow: /business/teams/search
188Disallow: /organizations/
189Disallow: /programs/
190Disallow: /5JKLVK/
191Disallow: /courses?page=
192Disallow: /courses?*&page=
193Allow: /courses?page=1$
194Allow: /courses?page=1&
195Allow: /courses?page=2$
196Allow: /courses?page=2&
197Allow: /courses?page=3$
198Allow: /courses?page=3&
199Allow: /courses?page=4$
200Allow: /courses?page=4&
201Allow: /courses?page=5$
202Allow: /courses?page=5&
203Allow: /courses?*&page=1$
204Allow: /courses?*&page=1&
205Allow: /courses?*&page=2$
206Allow: /courses?*&page=2&
207Allow: /courses?*&page=3$
208Allow: /courses?*&page=3&
209Allow: /courses?*&page=4$
210Allow: /courses?*&page=4&
211Allow: /courses?*&page=5$
212Allow: /courses?*&page=5&
213
214
215User-agent: meta-externalagent
216Disallow: /
217
218Sitemap: https://www.coursera.org/sitemap.xml
What do the bots actually get?
This page reads robots.txt, which states what coursera.org asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on coursera.org Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does coursera.org block GPTBot?
Partly. coursera.org’s robots.txt keeps GPTBot out of some paths, such as /lecture/, and lets it crawl the rest. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does coursera.org block ClaudeBot?
Partly. coursera.org’s robots.txt keeps ClaudeBot out of some paths, such as /lecture/, and lets it crawl the rest. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does coursera.org block Google-Extended?
Partly. coursera.org’s robots.txt keeps Google-Extended out of some paths, such as /lecture/, and lets it crawl the rest. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does coursera.org block PerplexityBot?
Partly. coursera.org’s robots.txt keeps PerplexityBot out of some paths, such as /lecture/, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find coursera.org?
Partly. coursera.org’s robots.txt keeps OAI-SearchBot out of some paths, such as /maestro/api/, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.