Who blocks AI · #751 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does theglobeandmail.com block AI crawlers?
Yes. theglobeandmail.com’s robots.txt blocks 18 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 15 more. It also blocks PetalBot and Baiduspider among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.theglobeandmail.com/robots.txt · 572 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 432 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Blocked | line 99 · Disallow: / under User-agent: BaiduSpider |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 381 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 429 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 144 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 195 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 54 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 567 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 246 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 153 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 228 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 126 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Blocked | line 237 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Blocked | line 333 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 120 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 138 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 147 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 426 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 348 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Blocked | line 339 · Disallow: / under User-agent: Meta-ExternalFetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 36 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.theglobeandmail.com/robots.txt · 572 lines
1User-agent: AmazonAdBot
2Allow: /
3
4User-agent: Googlebot-News
5Disallow: /feeds/
6Disallow: /incoming/
7Disallow: /test/
8Disallow: /partners/
9Disallow: /search/
10Disallow: /business/adv/appointmentnotices/search/
11Disallow: /gift/
12
13User-agent: AdsBot-Google
14Disallow: /feeds/
15Disallow: /incoming/
16Disallow: /test/
17Disallow: /search/
18Disallow: /business/adv/appointmentnotices/search/
19Disallow: /gift/
20
21User-agent: *
22Disallow: /feeds/
23Disallow: /incoming/
24Disallow: /test/
25Disallow: /search/
26Disallow: /business/adv/appointmentnotices/search/
27Disallow: /marketing-containers/
28Disallow: /coupons/
29Disallow: /files/advertising/
30Disallow: /gift/
31
32User-agent: AddSearchBot
33Disallow: /
34
35User-agent: AhrefsBot
36Disallow: /
37
38User-agent: AI2Bot
39Disallow: /
40
41User-agent: AI2Bot-DeepResearchEval
42Disallow: /
43
44User-agent: AI2Bot-Dolma
45Disallow: /
46
47User-agent: aiHitBot
48Disallow: /
49
50User-agent: AliyunSecBot
51Disallow: /
52
53User-agent: Amazonbot
54Disallow: /
55
56User-agent: AmazonBuyForMe
57Disallow: /
58
59User-agent: amazon-kendra
60Disallow: /
61
62User-agent: Amzn-SearchBot
63Disallow: /
64
65User-agent: Amzn-User
66Disallow: /
67
68User-agent: Andibot
69Disallow: /
70
71User-agent: Anthropic-ai
72Disallow: /
73
74User-agent: Anomura
75Disallow: /
76
77User-agent: archive.org_bot
78Disallow: /
79
80User-agent: Archive-It
81Disallow: /
82
83User-agent: atlassian-bot
84Disallow: /
85
86User-agent: AudigentAdBot
87Disallow: /
88
89User-agent: Awario
90Disallow: /
91
92User-agent: AwarioRssBot
93Disallow: /
94
95User-agent: AwarioSmartBot
96Disallow: /
97
98User-agent: BaiduSpider
99Disallow: /
100
101User-agent: bedrockbot
102Disallow: /
103
104User-agent: bigsur.ai
105Disallow: /
106
107User-agent: BLEXBot
108Disallow: /
109
110User-agent: Brandwatch
111Disallow: /
112
113User-agent: Bravebot
114Disallow: /
115
116User-agent: Brightbot 1.0
117Disallow: /
118
119User-agent: Bytespider
120Disallow: /
121
122User-agent: carbon-umbrella-bot/1.0
123Disallow: /
124
125User-agent: CCBot
126Disallow: /
127
128User-agent: Channel3Bot
129Disallow: /
130
131User-agent: ChatGLM-Spider
132Disallow: /
133
134User-agent: ChatGPT Agent
135Disallow: /
136
137User-agent: ChatGPT-User
138Disallow: /
139
140User-agent: Cision
141Disallow: /
142
143User-agent: Claude-SearchBot
144Disallow: /
145
146User-agent: Claude-User
147Disallow: /
148
149User-agent: Claude-Web
150Disallow: /
151
152User-agent: ClaudeBot
153Disallow: /
154
155User-agent: Cloudflare-AutoRAG
156Disallow: /
157
158User-agent: CloudVertexBot
159Disallow: /
160
161User-agent: cohere-ai
162Disallow: /
163
164User-agent: cohere-training-data-crawler
165Disallow: /
166
167User-agent: Cotoyogi
168Disallow: /
169
170User-agent: Crawl4AI
171Disallow: /
172
173User-agent: Crawlspace
174Disallow: /
175
176User-agent: DataForSeoBot
177Disallow: /
178
179User-agent: Datenbank Crawler
180Disallow: /
181
182User-agent: DeepSeekBot
183Disallow: /
184
185User-agent: Deepseek
186Disallow: /
187
188User-agent: Devin
189Disallow: /
190
191User-agent: Diffbot
192Disallow: /
193
194User-agent: DuckAssistBot
195Disallow: /
196
197User-agent: EchoboxBot
198Disallow: /
199
200User-agent: ErisBot
201Disallow: /
202
203User-agent: Factset_spyderbot
204Disallow: /
205
206User-agent: fiperbot/0.1
207Disallow: /
208
209User-agent: FirecrawlAgent
210Disallow: /
211
212User-agent: FriendlyCrawler
213Disallow: /
214
215User-agent: Gemini-Deep-Research
216Disallow: /
217
218User-agent: Gnowit
219Disallow: /
220
221User-agent: GnowitNewsbot
222Disallow: /
223
224User-agent: Google-CloudVertexBot
225Disallow: /
226
227User-agent: Google-Extended
228Disallow: /
229
230User-agent: Google-NotebookLM
231Disallow: /
232
233User-agent: GoogleAgent-Mariner
234Disallow: /
235
236User-agent: GoogleOther
237Disallow: /
238
239User-agent: GoogleOther-Image
240Disallow: /
241
242User-agent: GoogleOther-Video
243Disallow: /
244
245User-agent: GPTBot
246Disallow: /
247
248User-agent: Grok
249Disallow: /
250
251User-agent: Heritrix
252Disallow: /
253
254User-agent: ia_archiver
255Disallow: /
256
257User-agent: iAskBot
258Disallow: /
259
260User-agent: iaskspider/2.0
261Disallow: /
262
263User-agent: IbouBot
264Disallow: /
265
266User-agent: ICC-Crawler
267Disallow: /
268
269User-agent: imageSpider
270Disallow: /
271
272User-agent: ImagesiftBot
273Disallow: /
274
275User-agent: img2dataset
276Disallow: /
277
278User-agent: Imprint (netEstate Imprint Crawler)
279Disallow: /
280
281User-agent: Info2424.info
282Disallow: /
283
284User-agent: ISSCyberRiskCrawler
285Disallow: /
286
287User-agent: Kangaroo Bot
288Disallow: /
289
290User-agent: KlaviyoAIBot
291Disallow: /
292
293User-agent: KunatoCrawler
294Disallow: /
295
296User-agent: LAIONDownloader
297Disallow: /
298
299User-agent: laion-huggingface-processor
300Disallow: /
301
302User-agent: LCC
303Disallow: /
304
305User-agent: LinerBot
306Disallow: /
307
308User-agent: Linguee Bot
309Disallow: /
310
311User-agent: linkfluence
312Disallow: /
313
314User-agent: LinkupBot
315Disallow: /
316
317User-agent: LinkupBot/1.0
318Disallow: /
319
320User-agent: magpie-crawler
321Disallow: /
322
323User-agent: Manus-User
324Disallow: /
325
326User-agent: mediatoolkit.com
327Disallow: /
328
329User-agent: Meltwater
330Disallow: /
331
332User-agent: Meta-ExternalAgent
333Disallow: /
334
335User-agent: meta-externalagent
336Disallow: /
337
338User-agent: Meta-ExternalFetcher
339Disallow: /
340
341User-agent: meta-externalfetcher
342Disallow: /
343
344User-agent: meta-webindexer
345Disallow: /
346
347User-agent: MistralAI-User
348Disallow: /
349
350User-agent: MistralAI-User/1.0
351Disallow: /
352
353User-agent: Muckrack
354Disallow: /
355
356User-agent: MyCentralAIScraperBot
357Disallow: /
358
359User-agent: netEstate
360Disallow: /
361
362User-agent: netEstate Imprint Crawler
363Disallow: /
364
365User-agent: news-api.org
366Disallow: /
367
368User-agent: NewsNow
369Disallow: /
370
371User-agent: news-please
372Disallow: /
373
374User-agent: NotebookLM
375Disallow: /
376
377User-agent: NovaAct
378Disallow: /
379
380User-agent: OAI-SearchBot
381Disallow: /
382
383User-agent: Observer/1.0 (obsrvr.net)
384Disallow: /
385
386User-agent: omgili
387Disallow: /
388
389User-agent: omgilibot
390Disallow: /
391
392User-agent: openai
393Disallow: /
394
395User-agent: OpenAI
396Disallow: /
397
398User-agent: Operator
399Disallow: /
400
401User-agent: Opoint
402Disallow: /
403
404User-agent: OTMedia
405Disallow: /
406
407User-agent: PanguBot
408Disallow: /
409
410User-agent: Panscient
411Disallow: /
412
413User-agent: panscient.com
414Disallow: /
415
416User-agent: peer39_crawler
417Disallow: /
418
419User-agent: peer39_crawler/1.0
420Disallow: /
421
422User-agent: Perplexity
423Disallow: /
424
425User-agent: Perplexity-User
426Disallow: /
427
428User-agent: PerplexityBot
429Disallow: /
430
431User-agent: PetalBot
432Disallow: /
433
434User-agent: PhindBot
435Disallow: /
436
437User-agent: Poggio-Citations
438Disallow: /
439
440User-agent: Poseidon Research Crawler
441Disallow: /
442
443User-agent: PressEngineBot
444Disallow: /
445
446User-agent: QualifiedBot
447Disallow: /
448
449User-agent: QuillBot
450Disallow: /
451
452User-agent: quillbot.com
453Disallow: /
454
455User-agent: Quora-Bot
456Disallow: /
457
458User-agent: Qwantbot
459Disallow: /
460
461User-agent: SBIntuitionsBot
462Disallow: /
463
464User-agent: scoopit
465Disallow: /
466
467User-agent: scpitspi-rs
468Disallow: /
469
470User-agent: Scrapy
471Disallow: /
472
473User-agent: SeekrBot
474Disallow: /
475
476User-agent: semantic-visions.com
477Disallow: /
478
479User-agent: SemrushBot-SWA
480Disallow: /
481
482User-agent: SentiBot
483Disallow: /
484
485User-agent: SeznamHomepageCrawler
486Disallow: /
487
488User-agent: ShapBot
489Disallow: /
490
491User-agent: Sidetrade indexer bot
492Disallow: /
493
494User-agent: Sindup
495Disallow: /
496
497User-agent: Slurp
498Disallow: /
499
500User-agent: Spider
501Disallow: /
502
503User-agent: Squidbot
504Disallow: /
505
506User-agent: Squidbot/1.0
507Disallow: /
508
509User-agent: TaraGroup Intelligent Bot
510Disallow: /
511
512User-agent: TavilyBot
513Disallow: /
514
515User-agent: TerraCotta
516Disallow: /
517
518User-agent: Thinkbot
519Disallow: /
520
521User-agent: Timpibot
522Disallow: /
523
524User-agent: Timpibot/0.9
525Disallow: /
526
527User-agent: Tor
528Disallow: /
529
530User-agent: trendictionbot
531Disallow: /
532
533User-agent: TurnitinBot
534Disallow: /
535
536User-agent: TwinAgent
537Disallow: /
538
539User-agent: VelenPublicWebCrawler
540Disallow: /
541
542User-agent: ViennaTinyBot
543Disallow: /
544
545User-agent: WARDBot
546Disallow: /
547
548User-agent: webzio
549Disallow: /
550
551User-agent: Webzio-Extended
552Disallow: /
553
554User-agent: wpbot
555Disallow: /
556
557User-agent: WRTNBot
558Disallow: /
559
560User-agent: YaK
561Disallow: /
562
563User-agent: YandexAdditionalBot
564Disallow: /
565
566User-agent: YouBot
567Disallow: /
568
569User-agent: ZanistaBot
570Disallow: /
571
572Sitemap: https://www.theglobeandmail.com/sitemap.xml?outputType=xml
What do the bots actually get?
This page reads robots.txt, which states what theglobeandmail.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on theglobeandmail.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does theglobeandmail.com block GPTBot?
Yes. theglobeandmail.com’s robots.txt disallows GPTBot from the whole site (line 246: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does theglobeandmail.com block ClaudeBot?
Yes. theglobeandmail.com’s robots.txt disallows ClaudeBot from the whole site (line 153: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does theglobeandmail.com block Google-Extended?
Yes. theglobeandmail.com’s robots.txt disallows Google-Extended from the whole site (line 228: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does theglobeandmail.com block PerplexityBot?
Yes. theglobeandmail.com’s robots.txt disallows PerplexityBot from the whole site (line 429: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find theglobeandmail.com?
Not through its own crawler. theglobeandmail.com’s robots.txt disallows OAI-SearchBot from the whole site (line 381: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.