Who blocks AI · #914 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does sfu.ca block AI crawlers?
No. sfu.ca’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.sfu.ca/robots.txt · 102 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.sfu.ca/robots.txt · 102 lines
1User-agent: Twitterbot
2Disallow: *
3Allow: /content/dam/sfu/sfunews
4
5User-agent: *
6Disallow: /phonebook
7Disallow: /services.htm
8Disallow: /stats3.html
9Disallow: /stat.html
10Disallow: /stat2.html
11Disallow: /stats
12Disallow: /swipe
13Disallow: /convocation/Invitation
14Disallow: /convocation/invitation
15Disallow: /convocation/information/invitation/
16Disallow: /convocation/information/Invitation/
17Disallow: /designgroup
18Disallow: /vpresearch/sfuhomepg/
19Disallow: /vpresearch/docs/
20Disallow: /policies/_pvw5d0992/
21Disallow: /~cmnsweb
22Disallow: /~extrel
23Disallow: /advancement/services_and_systems/
24Disallow: /advancement/files/System_Services/
25Disallow: /advancement/files/
26Disallow: /sfunews/Archives/By_Category/
27Disallow: /sfunews/Archives/By_Year/
28Disallow: /sfunews/learning/index.shtml
29Disallow: /sfunews/community/index.shtml
30Disallow: /sfunews/research/index.shtml
31Disallow: /sfunews/sports/index.shtml
32Disallow: /sfunews/people/index.shtml
33Disallow: /dean-gradstudies/administration/staff/
34Disallow: /dean-gradstudies/administration/staff.html
35Disallow: /pamr/issues_experts/featured_experts_archive/index.html
36Disallow: /pamr/issues_experts/issues_experts_archive/index.html
37Disallow: /pamr/issues_experts/issues_experts_archive/index.html
38Disallow: /pamr/media_releases/media_releases_archive/index.html
39Disallow: /pamr/media_releases/index.html
40Disallow: /pamr/files
41Disallow: /archive-university-communications
42Disallow: /fins/
43Disallow: /continuing-studies/certificates.htm
44Disallow: /ihr/
45Disallow: /~educwww
46Disallow: /~educfp
47Disallow: /~educpdp
48Disallow: /~educugp
49Disallow: /~educgp
50Disallow: /~educint
51Disallow: /~educfal
52Disallow: /~fosimon
53Disallow: /learn/temp
54Disallow: /content/dam/alumni/2012elections
55Disallow: /content/dam
56Allow: /content/dam/math
57Allow: /content/dam/sfu/retirees/Newsletters
58Allow: /content/dam/sfu/julita-clf4
59Allow: /content/dam/sfu/senate/senate-documents
60Disallow: /students-resources
61Disallow: /rs/
62Disallow: /~jtoal/academic
63Disallow: /~itsvs/
64Disallow: /~mstanger/canvas
65Disallow: /~surreywb
66Disallow: /sakai
67Disallow: /students/academic-calendar-archive
68Disallow: /students/calendar/2013
69Disallow: /students/calendar/2014
70Disallow: /students/calendar/2015
71Disallow: /students/calendar/2016
72Disallow: /students/calendar/2017
73Disallow: /students/calendar/2018
74Disallow: /students/calendar/2019
75Disallow: /students/calendar/2020
76Disallow: /students/calendar/2021
77Disallow: /students/calendar/2022
78Disallow: /students/calendar/2023
79Disallow: /students/calendar/2024
80Disallow: /students/calendar/2025
81Disallow: /students/calendar/2026/spring
82Disallow: /sustainabledevelopment/search
83Disallow: /devsprogram
84Disallow: /cscd
85Disallow: /content/clf
86Disallow: /itservices/sfuconnect
87Disallow: /sustainable-development-goals/sustainable-cities
88Disallow: /idcard/getting-card/new-replacement-apply.html
89Disallow: /content/sfu/deployment-test-clf4
90Disallow: /content/sfu/deployment-test-clf3
91Disallow: /content/sfu/deployment-test-clf2
92Disallow: /deployment-test-clf4
93Disallow: /deployment-test-clf3
94Disallow: /deployment-test-clf2
95Disallow: /content/sfu/sfu-forms
96Disallow: /sfu-forms
97Disallow: /siat/student_projects
98Disallow: /~sfufa
99Disallow: /academic-discussion
100Disallow: /content/sfu/students/calendar/2025
101Disallow: /students/calendar/2026/summer
102
What do the bots actually get?
This page reads robots.txt, which states what sfu.ca asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on sfu.ca Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does sfu.ca block GPTBot?
No. sfu.ca’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does sfu.ca block ClaudeBot?
No. sfu.ca’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does sfu.ca block Google-Extended?
No. sfu.ca’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does sfu.ca block PerplexityBot?
No. sfu.ca’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find sfu.ca?
Yes. sfu.ca’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.