Who blocks AI · #85 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does sourceforge.net block AI crawlers?
No. sourceforge.net’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- sourceforge.net/robots.txt · 198 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View sourceforge.net/robots.txt · 198 lines
1User-agent: Mediapartners-Google
2Allow: /
3
4User-agent: AmazonAdBot
5Allow: /
6
7User-agent: *
8Allow: /p/*/ci/master/tree/$
9Allow: /p/*/ci/main/tree/$
10Allow: /p/*/ci/default/tree/$
11Allow: /p/*/HEAD/tree/$
12Allow: /p/*/HEAD/tree/trunk/$
13Allow: /adobe/*/HEAD/tree/trunk/$
14Allow: /software/vendors/optimize-software-for-ai-search/
15Disallow: /p/*/code/
16Disallow: /p/*/git/
17Disallow: /p/*/svn/
18Disallow: /p/*/hg/
19Disallow: /p/*/search
20Disallow: /p/*/code-0/
21Disallow: /p/*/tree/
22Disallow: /adobe/*/tree/
23Disallow: /p/*/search_feed/
24Disallow: /p/*/search_help/
25Disallow: /p/*/wiki/*/edit
26Disallow: /p/*/wiki/browse_tags/
27Disallow: /p/*/discussion/create_topic/
28Disallow: /p/*/discussion/stats
29Disallow: /*/ci/
30Disallow: /*/commit_browser
31Disallow: /*/commit_browser_data
32Disallow: /*?barediff=
33Disallow: /*?diff=
34Disallow: /*/tarball
35Disallow: /code-snapshots/
36Disallow: /*/bugs/new/
37Disallow: /*/support-requests/new/
38Disallow: /*/patches/new/
39Disallow: /*/feature-requests/new/
40Disallow: /*/tickets/new/
41Disallow: /*/edit/
42Disallow: /*/attachment/
43Disallow: /*/_discuss/
44Disallow: /projects/*/files/*/stats/timeline
45Disallow: /projects/*/files/stats/timeline
46Disallow: /projects/*/files/*/stats/map
47Disallow: /projects/*/files/stats/map
48Disallow: /projects/*/files/*/stats/os
49Disallow: /projects/*/files/stats/os
50Disallow: /projects/*/files/stats/json
51Disallow: /projects/*/files/*/stats/json
52Disallow: /projects/*/files/latest/download
53Disallow: /projects/*/latest.json
54Disallow: /projects/*/report_inappropriate
55Disallow: /projects/*/reviews/new
56Disallow: /projects/*/rss
57Disallow: /projects/*/lists/*/unsubscribe
58Disallow: /projects/*/moderate_review
59Disallow: /projects/*/rate_review
60Disallow: /projects/*/best_release.html
61Disallow: /projects/*/postdownload
62Disallow: /projects/*/get_updates
63Disallow: /app/*/files/
64Disallow: /app/*/latest.json
65Disallow: /app/*/report_inappropriate
66Disallow: /app/*/reviews/
67Disallow: /app/*/rss
68Disallow: /app/*/moderate_review
69Disallow: /app/*/rate_review
70Disallow: /app/*/best_release.html
71Disallow: /app/*/postdownload
72Disallow: /app/*/get_updates
73Disallow: /*_escaped_fragment_
74Disallow: /auth/
75Disallow: /auth/do_login
76Disallow: /software/link?
77Disallow: /software/link$
78Disallow: /software/visit?
79Disallow: /software/visit$
80Disallow: /software/transfer_site
81Disallow: /software/compare/*-vs-*-vs-*-vs-
82Disallow: /software/compare/*/add-software
83Disallow: /software/product/*/reviews/new
84Disallow: /software/product/*/reviews/add
85Disallow: /software/product/*/claim
86Disallow: /software/vendors/new
87Disallow: /software/vendors/inquire
88Disallow: /software/vendors/ai-llm-optimization
89Disallow: /software/*?q=
90Disallow: /software/*&q=
91Disallow: /software/*?categories=
92Disallow: /software/*&categories=
93Disallow: /software/*?company_sizes=
94Disallow: /software/*&company_sizes=
95Disallow: /software/*?deployment=
96Disallow: /software/*&deployment=
97Disallow: /software/*?free_options=
98Disallow: /software/*&free_options=
99Disallow: /software/*?integrates_with=
100Disallow: /software/*&integrates_with=
101Disallow: /software/*?org_types=
102Disallow: /software/*&org_types=
103Disallow: /software/*?api=
104Disallow: /software/*&api=
105Disallow: /software/*?regions=
106Disallow: /software/*®ions=
107Disallow: /software/*?support=
108Disallow: /software/*&support=
109Disallow: /software/*?training=
110Disallow: /software/*&training=
111Disallow: /software/*?feature_
112Disallow: /software/*&feature_
113Disallow: /rest/
114Allow: /rest/*/activity
115Disallow: /user/registration
116Disallow: /user/verified
117Disallow: */bin_counts
118Disallow: */milestone_counts
119Disallow: */stats_data
120Disallow: *//users$
121Disallow: */users$
122Disallow: /u/*/link/
123Disallow: /p/*/link/
124Disallow: /p/*/donate
125Disallow: */feed.rss
126Disallow: */feed.atom
127Disallow: /p/*/news/feed
128Disallow: /p/*/news/*/history
129Disallow: /arris/*/feed$
130Disallow: */activity/feed
131Disallow: */activity/pjax
132Disallow: /p/*/admin/
133Disallow: /directory/release_feed
134Disallow: *?css-reload=
135Disallow: *&css-reload=
136Disallow: /settings/mirror_choices
137Disallow: /directory/os:windows/?q=
138Disallow: /directory/tp
139Disallow: /software/tp
140Disallow: /sd-jobs
141Disallow: /about/mirror-agreement
142Disallow: /t7
143Disallow: /p/*/wiki/*/history
144Disallow: /p/*/*/milestone/
145Disallow: /p/*/activity/
146Disallow: /u/*/wiki/*
147Disallow: /u/*/activity/
148Allow: /u/*/profile/$
149Disallow: /u/*/*/
150Disallow: /*/wp-login.php
151Disallow: /blog/tag/
152Disallow: /blog/category/
153Disallow: /articles/tag/
154Disallow: /articles/category/open-source/
155Disallow: /articles/category/developer-tools/
156Disallow: /articles/category/artificial-intelligence/
157Disallow: /articles/category/database/
158Disallow: /articles/category/mobile/
159Disallow: /articles/category/payroll/
160Disallow: /articles/category/telephony/
161Disallow: /articles/category/networking/
162Disallow: /articles/category/cloud/
163Disallow: /articles/category/general/
164Disallow: /articles/category/software/
165Disallow: /p/allura/chat/
166Disallow: /p/allura/tickets/
167Disallow: /about/maxnav
168Disallow: /hi
169Allow: /llms.txt
170
171# /directory param rules: permit ?q= and ?page= on their own, but no other params
172# longest match takes precedence. listed in length order
173Disallow: /directory/*?
174Allow: /directory/*?q=
175Disallow: /directory/*?q=*&
176Allow: /directory/*?page=
177Disallow: /directory/*?page=*&
178
179# mailman flags
180Disallow: *?style=flat
181Disallow: *&style=flat
182Disallow: *?style=threaded
183Disallow: *&style=threaded
184Disallow: *?viewmonth=
185Disallow: *&viewmonth=
186Disallow: *?viewmont=
187Disallow: *&viewmont=
188Disallow: *?viewday=
189Disallow: *&viewday=
190
191Sitemap: https://sourceforge.net/sitemap.xml
192Sitemap: https://sourceforge.net/allura_sitemap/sitemap.xml
193Sitemap: https://sourceforge.net/directory_sitemap.xml
194Sitemap: https://sourceforge.net/software_sitemap.xml
195Sitemap: https://sourceforge.net/blog/sitemap_index.xml
196Sitemap: https://sourceforge.net/articles/sitemap_index.xml
197Sitemap: https://sourceforge.net/freeware_sitemap.xml
198
What do the bots actually get?
This page reads robots.txt, which states what sourceforge.net asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on sourceforge.net Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does sourceforge.net block GPTBot?
No. sourceforge.net’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does sourceforge.net block ClaudeBot?
No. sourceforge.net’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does sourceforge.net block Google-Extended?
No. sourceforge.net’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does sourceforge.net block PerplexityBot?
No. sourceforge.net’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find sourceforge.net?
Yes. sourceforge.net’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.