Who blocks AI · #658 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does slashdot.org block AI crawlers?
No. slashdot.org’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- slashdot.org/robots.txt · 121 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View slashdot.org/robots.txt · 121 lines
1# robots.txt for Slashdot.org
2
3User-agent: *
4Disallow: /authors.pl
5Disallow: /index.pl
6Disallow: /index2.pl
7Disallow: /firehose.pl
8Disallow: /messages.pl
9Disallow: /metamod.pl
10Disallow: /users.pl
11Disallow: /users2.pl
12Disallow: /search.pl
13Disallow: /search
14Disallow: /submit.pl
15Disallow: /pollBooth.pl
16Disallow: /pubkey.pl
17Disallow: /topics.pl
18Disallow: /zoo.pl
19Disallow: /palm
20Disallow: /bookmark.pl
21Disallow: /%7E
22Disallow: /~*/friends
23Disallow: /~*/fans
24Disallow: /~*/foes
25Disallow: /~*/freaks
26Disallow: /~*/tags
27Disallow: /~*/achievements
28Disallow: /submission
29Disallow: /journal
30Disallow: /comments.pl
31Disallow: /slashdot-it.pl
32Disallow: /tag
33Disallow: /stories/
34Disallow: /interviews/
35Disallow: /?issue=
36Disallow: /software/link?
37Disallow: /software/link$
38Disallow: /software/visit?
39Disallow: /software/visit$
40Disallow: /software/tp/?
41# tracking pixels moved off /software/ (SF-12035)
42Disallow: /svc/
43Disallow: /software/comparison/*-vs-*-vs-*-vs-
44Disallow: /software/comparison/*/add-software
45Disallow: /software/p/*/reviews/new
46Disallow: /software/p/*/reviews/add
47Disallow: /software/*?q=
48Disallow: /software/*&q=
49Disallow: /software/*?categories=
50Disallow: /software/*&categories=
51Disallow: /software/*?company_sizes=
52Disallow: /software/*&company_sizes=
53Disallow: /software/*?deployment=
54Disallow: /software/*&deployment=
55Disallow: /software/*?free_options=
56Disallow: /software/*&free_options=
57Disallow: /software/*?integrates_with=
58Disallow: /software/*&integrates_with=
59Disallow: /software/*?org_types=
60Disallow: /software/*&org_types=
61Disallow: /software/*?api=
62Disallow: /software/*&api=
63Disallow: /software/*?regions=
64Disallow: /software/*®ions=
65Disallow: /software/*?support=
66Disallow: /software/*&support=
67Disallow: /software/*?training=
68Disallow: /software/*&training=
69Disallow: /software/*?feature_
70Disallow: /software/*&feature_
71Disallow: /software/*?deployment=
72Disallow: /software/*&deployment=
73Disallow: /software/transfer_site
74Disallow: /about/floating_sitewide
75Disallow: /about/review-guidelines
76Disallow: /jobs
77Disallow: /software/tp3
78Disallow: /directory/tp
79Disallow: /my/
80Disallow: /*/wp-login.php
81Disallow: /supporters.shtml
82Disallow: /topics.shtml
83Disallow: /awards.shtml
84Disallow: /login.pl
85Disallow: /api/v2/record_user_opt_out
86Disallow: /prefs
87Disallow: /content/category/
88Disallow: /content/tag/
89Disallow: /hi
90Allow: /llms.txt
91
92Sitemap: https://slashdot.org/sitemap-slashdot-index.xml
93Sitemap: https://apache.slashdot.org/sitemap-apache-index.xml
94Sitemap: https://apple.slashdot.org/sitemap-apple-index.xml
95Sitemap: https://askslashdot.slashdot.org/sitemap-askslashdot-index.xml
96Sitemap: https://books.slashdot.org/sitemap-bookreview-index.xml
97Sitemap: https://bsd.slashdot.org/sitemap-bsd-index.xml
98Sitemap: https://build.slashdot.org/sitemap-build-index.xml
99Sitemap: https://developers.slashdot.org/sitemap-developers-index.xml
100Sitemap: https://devices.slashdot.org/sitemap-devices-index.xml
101Sitemap: https://entertainment.slashdot.org/sitemap-entertainment-index.xml
102Sitemap: https://features.slashdot.org/sitemap-features-index.xml
103Sitemap: https://games.slashdot.org/sitemap-games-index.xml
104Sitemap: https://hardware.slashdot.org/sitemap-hardware-index.xml
105Sitemap: https://idle.slashdot.org/sitemap-idle-index.xml
106Sitemap: https://interviews.slashdot.org/sitemap-interviews-index.xml
107Sitemap: https://it.slashdot.org/sitemap-it-index.xml
108Sitemap: https://linux.slashdot.org/sitemap-linux-index.xml
109Sitemap: https://meta.slashdot.org/sitemap-meta-index.xml
110Sitemap: https://mobile.slashdot.org/sitemap-mobile-index.xml
111Sitemap: https://news.slashdot.org/sitemap-news-index.xml
112Sitemap: https://politics.slashdot.org/sitemap-politics-index.xml
113Sitemap: https://radio.slashdot.org/sitemap-radio-index.xml
114Sitemap: https://science.slashdot.org/sitemap-science-index.xml
115Sitemap: https://search.slashdot.org/sitemap-search-index.xml
116Sitemap: https://technology.slashdot.org/sitemap-technology-index.xml
117Sitemap: https://yro.slashdot.org/sitemap-yro-index.xml
118
119Sitemap: https://slashdot.org/software/software_sitemap.xml
120Sitemap: https://slashdot.org/content/sitemap_index.xml
121
What do the bots actually get?
This page reads robots.txt, which states what slashdot.org asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on slashdot.org Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does slashdot.org block GPTBot?
No. slashdot.org’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does slashdot.org block ClaudeBot?
No. slashdot.org’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does slashdot.org block Google-Extended?
No. slashdot.org’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does slashdot.org block PerplexityBot?
No. slashdot.org’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find slashdot.org?
Yes. slashdot.org’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.