Who blocks AI · #640 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does semrush.com block AI crawlers?
No. semrush.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.semrush.com/robots.txt · 156 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Partly blocked | line 22 · Disallow: /archive/graphs.php |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.semrush.com/robots.txt · 156 lines
1User-agent: 008
2Disallow: /
3
4User-agent: SiteAuditBot
5Crawl-delay: 1
6Allow: /
7
8User-agent: Semrushbot-SI
9Allow: /
10
11User-agent: Yahoo Pipes 2.0
12Disallow: /
13
14User-agent: Facebot
15Disallow: /archive/graphs.php
16Allow: /academy/certificates/*
17
18User-agent: facebookexternalhit
19Allow: /academy/certificates/*
20
21User-agent: Bingbot
22Disallow: /archive/graphs.php
23
24User-agent: BingPreview
25Disallow: /archive/graphs.php
26
27User-agent: LinkedInBot
28Allow: /academy/certificates/*
29
30User-agent: Twitterbot
31Allow: /blog/*
32Allow: /ranking-factors/*
33Allow: /academy/certificates/*
34
35User-agent: *
36Disallow: /blog/*?img=*
37Disallow: /archive/graphs.php
38Disallow: /forrester-marketing-silos/*
39Disallow: /partner/
40Disallow: /blog/search/*
41Disallow: /analytics/seomagic/
42Disallow: /my_reports/reports/*
43Disallow: /archive/
44Disallow: /admin/
45Disallow: /users/
46Disallow: /clients/
47Disallow: /custom_report/
48Disallow: /limit_hits/
49Disallow: /payment/
50Disallow: /no_cookies/
51Disallow: /double_activation/
52Disallow: /projects/?*
53Disallow: /?*/
54Disallow: /activation_successful*
55Disallow: /api.html?*
56Disallow: /partners/
57Disallow: /partner/
58Disallow: */dvd-mode/
59Disallow: /unsubscribe/
60Disallow: /unsubscribe/*
61Disallow: /demonstration/
62Disallow: /demonstration/*
63Allow: /projects/
64Disallow: */custom-report/
65Disallow: */request-demo/
66Allow: */info/kdt/
67Disallow: */pleasereg/
68Disallow: */custom-report/
69Disallow: */custom-report
70Allow: */sensor/*
71Disallow: /swa/api/*
72Disallow: /redirect?url=*
73Disallow: /sso/
74Disallow: /_/semrush_regular_1_blank/
75Disallow: /seo-old/*
76Disallow: /projects2/*
77Disallow: /free-tools/plagiarism-checker/?q=
78Allow: /news/*?page=*
79Allow: /analytics/keywordoverview/*
80
81# Community rules
82Allow: /blog/*
83Disallow: /blog/search/*
84Disallow: /blog/*utm_source=*
85Disallow: /blog-manager/*
86Allow: /user/*
87
88# Disallow technical signup pages
89Allow: /login/$
90Allow: /signup/$
91Disallow: /login/*
92Disallow: /signup/*
93
94# Enterprise
95Allow: /enterprise/seo/
96
97#landing
98Disallow: /landing/
99Disallow: /lp/
100Allow: /lp/affiliate-program/*
101Allow: /lp/for-agencies/en/
102Allow: /lp/cannes2026/*
103Allow: /lp/enterprise-aio/*
104Allow: /lp/enterprise-crawler/*
105Allow: /lp/enterprise-partner-program/*
106Allow: /lp/seasonal-ai/en/
107Allow: /lp/black-friday/*
108Disallow: /lp/black-friday-aff/*
109Allow: /lp/cyber-week/*
110Allow: /lp/journeys/*
111Allow: /lp/insights/*
112Allow: /lp/growth-quadrant/
113Allow: /lp/mfour/*
114Allow: /lp/semrush-circle/*
115Allow: /lp/semrush-one/*
116Allow: /seo/
117Allow: /lp/seo-consultation/*
118Allow: /content-marketing/
119Allow: /ppc/
120Allow: /competitive-research/
121Allow: /analytics/traffic/overview/
122Disallow: /listings-management/superpower-landing2/
123Allow: /semrush-questions/*
124
125#academy
126Allow: /academy/*
127Disallow: /academy/*utm_source=*
128Disallow: /academy/*/results
129Disallow: /academy/certificates/*
130
131#websites
132Allow: /website/*
133Disallow: /website/search/*
134
135Disallow: /*/?*
136Crawl-delay: 20
137
138# Sitemap files
139#Blog
140Sitemap: https://www.semrush.com/sitemap.xml
141Sitemap: https://pt.semrush.com/blog/sitemap/
142Sitemap: https://es.semrush.com/blog/sitemap/
143Sitemap: https://it.semrush.com/blog/sitemap/
144Sitemap: https://fr.semrush.com/blog/sitemap/
145Sitemap: https://de.semrush.com/blog/sitemap/
146Sitemap: https://ja.semrush.com/blog/sitemap/
147#Academy
148Sitemap: https://www.semrush.com/academy/sitemap.xml
149Sitemap: https://es.semrush.com/academy/sitemap.xml
150Sitemap: https://pt.semrush.com/academy/sitemap.xml
151Sitemap: https://it.semrush.com/academy/sitemap.xml
152Sitemap: https://fr.semrush.com/academy/sitemap.xml
153Sitemap: https://de.semrush.com/academy/sitemap.xml
154#Free tools:
155Sitemap: https://www.semrush.com/free-tools/sitemap.xml
156
What do the bots actually get?
This page reads robots.txt, which states what semrush.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on semrush.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does semrush.com block GPTBot?
No. semrush.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does semrush.com block ClaudeBot?
No. semrush.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does semrush.com block Google-Extended?
No. semrush.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does semrush.com block PerplexityBot?
No. semrush.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find semrush.com?
Yes. semrush.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.