Who blocks AI · #383 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does gartner.com block AI crawlers?
Yes. gartner.com’s robots.txt blocks 5 of the 19 AI crawlers we track, including GPTBot, Google-Extended, OAI-SearchBot and 2 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotPartly blockedPerplexityBotAllowedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.gartner.com/robots.txt · 160 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 37 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 37 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Partly blocked | line 149 · Disallow: /peer-community/* line 150 · Disallow: /reviews/ and 1 more |
Google-Extended | Blocked | line 37 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 31 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Partly blocked | line 149 · Disallow: /peer-community/* line 150 · Disallow: /reviews/ and 1 more |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Partly blocked | line 149 · Disallow: /peer-community/* line 150 · Disallow: /reviews/ and 1 more |
Bytespider | ByteDance | Partly blocked | line 149 · Disallow: /peer-community/* line 150 · Disallow: /reviews/ and 1 more |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 37 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.gartner.com/robots.txt · 160 lines
1User-agent: *
2Sitemap: https://www.gartner.com/ngw/commonassets/code/xml/sitemap-index.xml
3Sitemap: https://www.gartner.com/reviews/sitemap.xml
4Sitemap: https://www.gartner.com/peer-community/sitemap_index1.xml.gz
5Sitemap: https://www.gartner.com/peer-community/sitemap.xml
6Disallow: /cgi-bin/
7Disallow: /technology/media-products/
8Disallow: /it/content/
9Disallow: /pv_obj_cache/
10Disallow: /technology/admin/adm_help.jsp
11Disallow: /resources/
12Disallow: /review/view/
13Disallow: /reviews/*/review/view/
14Disallow: /reviews/market/*/compare/*-vs-*-vs-*
15Disallow: /reviews/*/pageInfo
16Disallow: /it/products/podcasting/attributes/
17Disallow: /technology/html_emails/
18Disallow: /login/
19Disallow: /it/sd/
20Disallow: /imagesrv/
21Disallow: /ngw/*.doc$
22Disallow: /ngw/*.docx$
23Disallow: /ngw/*.pdf$
24Disallow: /ngw/*.ppt$
25Disallow: /ngw/*.pptx$
26Disallow: /ngw/*.xls$
27Disallow: /ngw/*.xlsx$
28Disallow: /en/*automation-test-area*
29
30User-agent: CCBot
31Disallow: /
32
33User-agent: Google-Extended
34User-agent: GPTBot
35User-agent: ChatGPT-User
36User-agent: OAI-SearchBot
37Disallow: /
38Allow: /en$
39Allow: /en/about$
40Allow: /en/about/
41Allow: /en/ai$
42Allow: /en/articles/
43Allow: /en/attendees-navigator/
44Allow: /en/audit-risk$
45Allow: /en/audit-risk/
46Allow: /en/become-a-client$
47Allow: /en/chief-information-officer$
48Allow: /en/chief-information-officer/
49Allow: /en/communications$
50Allow: /en/communications/
51Allow: /en/conferences$
52Allow: /en/conferences/
53Allow: /en/connect-with-peers$
54Allow: /en/connect-with-peers/
55Allow: /en/consulting$
56Allow: /en/consulting/
57Allow: /en/contact/
58Allow: /en/corporate-communications/
59Allow: /en/customer-service-support$
60Allow: /en/customer-service-support/
61Allow: /en/customer-success-stories$
62Allow: /en/customer-success-stories/
63Allow: /en/cybersecurity$
64Allow: /en/cybersecurity/
65Allow: /en/data-analytics$
66Allow: /en/data-analytics/
67Allow: /en/doc/
68Allow: /en/documents/
69Allow: /en/enterprise-architecture$
70Allow: /en/enterprise-architecture/
71Allow: /en/executive-guidance/
72Allow: /en/experts$
73Allow: /en/experts/
74Allow: /en/finance$
75Allow: /en/finance/
76Allow: /en/financial-services$
77Allow: /en/financial-services/
78Allow: /en/glossary/
79Allow: /en/human-resources$
80Allow: /en/human-resources/
81Allow: /en/industries/
82Allow: /en/information-technology$
83Allow: /en/information-technology/
84Allow: /en/infrastructure-and-it-operations-leaders$
85Allow: /en/infrastructure-and-it-operations-leaders/
86Allow: /en/innovation-strategy$
87Allow: /en/innovation-strategy/
88Allow: /en/insights$
89Allow: /en/insights/
90Allow: /en/legal-compliance$
91Allow: /en/legal-compliance/
92Allow: /en/marketing$
93Allow: /en/marketing/
94Allow: /en/newsletter-signup$
95Allow: /en/newsroom$
96Allow: /en/newsroom/
97Allow: /en/podcasts/
98Allow: /en/product-management$
99Allow: /en/product-management/
100Allow: /en/products/
101Allow: /en/publications$
102Allow: /en/publications/
103Allow: /en/research/
104Allow: /en/research-development$
105Allow: /en/research-development/
106Allow: /en/sales$
107Allow: /en/sales/
108Allow: /en/site-index$
109Allow: /en/site-map$
110Allow: /en/smarterwithgartner/
111Allow: /en/software-engineering$
112Allow: /en/software-engineering/
113Allow: /en/strategy-insights-tribal-enterprises$
114Allow: /en/strategy-insights-tribal-enterprises/
115Allow: /en/supply-chain$
116Allow: /en/supply-chain/
117Allow: /en/tech-ceo$
118Allow: /en/tech-ceo/
119Allow: /en/tech-marketing$
120Allow: /en/tech-marketing/
121Allow: /en/tools$
122Allow: /en/tools/
123Allow: /en/topics/
124Allow: /en/trends/
125Allow: /en/videos/
126Allow: /en/webinar/
127Allow: /en/webinars$
128Allow: /en/why-gartner$
129Allow: /jp/conferences/
130Allow: /pt-br/conferences/
131
132
133
134User-agent: Applebot-Extended
135User-agent: Arquivo-web-crawler
136User-agent: archive.org_bot
137User-agent: Bytespider
138User-agent: CCBot
139User-agent: ClaudeBot
140User-agent: Crawlspace
141User-agent: Diffbot
142User-agent: Google-Extended
143User-agent: GPTBot
144User-agent: Meta-ExternalAgent
145User-agent: PanguBot
146User-agent: Timpibot
147User-agent: YandexAdditional
148User-agent: YandexAdditionalBot
149Disallow: /peer-community/*
150Disallow: /reviews/
151Disallow: /peer-insights/
152
153User-agent: Google-Extended
154User-agent: ChatGPT-User
155User-agent: OAI-SearchBot
156Disallow: /
157Allow: /reviews/
158Allow: /peer-insights/
159Disallow: /review/view/
160Disallow: /reviews/*/review/view/
What do the bots actually get?
This page reads robots.txt, which states what gartner.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on gartner.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does gartner.com block GPTBot?
Yes. gartner.com’s robots.txt disallows GPTBot from the whole site (line 37: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does gartner.com block ClaudeBot?
Partly. gartner.com’s robots.txt keeps ClaudeBot out of some paths, such as /peer-community/*, and lets it crawl the rest. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does gartner.com block Google-Extended?
Yes. gartner.com’s robots.txt disallows Google-Extended from the whole site (line 37: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does gartner.com block PerplexityBot?
No. gartner.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find gartner.com?
Not through its own crawler. gartner.com’s robots.txt disallows OAI-SearchBot from the whole site (line 37: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.