canaicrawl Get early access

Who blocks AI · #568 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does inc.com block AI crawlers?

Yes. inc.com’s robots.txt blocks 10 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, PerplexityBot and 7 more. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotBlockedOAI-SearchBotPartly blockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedPartly blockedCCBotBlocked
robots.txt
www.inc.com/robots.txt · 146 lines
llms.txt
none
Checked
9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGooglePartly blockedline 25 · Disallow: /rest
BingbotMicrosoftPartly blockedline 25 · Disallow: /rest
DuckDuckBotDuckDuckGoPartly blockedline 25 · Disallow: /rest
ApplebotApplePartly blockedline 25 · Disallow: /rest
PetalBotHuaweiAllowedline 5 · Allow: / under User-agent: *
YandexBotYandexAllowedline 5 · Allow: / under User-agent: *
BaiduspiderBaiduPartly blockedline 25 · Disallow: /rest

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIPartly blockedline 37 · Disallow: /rest
PerplexityBotPerplexityBlockedline 62 · Disallow: / under User-agent: PerplexityBot
Claude-SearchBotAnthropicPartly blockedline 37 · Disallow: /rest
DuckAssistBotDuckDuckGoBlockedline 137 · Disallow: / under User-agent: DuckAssistBot
AmazonbotAmazonBlockedline 95 · Disallow: / under User-agent: Amazonbot
YouBotYou.comBlockedline 56 · Disallow: / under User-agent: YouBot

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIBlockedline 41 · Disallow: / under User-agent: GPTBot
ClaudeBotAnthropicBlockedline 71 · Disallow: / under User-agent: ClaudeBot
Google-ExtendedGooglePartly blockedline 25 · Disallow: /rest
CCBotCommon CrawlBlockedline 104 · Disallow: / under User-agent: ccbot
Applebot-ExtendedAppleBlockedline 98 · Disallow: / under User-agent: Applebot-Extended
GoogleOtherGoogleAllowedline 5 · Allow: / under User-agent: *
Meta-ExternalAgentMetaPartly blockedline 25 · Disallow: /rest
BytespiderByteDanceBlockedline 101 · Disallow: / under User-agent: Bytespider

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIPartly blockedline 32 · Disallow: /rest
Claude-UserAnthropicPartly blockedline 32 · Disallow: /rest
Perplexity-UserPerplexityPartly blockedline 32 · Disallow: /rest
MistralAI-UserMistralBlockedline 134 · Disallow: / under User-agent: MistralAI-User
Meta-ExternalFetcherMetaAllowedline 5 · Allow: / under User-agent: *

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsAllowedline 5 · Allow: / under User-agent: *
SemrushBotSemrushAllowedline 5 · Allow: / under User-agent: *
MJ12botMajesticAllowedline 5 · Allow: / under User-agent: *
DotBotMozAllowedline 5 · Allow: / under User-agent: *
View www.inc.com/robots.txt · 146 lines
1
2
3# from CDN
4User-Agent: *
5Allow: /
6Disallow: /rest
7
8# Legitimate Search Engines & Analytics
9User-Agent: Googlebot
10User-Agent: msnbot
11User-Agent: bingbot
12User-Agent: Twitterbot
13User-Agent: Mediapartners-Google
14User-Agent: Google-Extended
15User-Agent: Meta-ExternalAgent
16User-Agent: Meta-WebIndexer
17User-Agent: facebookexternalhit
18User-Agent: ScoutJet
19User-Agent: Yandex
20User-Agent: baiduspider
21User-Agent: DuckDuckBot
22User-Agent: BomboraBot
23User-Agent: grapeshot
24Allow: /
25Disallow: /rest
26
27# User-Facing AI Assistants (ALLOWED)
28User-Agent: ChatGPT-User
29User-Agent: Claude-User
30User-Agent: Perplexity-User
31Allow: /
32Disallow: /rest
33
34User-Agent: OAI-SearchBot
35User-Agent: Claude-SearchBot
36Allow: /
37Disallow: /rest
38
39# AI Training Bots (BLOCKED)
40User-Agent: GPTBot
41Disallow: /
42
43User-Agent: ChatGPT
44Disallow: /
45
46User-Agent: Bard
47Disallow: /
48
49User-Agent: Jasper
50Disallow: /
51
52User-Agent: YouChat
53Disallow: /
54
55User-Agent: YouBot
56Disallow: /
57
58User-Agent: Perplexity
59Disallow: /
60
61User-Agent: PerplexityBot
62Disallow: /
63
64User-Agent: Claude
65Disallow: /
66
67User-Agent: Claude-Web
68Disallow: /
69
70User-Agent: ClaudeBot
71Disallow: /
72
73User-Agent: anthropic-ai
74Disallow: /
75
76User-Agent: Neeva
77Disallow: /
78
79User-Agent: Copy.ai
80Disallow: /
81
82User-Agent: Writesonic
83Disallow: /
84
85User-Agent: Anyword
86Disallow: /
87
88User-Agent: AI21
89Disallow: /
90
91User-Agent: FacebookBot
92Disallow: /
93
94User-Agent: Amazonbot
95Disallow: /
96
97User-Agent: Applebot-Extended
98Disallow: /
99
100User-Agent: Bytespider
101Disallow: /
102
103User-Agent: ccbot
104Disallow: /
105
106User-Agent: CCBot
107Disallow: /
108
109User-Agent: Cohere-AI
110Disallow: /
111
112User-Agent: Diffbot
113Disallow: /
114
115User-Agent: ImagesiftBot
116Disallow: /
117
118User-Agent: Omgilibot
119Disallow: /
120
121User-Agent: Omgili
122Disallow: /
123
124User-Agent: PiplBot
125Disallow: /
126
127User-Agent: Scrapy
128Disallow: /
129
130User-Agent: Webzio-Extended
131Disallow: /
132
133User-agent: MistralAI-User
134Disallow: /
135
136User-agent: DuckAssistBot
137Disallow: /
138
139User-agent: DeepSeekBot
140Disallow: /
141
142Sitemap: https://www.inc.com/sitemap_index.xml
143Sitemap: https://www.inc.com/sitemap/sitemap_news.xml
144Sitemap: https://www.inc.com/sitemap/podcasts.xml
145
146

What do the bots actually get?

This page reads robots.txt, which states what inc.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on inc.com Test a path in the robots.txt tester

History

Questions

Does inc.com block GPTBot?

Yes. inc.com’s robots.txt disallows GPTBot from the whole site (line 41: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does inc.com block ClaudeBot?

Yes. inc.com’s robots.txt disallows ClaudeBot from the whole site (line 71: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does inc.com block Google-Extended?

Partly. inc.com’s robots.txt keeps Google-Extended out of some paths, such as /rest, and lets it crawl the rest. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does inc.com block PerplexityBot?

Yes. inc.com’s robots.txt disallows PerplexityBot from the whole site (line 62: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find inc.com?

Partly. inc.com’s robots.txt keeps OAI-SearchBot out of some paths, such as /rest, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.