Who blocks AI · #97 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does vk.com block AI crawlers?
No. vk.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- vk.com/robots.txt · 270 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 94 · Disallow: /doc-* line 95 · Disallow: /away.php and 72 more | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Partly blocked | line 94 · Disallow: /doc-* line 95 · Disallow: /away.php and 72 more |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View vk.com/robots.txt · 270 lines
1User-agent: *
2Disallow: /doc-*
3Disallow: /away.php
4Disallow: /im?
5Disallow: /search*&*&*&
6Disallow: *?w=story
7Disallow: *?w=wall
8Disallow: *?w=page
9Disallow: *?w=app
10Disallow: *?w=poll
11Disallow: *?w=service-booking-*
12Disallow: *?w=likes
13Disallow: *?w=shares
14Disallow: *?w=note
15Disallow: *?w=away
16Disallow: /call?id=
17Disallow: /feed$
18Disallow: /feed*
19Disallow: /bookmarks*
20Disallow: /friends*
21Disallow: /gifts
22Disallow: /groups?id=
23Disallow: /narrative-
24Disallow: /audio?performer=1&q=
25Disallow: /stats?gid=
26Disallow: /write-
27Disallow: /app*_
28Disallow: *api_access_key=
29Disallow: *_query=
30Disallow: /wall*?reply=
31Disallow: /wall*?thread=
32Disallow: /search*sort=
33Disallow: *m=1$
34Disallow: *loc=
35Disallow: *from_id=
36Disallow: *ad_id=
37Disallow: *_smt=
38Disallow: /app*form_id=
39Disallow: /podcast*t=
40Disallow: /coin*mid=
41Disallow: *z=album
42Disallow: *z=clip
43Disallow: *z=photo
44Disallow: *z=video
45Disallow: /market_catalog/*?w=product-
46Disallow: /page-
47Disallow: /ua_mailing
48Disallow: *?w=address
49Disallow: /*?null$
50Disallow: /pages?oid=-
51Disallow: /*/video-
52Disallow: *?order=
53Disallow: /wall-*?q=
54Disallow: /widget_auth.php
55Disallow: /wall*?own=1
56Disallow: /wall*?search=1
57Disallow: /*?offset=0
58Disallow: /*?utm_*=
59Disallow: /topic*?post=
60Disallow: /login?*=
61Disallow: /feed?section=search&q=
62Disallow: /wall*?with=
63Disallow: *?act=
64Disallow: /call/*
65Disallow: /calls*
66Disallow: /429.html
67Disallow: *?slogin*
68Disallow: *?ysclid*
69Allow: /apps?act=search_page&tags=
70Allow: /games?act=search_page&tags=
71Allow: /apps*&tags=
72Allow: /page-777107_28406709$
73Allow: /page-2158488_44000074$
74Allow: /page-59800369_45472411$
75Allow: /page-2158488_44596199$
76Allow: /page-2158488_50129817$
77Allow: /search*offset=
78Allow: /feed?section=coronavirus$
79Allow: /feed?section=stayhome$
80Disallow: *?reactions_opened*
81Disallow: *share.php?*
82Disallow: *?list=wall
83Disallow: *?reply=
84Disallow: *?from
85Disallow: *?from=profile
86Disallow: *?post_add
87Disallow: *?menu_opened
88Disallow: */challenge.html*
89Disallow: */widget_*
90Disallow: */_widget*
91Disallow: *err404.php*
92
93User-agent: Googlebot
94Disallow: /doc-*
95Disallow: /away.php
96Disallow: /im?
97Disallow: /search*&*&*&
98Disallow: *?w=story
99Disallow: *?w=wall
100Disallow: *?w=page
101Disallow: *?w=app
102Disallow: *?w=poll
103Disallow: *?w=service-booking-*
104Disallow: *?w=likes
105Disallow: *?w=shares
106Disallow: *?w=note
107Disallow: *?w=away
108Disallow: /call?id=
109Disallow: /feed$
110Disallow: /feed*
111Disallow: /bookmarks*
112Disallow: /friends*
113Disallow: /gifts
114Disallow: /groups?id=
115Disallow: /narrative-
116Disallow: /audio?performer=1&q=
117Disallow: /stats?gid=
118Disallow: /write-
119Disallow: /app*_
120Disallow: *api_access_key=
121Disallow: *_query=
122Disallow: /wall*?reply=
123Disallow: /wall*?thread=
124Disallow: /search*sort=
125Disallow: *m=1$
126Disallow: *loc=
127Disallow: *from_id=
128Disallow: *ad_id=
129Disallow: *_smt=
130Disallow: /app*form_id=
131Disallow: /podcast*t=
132Disallow: /coin*mid=
133Disallow: *z=album
134Disallow: *z=clip
135Disallow: *z=photo
136Disallow: *z=video
137Disallow: /market_catalog/*?w=product-
138Disallow: /ua_mailing
139Disallow: *?w=address
140Disallow: /*?null$
141Disallow: /*/video-
142Disallow: *?order=
143Disallow: /wall-*?q=
144Disallow: /widget_auth.php
145Disallow: /wall*?own=1
146Disallow: /wall*?search=1
147Disallow: /*?offset=0
148Disallow: /*?utm_*=
149Disallow: /topic*?post=
150Disallow: /feed?section=search&q=
151Disallow: /wall*?with=
152Disallow: *?act=
153Disallow: /call/*
154Disallow: *?slogin*
155Disallow: *?ysclid*
156Allow: /apps?act=search_page&tags=
157Allow: /games?act=search_page&tags=
158Allow: /apps*&tags=
159Allow: /page-777107_28406709$
160Allow: /page-2158488_44000074$
161Allow: /page-59800369_45472411$
162Allow: /page-2158488_44596199$
163Allow: /page-2158488_50129817$
164Allow: /search*offset=
165Allow: /feed?section=coronavirus$
166Allow: /feed?section=stayhome
167Disallow: *?reactions_opened*
168Disallow: *share.php?*
169Disallow: *?list=wall
170Disallow: *?reply=
171Disallow: *?from
172Disallow: *?from=profile
173Disallow: *?post_add
174Disallow: *?menu_opened
175Disallow: */challenge.html*
176Disallow: */widget_*
177Disallow: */_widget*
178Disallow: *err404.php*
179
180User-agent: Yandex
181Disallow: /doc-*
182Disallow: /away.php
183Disallow: /im?
184Disallow: /search*&*&*&
185Disallow: *?w=story
186Disallow: *?w=service-booking-*
187Disallow: *?w=wall
188Disallow: *?w=page
189Disallow: *?w=app
190Disallow: *?w=poll
191Disallow: *?w=likes
192Disallow: *?w=shares
193Disallow: *?w=note
194Disallow: *?w=away
195Disallow: /call?id=
196Disallow: /feed$
197Disallow: /feed*
198Disallow: /bookmarks*
199Disallow: /friends*
200Disallow: /gifts
201Disallow: /groups?id=
202Disallow: /narrative-
203Disallow: /audio?performer=1&q=
204Disallow: /stats?gid=
205Disallow: /write-
206Disallow: /app*_
207Disallow: *api_access_key=
208Disallow: *_query=
209Disallow: /wall*?reply=
210Disallow: /wall*?thread=
211Disallow: /search*sort=
212Disallow: *m=1$
213Disallow: *loc=
214Disallow: *from_id=
215Disallow: *ad_id=
216Disallow: *_smt=
217Disallow: /app*form_id=
218Disallow: /podcast*t=
219Disallow: /coin*mid=
220Disallow: *z=album
221Disallow: *z=clip
222Disallow: *z=photo
223Disallow: *z=video
224Disallow: /market_catalog/*?w=product-
225Disallow: /page-
226Disallow: /ua_mailing
227Disallow: *?w=address
228Disallow: /*?null$
229Disallow: /pages?oid=-
230Disallow: /*/video-
231Disallow: *?order=
232Disallow: /wall-*?q=
233Disallow: /widget_auth.php
234Disallow: /wall*?own=1
235Disallow: /wall*?search=1
236Disallow: /*?offset=0
237Disallow: /*?utm_*=
238Disallow: /topic*?post=
239Disallow: /login?*=
240Disallow: /feed?section=search&q=
241Disallow: /wall*?with=
242Disallow: *?act=
243Disallow: /call/*
244Disallow: *?slogin*
245Disallow: *?ysclid*
246Allow: /apps?act=search_page&tags=
247Allow: /games?act=search_page&tags=
248Allow: /apps*&tags=
249Allow: /page-777107_28406709$
250Allow: /page-2158488_44000074$
251Allow: /page-59800369_45472411$
252Allow: /page-2158488_44596199$
253Allow: /page-2158488_50129817$
254Allow: /search*offset=
255Allow: /feed?section=coronavirus$
256Allow: /feed?section=stayhome$
257Disallow: *?reactions_opened*
258Disallow: *share.php?*
259Disallow: *?list=wall
260Disallow: *?reply=
261Disallow: *?from
262Disallow: *?from=profile
263Disallow: *?post_add
264Disallow: *?menu_opened
265Disallow: */challenge.html*
266Disallow: */widget_*
267Disallow: */_widget*
268Disallow: *err404.php*
269Allow: /s/v1/doc/HumbuSZq4Cz8r12yDSagIdh_NxHYmKpay8X54ZO2sua7_f_yyQk
270Clean-param: ref&list&from&rev&anchor&utm_source&utm_medium&utm_campaign&01&02&03&04&05&06&07&08&09&10&11&12&13&14&15&16&17&18&19&20
What do the bots actually get?
This page reads robots.txt, which states what vk.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on vk.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does vk.com block GPTBot?
No. vk.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does vk.com block ClaudeBot?
No. vk.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does vk.com block Google-Extended?
No. vk.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does vk.com block PerplexityBot?
No. vk.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find vk.com?
Yes. vk.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.