Who blocks AI · #554 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does ok.ru block AI crawlers?
No. ok.ru’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- ok.ru/robots.txt · 240 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View ok.ru/robots.txt · 240 lines
1User-agent: *
2Disallow: /cdk/*
3Disallow: *jsessionid*
4Disallow: *tkn/*
5Disallow: /mapi?*
6Disallow: /?ffsputnik=*
7Disallow: /?_erv=*
8Disallow: /?sputnik=*
9Disallow: /?cat=*
10Disallow: *st.redirect*
11Disallow: *cmd=logExternal*
12Disallow: *?fromTime=*
13Disallow: *?cmd*
14Disallow: /messages/join/
15Disallow: /?st.password=*
16Disallow: /joincall/*
17Disallow: /gifts/link/*
18Disallow: /qrCode/*
19Disallow: /groupcallembed/*
20Disallow: /dk?*
21Disallow: /discussions/*
22Disallow: /hobby-mailru
23Disallow: /*;jsessionid
24Disallow: /music/search/*
25Disallow: /search/content/.*
26Disallow: /search/profiles/.*
27Disallow: /games/.*
28Disallow: /vitrine/.*
29Disallow: /albums/.*
30Disallow: /.*
31Disallow: /hobby/*/post/ugc$
32Disallow: /stickers/set/*
33Disallow: /stickers$
34Disallow: /profile/*/hobby$
35Disallow: /group/*/hobby$
36
37Clean-param: st.cmd
38Clean-param: &st.layer.cmd
39Clean-param: &st._forceSetHistory
40Clean-param: &st._aid
41Clean-param: &st.category
42Clean-param: &st._surl
43Clean-param: &st.error
44Clean-param: &st.email
45Clean-param: &st.refplace
46Clean-param: &st.layer.cmd
47Clean-param: &st.refplace
48Clean-param: &__ref
49Clean-param: &cmn
50Clean-param: &autoplay
51Clean-param: &bonus
52Clean-param: &_cp_
53Clean-param: &promo
54Clean-param: &ad_id
55Clean-param: &group_bonus
56Clean-param: &imageId
57Clean-param: &nochat
58Clean-param: &promokey
59Clean-param: &adv_id
60Clean-param: &ref
61Clean-param: &gid
62Clean-param: &h_a
63Clean-param: &gift
64Clean-param: &silent
65Clean-param: &shareBonusFromPlayerGuid
66Clean-param: &_ref
67Clean-param: &auto
68Clean-param: &mute
69Clean-param: &hd
70Clean-param: &ad_level
71Clean-param: &rel
72Clean-param: &ya
73Clean-param: &textId
74Clean-param: &wmode
75Clean-param: &title
76Clean-param: &etext
77Clean-param: &auto_play
78Clean-param: &muted
79Clean-param: &fromTime
80Clean-param: &linknhung
81Clean-param: &mad
82Clean-param: &brandsafety
83Clean-param: &embed
84Clean-param: &hl
85Clean-param: &mt_link_id
86Clean-param: &tv
87Clean-param: &showinfo
88Clean-param: &refplace
89Clean-param: &mt_creative
90Clean-param: &mt_adset
91Clean-param: &mt_click_id
92Clean-param: &mt_network
93Clean-param: &mt_campaign
94Clean-param: &mt_sub1
95Clean-param: &mt_sub2
96Clean-param: &mt_sub3
97Clean-param: &mt_sub4
98Clean-param: &mt_sub5
99Clean-param: &__dp
100Clean-param: &ip
101Clean-param: &_cl.id
102Clean-param: &_clickLog
103Clean-param: &_cl.list
104Clean-param: &_cl.sID
105Clean-param: &just-logged-in
106Clean-param: &_sm_au_
107Clean-param: &__ref
108Clean-param: &cmn
109Clean-param: &autoplay
110Clean-param: &bonus
111Clean-param: &_cp_
112Clean-param: &promo
113Clean-param: &ad_id
114Clean-param: &group_bonus
115Clean-param: &imageId
116Clean-param: &nochat
117Clean-param: &promokey
118Clean-param: &adv_id
119Clean-param: &ref
120Clean-param: &gid
121Clean-param: &h_a
122Clean-param: &gift
123Clean-param: &silent
124Clean-param: &shareBonusFromPlayerGuid
125Clean-param: &st.error
126Clean-param: &st.email
127Clean-param: &_ref
128Clean-param: &auto
129Clean-param: &mute
130Clean-param: &hd
131Clean-param: &ad_level
132Clean-param: &rel
133Clean-param: &ya
134Clean-param: &textId
135Clean-param: &wmode
136Clean-param: &title
137Clean-param: &etext
138Clean-param: &auto_play
139Clean-param: &muted
140Clean-param: &fromTime
141Clean-param: &linknhung
142Clean-param: &mad
143Clean-param: &brandsafety
144Clean-param: &embed
145Clean-param: &hl
146Clean-param: &mt_link_id
147Clean-param: &tv
148Clean-param: &showinfo
149Clean-param: &refplace
150Clean-param: &mt_creative
151Clean-param: &mt_adset
152Clean-param: &mt_click_id
153Clean-param: &mt_network
154Clean-param: &mt_campaign
155Clean-param: &mt_sub1
156Clean-param: &mt_sub2
157Clean-param: &mt_sub3
158Clean-param: &mt_sub4
159Clean-param: &mt_sub5
160Clean-param: &__dp
161Clean-param: &ip
162Clean-param: &_cl.id
163Clean-param: &_clickLog
164Clean-param: &_cl.list
165Clean-param: &_cl.sID
166Clean-param: &just-logged-in
167Clean-param: &_sm_au_
168Clean-param: &__fd__
169Clean-param: &dc_video_web_url
170Clean-param: &__hrp
171Clean-param: &__ca__chat
172Clean-param: &_sm_byp
173Clean-param: &1
174Clean-param: &st.theme
175Clean-param: &dc_video_name
176Clean-param: &time
177Clean-param: &scrlybrkr
178Clean-param: &_om
179Clean-param: &st.author
180Clean-param: &st.hobbySlug
181Clean-param: &st.question
182Clean-param: &twoseven-iframe
183Clean-param: &mibextid
184Clean-param: &from
185Clean-param: &sfnsn
186Clean-param: &controls
187Clean-param: &modestbranding
188Clean-param: &page-ref
189Clean-param: &mref
190Clean-param: &start
191Clean-param: &feature
192Clean-param: &iv_load_policy
193Clean-param: &loop
194Clean-param: &token
195Clean-param: &autopause
196Clean-param: &white
197Clean-param: &st.layer.lg.ff
198Clean-param: &st.layer.lg.fidm
199Clean-param: &st.layer.lg.ftid
200Clean-param: &st.layer.lg.fp
201Clean-param: &st.layer.lg.flt
202Clean-param: &st.friendId
203Clean-param: &st.filterGroupId
204Clean-param: &subject
205Clean-param: "
206Clean-param: &theme
207Clean-param: &377
208Clean-param: &5682
209Clean-param: &2687
210Clean-param: &127
211Clean-param: &249
212Clean-param: &spm
213Clean-param: &1031
214Clean-param: &112
215Clean-param: &123
216Clean-param: &1150
217Clean-param: &131
218Clean-param: &1415
219Clean-param: &1492
220Clean-param: &140
221Clean-param: &1505
222Clean-param: &1584
223
224Sitemap: https://ok.ru/sitemap-index-game.xml.gz
225Sitemap: https://ok.ru/sitemap-index-gift.xml.gz
226Sitemap: https://ok.ru/sitemap-index-content-search.xml.gz
227Sitemap: https://ok.ru/sitemap-index-help.xml.gz
228Sitemap: https://ok.ru/sitemap-index-user-name.xml.gz
229Sitemap: https://ok.ru/sitemap-index-vip-profile.xml.gz
230Sitemap: https://ok.ru/sitemap-index-vip-group.xml.gz
231Sitemap: https://ok.ru/sitemap-index-profile-web.xml.gz
232Sitemap: https://ok.ru/sitemap-index-group-catalog.xml.gz
233Sitemap: https://ok.ru/sitemap-index-group.xml.gz
234Sitemap: https://ok.ru/sitemap-index-group-topic.xml.gz
235Sitemap: https://ok.ru/sitemap-index-music-artist.xml.gz
236Sitemap: https://ok.ru/sitemap-index-music-album.xml.gz
237Sitemap: https://ok.ru/sitemap-index-hobby.xml.gz
238Sitemap: https://ok.ru/sitemap-index-video.xml.gz
239Sitemap: https://ok.ru/sitemap-index-congratulation.xml.gz
240Sitemap: https://ok.ru/sitemap-index-collection-article.xml.gz
What do the bots actually get?
This page reads robots.txt, which states what ok.ru asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on ok.ru Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does ok.ru block GPTBot?
No. ok.ru’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does ok.ru block ClaudeBot?
No. ok.ru’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does ok.ru block Google-Extended?
No. ok.ru’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does ok.ru block PerplexityBot?
No. ok.ru’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find ok.ru?
Yes. ok.ru’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.