Search engines · Yandex

YandexBot

The main crawler of Yandex, the largest search engine in Russia.

Operator: Yandex
Purpose: Search index. The crawlers behind Google, Bing and the other search engines. Blocking one drops you out of its results.
robots.txt token: YandexBot
User agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Documentation: yandex.com/support/webmaster/robot-workings/check-yandex-robots.html

Block YandexBot

Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt.

User-agent: YandexBot
Disallow: /

Allow YandexBot when everything else is blocked

A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.

User-agent: YandexBot
Allow: /

User-agent: *
Disallow: /

Does YandexBot reach your site?

The checker applies your robots.txt the way YandexBot does, then requests your page with the user agent above, alongside 29 other bots.

Other search engines: Googlebot, Bingbot, DuckDuckBot, Applebot, PetalBot, Baiduspider.

Build a complete file with the robots.txt generator.