Search engines · Google

Googlebot

Google’s main web crawler. It builds the index behind Google Search, and its fetches also feed AI Overviews unless Google-Extended is disallowed.

Operator: Google
Purpose: Search index. The crawlers behind Google, Bing and the other search engines. Blocking one drops you out of its results.
robots.txt token: Googlebot
User agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Documentation: developers.google.com/search/docs/crawling-indexing/overview-google-crawlers

Block Googlebot

Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt.

User-agent: Googlebot
Disallow: /

Allow Googlebot when everything else is blocked

A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /

Does Googlebot reach your site?

The checker applies your robots.txt the way Googlebot does, then requests your page with the user agent above, alongside 29 other bots.

Also run by Google: Google-Extended, GoogleOther.

Other search engines: Bingbot, DuckDuckBot, Applebot, PetalBot, YandexBot, Baiduspider.

Build a complete file with the robots.txt generator.