Bot directory · Search engines

Search engines · Google

Googlebot user agent

Googlebot is Google’s main web crawler, the one that builds the index behind Google Search. It identifies itself with two user agent strings, smartphone and desktop, and most of its requests come from the smartphone one. Below are the exact strings, every other Google crawler, and how to tell the real Googlebot from a scraper using its name. To test a page, use the Googlebot checker.

Checked against Google’s documentation on 10 October 2026

Operator
Google
Purpose
Search index. The crawlers behind Google, Bing and the other search engines. Blocking one drops you out of its results.
robots.txt token
Googlebot
We test with
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot’s short compatible string, above. Firewall rules and robots.txt match the Googlebot token, so they treat every Googlebot string the same way.
Documentation
developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#googlebot

The Googlebot user agent strings

Google lists these for Googlebot, its crawler for Google Search, Images, Video, News and Discover:

CrawlerUser agent string
Googlebot SmartphoneMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot DesktopMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Older forms, rarely seenMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)

Chrome/W.X.Y.Z is a placeholder. In a real request it is the version of Chrome that Googlebot renders pages with, such as Chrome/41.0.2272.96 in Google’s own example, and it rises as Google updates its rendering engine. Never match on an exact version. Match on Googlebot, or use a wildcard for the version:

grep -E "Googlebot/2\.1" /var/log/nginx/access.log

Every string carries Googlebot/2.1 and a link to google.com/bot.html. That only says what the request claims to be. Anyone can send it, so verify the address before you trust it (see below).

Smartphone or desktop?

Google indexes the mobile version of most sites (mobile-first indexing), so most Googlebot requests use the smartphone string and a minority use the desktop one. You can tell them apart by Android and Mobile Safari in the smartphone string. You can’t choose between them in robots.txt: both obey the same Googlebot token. Search Console’s Crawl stats report shows the split for your site under “Googlebot type”.

Every Google crawler and fetcher

Google sorts its crawlers into three kinds, and how each treats robots.txt depends on the kind. A robots.txt rule applies when it names any one of a crawler’s tokens.

Common crawlers: always obey robots.txt

Crawler and robots.txt tokenUser agentUsed for
Googlebot ImageGooglebot-Image, GooglebotGooglebot-Image/1.0Google Images, Discover and Search features that show images, logos and favicons
Googlebot VideoGooglebot-Video, GooglebotGooglebot-Video/1.0Video features in Search
Googlebot NewsGooglebot-News, GooglebotGooglebot’s own stringsGoogle News and the Google News app
Google StoreBotStorebot-GoogleMozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
Google Shopping
Google-InspectionToolGoogle-InspectionTool, GooglebotMozilla/5.0 (compatible; Google-InspectionTool/1.0)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0)
URL Inspection in Search Console and the Rich Results Test. No effect on Search.
GoogleOtherGoogleOtherMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36
One-off crawls for research and development. Not Search. More
GoogleOther-Image, GoogleOther-VideoGoogleOther-Image or GoogleOther-Video, GoogleOtherGoogleOther-Image/1.0
GoogleOther-Video/1.0
GoogleOther for image and video URLs
Google-CloudVertexBotGoogle-CloudVertexBot, GooglebotContains Google-CloudVertexBotCrawls that a site’s owner requests to build Vertex AI agents. No effect on Search.
Google-ExtendedGoogle-ExtendedNone: a robots.txt token onlyWhether pages may train Gemini and ground its answers. No effect on Search. More

Special-case crawlers: may ignore robots.txt

These skip the User-agent: * group, so a rule for them must name them, and Google says they may ignore robots.txt rules.

Crawler and robots.txt tokenUser agentUsed for
AdsBotAdsBot-GoogleAdsBot-Google (+http://www.google.com/adsbot.html)Google Ads checks of landing page quality
AdsBot Mobile WebAdsBot-Google-MobileMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)The same, for mobile pages
Mediapartners-GoogleMediapartners-GoogleMediapartners-Google
(Various mobile device types) (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
AdSense and Ad Manager
APIs-GoogleAPIs-GoogleAPIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)Push notifications for Google APIs
Google-SafetyNone; it ignores robots.txtGoogle-SafetyAbuse and malware checks of links posted on Google products

User-triggered fetchers: ignore robots.txt

These fetch a page because a person asked for it, so Google says they generally ignore robots.txt. They have no tokens.

FetcherUser agent (or the part that identifies it)Used for
FeedfetcherFeedFetcher-Google; (+http://www.google.com/feedfetcher.html)RSS and Atom feeds for Google News and WebSub
Google Site VerifierMozilla/5.0 (compatible; Google-Site-Verification/1.0)Search Console verification tokens
Google-AgentContains Google-AgentAgents on Google’s infrastructure that browse and act on a user’s request
Gemini NotebookContains Google-GeminiNotebook (formerly Google-NotebookLM)Pages users add as sources
Google Read AloudContains Google-Read-AloudText-to-speech. Opt out with <meta name="google" content="nopagereadaloud">, not robots.txt.
Google MessagesGoogleMessagesLink previews in chats
Google Publisher CenterGoogleProducer; (+https://developers.google.com/search/docs/crawling-indexing/google-producer)Feeds publishers supply for Google News
Google PinpointGoogle-PinpointDocuments users add to Pinpoint collections
Chrome Web StoreMozilla/5.0 (compatible; Google-CWS)URLs in Chrome extension and theme listings

From Google’s lists of common crawlers, special-case crawlers and user-triggered fetchers.

How to verify a real Googlebot

Scrapers and SEO tools often borrow Googlebot’s user agent to get past rules meant for bots. The address proves who sent a request; the user agent doesn’t. Google documents two checks, and either is enough.

1. Reverse, then forward DNS. Look up the hostname of the IP in your log. It must end in googlebot.com, google.com or googleusercontent.com. Then look up that hostname and confirm it points back to the same IP; anyone can set a reverse DNS name, but only Google can make the forward lookup agree.

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

2. Match the IP against Google’s published lists, which suit automated checks at scale:

KindReverse DNS hostnameIP list
Common crawlers (Googlebot)crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.comcommon-crawlers.json
Special-case crawlersrate-limited-proxy-***-***-***-***.google.comspecial-crawlers.json
Google’s user-triggered fetchersgoogle-proxy-***-***-***-***.google.comuser-triggered-fetchers-google.json, user-triggered-agents.json
Fetchers Google Cloud customers run***-***-***-***.gae.googleusercontent.comuser-triggered-fetchers.json

The last row is Google’s network but not Google’s crawler: Google says these addresses are used when, for example, a site hosted on Google Cloud fetches a feed for one of its own users. In February 2026 Google moved the lists from /search/apis/ipranges/ to /crawling/ipranges/; the old googlebot.json address redirects to common-crawlers.json.

Check an IP now

Paste an address from your logs. We run both checks, against Google’s and Microsoft’s current lists, and say which crawler it belongs to.

How Googlebot crawls

Test your site as Googlebot

Block Googlebot

Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt. Disallowing Googlebot removes your pages from Google Search over time. To keep Google from training Gemini on your pages while staying in Search, disallow Google-Extended instead.

User-agent: Googlebot
Disallow: /

Allow Googlebot when everything else is blocked

A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /

Does Googlebot reach your site?

The checker applies your robots.txt the way Googlebot does, then requests your page with the user agent above, alongside 29 other bots, and opens the report on Googlebot.

Questions about Googlebot

What is the Googlebot user agent string?

Googlebot Smartphone sends “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” and Googlebot Desktop sends “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36”, where W.X.Y.Z is the current Chrome version Googlebot uses. Both contain Googlebot/2.1.

What does Chrome/W.X.Y.Z mean in the Googlebot user agent?

It is a placeholder for the version of Chrome Googlebot renders pages with. Real requests carry a real version number, which rises over time, so match on “Googlebot” or use a wildcard rather than an exact version.

Is Googlebot mobile or desktop?

Both, but mostly mobile. Google indexes the mobile version of most sites, so the majority of requests come from Googlebot Smartphone. Both obey the same Googlebot token in robots.txt, so you can’t allow one and block the other there.

How do I know if a request is really from Googlebot?

Check the IP, not the user agent. A reverse DNS lookup must return a hostname ending in googlebot.com, google.com or googleusercontent.com, and a forward lookup of that hostname must return the same IP. Alternatively, match the IP against Google’s common-crawlers.json list. The verifier on this page does both.

Does Googlebot obey robots.txt?

Yes. Googlebot and Google’s other common crawlers always obey robots.txt. Special-case crawlers such as AdsBot may ignore it, and user-triggered fetchers such as Feedfetcher generally ignore it because a person asked for the page.

How do I stop Google using my site for AI training without leaving Search?

Disallow the Google-Extended token in robots.txt. It controls whether pages may be used to train Gemini and ground its answers, and Google says it does not affect inclusion or ranking in Google Search.

Also run by Google: Google-Extended, GoogleOther.

Other search engines: Bingbot, DuckDuckBot, Applebot, PetalBot, YandexBot, Baiduspider.

Build a complete file with the robots.txt generator.