Bot directory · Search engines
Googlebot user agent
Googlebot is Google’s main web crawler, the one that builds the index behind Google Search. It identifies itself with two user agent strings, smartphone and desktop, and most of its requests come from the smartphone one. Below are the exact strings, every other Google crawler, and how to tell the real Googlebot from a scraper using its name. To test a page, use the Googlebot checker.
- Operator
- Purpose
- Search index. The crawlers behind Google, Bing and the other search engines. Blocking one drops you out of its results.
- robots.txt token
Googlebot- We test with
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot’s short compatible string, above. Firewall rules and robots.txt match theGooglebottoken, so they treat every Googlebot string the same way.
The Googlebot user agent strings
Google lists these for Googlebot, its crawler for Google Search, Images, Video, News and Discover:
| Crawler | User agent string |
|---|---|
| Googlebot Smartphone | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| Googlebot Desktop | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36 |
| Older forms, rarely seen | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Googlebot/2.1 (+http://www.google.com/bot.html) |
Chrome/W.X.Y.Z is a placeholder. In a real request it is the version of Chrome that Googlebot renders pages with, such as Chrome/41.0.2272.96 in Google’s own example, and it rises as Google updates its rendering engine. Never match on an exact version. Match on Googlebot, or use a wildcard for the version:
grep -E "Googlebot/2\.1" /var/log/nginx/access.log
Every string carries Googlebot/2.1 and a link to google.com/bot.html. That only says what the request claims to be. Anyone can send it, so verify the address before you trust it (see below).
Smartphone or desktop?
Google indexes the mobile version of most sites (mobile-first indexing), so most Googlebot requests use the smartphone string and a minority use the desktop one. You can tell them apart by Android and Mobile Safari in the smartphone string. You can’t choose between them in robots.txt: both obey the same Googlebot token. Search Console’s Crawl stats report shows the split for your site under “Googlebot type”.
Every Google crawler and fetcher
Google sorts its crawlers into three kinds, and how each treats robots.txt depends on the kind. A robots.txt rule applies when it names any one of a crawler’s tokens.
Common crawlers: always obey robots.txt
| Crawler and robots.txt token | User agent | Used for |
|---|---|---|
Googlebot ImageGooglebot-Image, Googlebot | Googlebot-Image/1.0 | Google Images, Discover and Search features that show images, logos and favicons |
Googlebot VideoGooglebot-Video, Googlebot | Googlebot-Video/1.0 | Video features in Search |
Googlebot NewsGooglebot-News, Googlebot | Googlebot’s own strings | Google News and the Google News app |
Google StoreBotStorebot-Google | Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 | Google Shopping |
Google-InspectionToolGoogle-InspectionTool, Googlebot | Mozilla/5.0 (compatible; Google-InspectionTool/1.0)Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0) | URL Inspection in Search Console and the Rich Results Test. No effect on Search. |
GoogleOtherGoogleOther | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther)Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36 | One-off crawls for research and development. Not Search. More |
GoogleOther-Image, GoogleOther-VideoGoogleOther-Image or GoogleOther-Video, GoogleOther | GoogleOther-Image/1.0GoogleOther-Video/1.0 | GoogleOther for image and video URLs |
Google-CloudVertexBotGoogle-CloudVertexBot, Googlebot | Contains Google-CloudVertexBot | Crawls that a site’s owner requests to build Vertex AI agents. No effect on Search. |
Google-ExtendedGoogle-Extended | None: a robots.txt token only | Whether pages may train Gemini and ground its answers. No effect on Search. More |
Special-case crawlers: may ignore robots.txt
These skip the User-agent: * group, so a rule for them must name them, and Google says they may ignore robots.txt rules.
| Crawler and robots.txt token | User agent | Used for |
|---|---|---|
AdsBotAdsBot-Google | AdsBot-Google (+http://www.google.com/adsbot.html) | Google Ads checks of landing page quality |
AdsBot Mobile WebAdsBot-Google-Mobile | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) | The same, for mobile pages |
Mediapartners-GoogleMediapartners-Google | Mediapartners-Google(Various mobile device types) (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html) | AdSense and Ad Manager |
APIs-GoogleAPIs-Google | APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html) | Push notifications for Google APIs |
| Google-SafetyNone; it ignores robots.txt | Google-Safety | Abuse and malware checks of links posted on Google products |
User-triggered fetchers: ignore robots.txt
These fetch a page because a person asked for it, so Google says they generally ignore robots.txt. They have no tokens.
| Fetcher | User agent (or the part that identifies it) | Used for |
|---|---|---|
| Feedfetcher | FeedFetcher-Google; (+http://www.google.com/feedfetcher.html) | RSS and Atom feeds for Google News and WebSub |
| Google Site Verifier | Mozilla/5.0 (compatible; Google-Site-Verification/1.0) | Search Console verification tokens |
| Google-Agent | Contains Google-Agent | Agents on Google’s infrastructure that browse and act on a user’s request |
| Gemini Notebook | Contains Google-GeminiNotebook (formerly Google-NotebookLM) | Pages users add as sources |
| Google Read Aloud | Contains Google-Read-Aloud | Text-to-speech. Opt out with <meta name="google" content="nopagereadaloud">, not robots.txt. |
| Google Messages | GoogleMessages | Link previews in chats |
| Google Publisher Center | GoogleProducer; (+https://developers.google.com/search/docs/crawling-indexing/google-producer) | Feeds publishers supply for Google News |
| Google Pinpoint | Google-Pinpoint | Documents users add to Pinpoint collections |
| Chrome Web Store | Mozilla/5.0 (compatible; Google-CWS) | URLs in Chrome extension and theme listings |
From Google’s lists of common crawlers, special-case crawlers and user-triggered fetchers.
How to verify a real Googlebot
Scrapers and SEO tools often borrow Googlebot’s user agent to get past rules meant for bots. The address proves who sent a request; the user agent doesn’t. Google documents two checks, and either is enough.
1. Reverse, then forward DNS. Look up the hostname of the IP in your log. It must end in googlebot.com, google.com or googleusercontent.com. Then look up that hostname and confirm it points back to the same IP; anyone can set a reverse DNS name, but only Google can make the forward lookup agree.
$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
2. Match the IP against Google’s published lists, which suit automated checks at scale:
| Kind | Reverse DNS hostname | IP list |
|---|---|---|
| Common crawlers (Googlebot) | crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com | common-crawlers.json |
| Special-case crawlers | rate-limited-proxy-***-***-***-***.google.com | special-crawlers.json |
| Google’s user-triggered fetchers | google-proxy-***-***-***-***.google.com | user-triggered-fetchers-google.json, user-triggered-agents.json |
| Fetchers Google Cloud customers run | ***-***-***-***.gae.googleusercontent.com | user-triggered-fetchers.json |
The last row is Google’s network but not Google’s crawler: Google says these addresses are used when, for example, a site hosted on Google Cloud fetches a feed for one of its own users. In February 2026 Google moved the lists from /search/apis/ipranges/ to /crawling/ipranges/; the old googlebot.json address redirects to common-crawlers.json.
Check an IP now
Paste an address from your logs. We run both checks, against Google’s and Microsoft’s current lists, and say which crawler it belongs to.
How Googlebot crawls
- Size limit. For Google Search, Googlebot reads the first 2 MB of an HTML or other supported text file and the first 64 MB of a PDF, measured uncompressed. CSS, JavaScript and other resources are fetched separately, each under the same limit. Content past the cut-off is not indexed.
- robots.txt. Google generally caches it for up to 24 hours, reads at most 500 KiB of it, and treats a server error on it as a reason to stop crawling the site. The robots.txt report in Search Console can request a fresh fetch.
- Rate. For most sites, Googlebot shouldn’t request more than once every few seconds on average. To slow it in an emergency, answer
503or429for a day or two at most; you can’t ask Google to crawl faster. - Protocol. HTTP/1.1 by default, HTTP/2 when that saves resources (with no ranking benefit), and gzip, deflate or Brotli compression.
- Location. Mostly from addresses in the United States, in Pacific Time, without an
Accept-Languageheader. If a site blocks the US, Google may try from other countries. A country block that includes the US is a common way to lose Googlebot. - Rendering. Googlebot runs JavaScript in an up-to-date Chromium, which is what the
Chrome/W.X.Y.Zin its user agent reports. Blocking your CSS or JavaScript files in robots.txt stops it rendering the page properly.
Test your site as Googlebot
- The Googlebot checker applies your robots.txt the way Googlebot does, requests the page with Googlebot’s user agent and reads its noindex signals, alongside 29 other bots.
- URL Inspection in Search Console, with Test live URL, fetches the page right now from Google’s own addresses as Google-InspectionTool and shows the rendered result. It is the definitive test, limited to sites you’ve verified and to a daily number of inspections.
- The robots.txt report in Search Console shows the robots.txt Google last fetched for each host, and any fetch errors.
- curl shows what your server answers to the user agent, from your own address:
curl -I -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://your-site.com/. Firewalls that verify Googlebot by IP will treat this differently from the real thing.
Block Googlebot
Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt. Disallowing Googlebot removes your pages from Google Search over time. To keep Google from training Gemini on your pages while staying in Search, disallow Google-Extended instead.
User-agent: Googlebot
Disallow: /
Allow Googlebot when everything else is blocked
A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /
Does Googlebot reach your site?
The checker applies your robots.txt the way Googlebot does, then requests your page with the user agent above, alongside 29 other bots, and opens the report on Googlebot.
Questions about Googlebot
What is the Googlebot user agent string?
Googlebot Smartphone sends “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” and Googlebot Desktop sends “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36”, where W.X.Y.Z is the current Chrome version Googlebot uses. Both contain Googlebot/2.1.
What does Chrome/W.X.Y.Z mean in the Googlebot user agent?
It is a placeholder for the version of Chrome Googlebot renders pages with. Real requests carry a real version number, which rises over time, so match on “Googlebot” or use a wildcard rather than an exact version.
Is Googlebot mobile or desktop?
Both, but mostly mobile. Google indexes the mobile version of most sites, so the majority of requests come from Googlebot Smartphone. Both obey the same Googlebot token in robots.txt, so you can’t allow one and block the other there.
How do I know if a request is really from Googlebot?
Check the IP, not the user agent. A reverse DNS lookup must return a hostname ending in googlebot.com, google.com or googleusercontent.com, and a forward lookup of that hostname must return the same IP. Alternatively, match the IP against Google’s common-crawlers.json list. The verifier on this page does both.
Does Googlebot obey robots.txt?
Yes. Googlebot and Google’s other common crawlers always obey robots.txt. Special-case crawlers such as AdsBot may ignore it, and user-triggered fetchers such as Feedfetcher generally ignore it because a person asked for the page.
How do I stop Google using my site for AI training without leaving Search?
Disallow the Google-Extended token in robots.txt. It controls whether pages may be used to train Gemini and ground its answers, and Google says it does not affect inclusion or ranking in Google Search.
Also run by Google: Google-Extended, GoogleOther.
Other search engines: Bingbot, DuckDuckBot, Applebot, PetalBot, YandexBot, Baiduspider.
Build a complete file with the robots.txt generator.