Who blocks AI · #1 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does google.com block AI crawlers?
No. google.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.google.com/robots.txt · 265 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.google.com/robots.txt · 265 lines
1User-agent: *
2User-agent: Yandex
3Disallow: /search
4Allow: /search/about
5Allow: /search/howsearchworks
6Disallow: /sdch
7Disallow: /groups
8Disallow: /index.html?
9Disallow: /?
10Disallow: /goto?
11Allow: /?hl=
12Disallow: /?hl=*&
13Allow: /?hl=*&gws_rd=ssl$
14Disallow: /?hl=*&*&gws_rd=ssl
15Allow: /?gws_rd=ssl$
16Allow: /?pt1=true$
17Disallow: /imgres
18Disallow: /u/
19Disallow: /setprefs
20Disallow: /m?
21Disallow: /m/
22Allow: /m/finance
23Disallow: /wml?
24Disallow: /wml/?
25Disallow: /wml/search?
26Disallow: /xhtml?
27Disallow: /xhtml/?
28Disallow: /xhtml/search?
29Disallow: /xml?
30Disallow: /imode?
31Disallow: /imode/?
32Disallow: /imode/search?
33Disallow: /jsky?
34Disallow: /jsky/?
35Disallow: /jsky/search?
36Disallow: /pda?
37Disallow: /pda/?
38Disallow: /pda/search?
39Disallow: /local?
40Disallow: /local_url
41Disallow: /products?
42Disallow: /product_
43Disallow: /products_
44Disallow: /products;
45Disallow: /print
46Disallow: /books/
47Disallow: /bkshp?*dq=
48Disallow: /bkshp?*q=
49Disallow: /books?*dq=
50Disallow: /books?*q=
51Disallow: /books?*qtid=
52Disallow: /books?*output=
53Disallow: /books?*pg=
54Disallow: /books?*jtp=
55Disallow: /books?*jscmd=
56Disallow: /books?*buy=
57Disallow: /books?*zoom=
58Allow: /books/about
59Allow: /books?*zoom=1
60Allow: /books?*zoom=5
61Allow: /books/content?*zoom=1
62Allow: /books/content?*zoom=5
63Disallow: /patents?
64Disallow: /patents/download/
65Disallow: /patents/pdf/
66Disallow: /patents/related/
67Disallow: /scholar
68Disallow: /citations?
69Allow: /citations?user=
70Allow: /citations?view_op=new_profile
71Allow: /citations?view_op=top_venues
72Allow: /scholar_share
73Disallow: /s?
74Disallow: /maps?
75Allow: /maps?daddr=
76Allow: /maps?entry=wc
77Allow: /maps?f=
78Allow: /maps?hl=
79Allow: /maps?q=
80Allow: /maps?saddr=
81Allow: /maps?sid=
82Allow: /maps?*output=classic
83Allow: /maps?*file=
84Disallow: /mapslt?
85Disallow: /maphp?
86Disallow: /maps/
87Allow: /maps/$
88Allow: /maps/@
89Allow: /maps/?daddr=
90Allow: /maps/?entry=wc
91Allow: /maps/?f=
92Allow: /maps/?hl=
93Allow: /maps/?q=
94Allow: /maps/?saddr=
95Allow: /maps/?sid=
96Allow: /maps/api/staticmap?
97Allow: /maps/search/
98Allow: /maps/sitemap.xml
99Allow: /maps/sitemaps/
100Allow: /maps/dir/
101Allow: /maps/d/
102Allow: /maps/reserve
103Allow: /maps/about
104Allow: /maps/contrib/
105Allow: /maps/match
106Allow: /maps/ms?
107Allow: /maps/place/
108Allow: /maps/_/
109Allow: /search?*tbm=map
110Allow: /maps/vt?
111Allow: /maps/preview
112Disallow: /maps/api/js/
113Allow: /maps/api/js
114Disallow: /mld?
115Disallow: /staticmap?
116Disallow: /help/maps/streetview/partners/welcome/
117Disallow: /help/maps/indoormaps/partners/
118Disallow: /lochp?
119Disallow: /ie?
120Disallow: /uds/
121Disallow: /transit?
122Disallow: /trends?
123Disallow: /trends/music?
124Disallow: /trends/hottrends?
125Disallow: /trends/viz?
126Disallow: /trends/embed.js?
127Disallow: /trends/fetchComponent?
128Disallow: /trends/beta
129Disallow: /trends/topics
130Disallow: /trends/explore?
131Disallow: /trends/api
132Disallow: /musica
133Disallow: /musicl
134Disallow: /musics
135Disallow: /urchin_test/
136Disallow: /movies?
137Disallow: /wapsearch?
138Disallow: /reviews/search?
139Disallow: /cbk
140Disallow: /profiles/me
141Disallow: /s2/profiles/me
142Allow: /s2/profiles
143Allow: /s2/oz
144Allow: /s2/photos
145Allow: /s2/search/social
146Allow: /s2/static
147Disallow: /s2
148Disallow: /transconsole/portal/
149Disallow: /aclk
150Disallow: /tbproxy/
151Disallow: /support/forum/search?
152Disallow: /reviews/polls/
153Disallow: /hosted/images/
154Disallow: /accounts/ClientLogin
155Disallow: /accounts/ClientAuth
156Disallow: /accounts/o8
157Allow: /accounts/o8/id
158Disallow: /quality_form?
159Disallow: /labs/popgadget/search
160Disallow: /compressiontest/
161Disallow: /analytics/feeds/
162Disallow: /analytics/partners/comments/
163Disallow: /analytics/portal/
164Disallow: /analytics/uploads/
165Allow: /alerts/manage
166Allow: /alerts/remove
167Disallow: /alerts/
168Allow: /alerts/$
169Disallow: /phone/compare/?
170Disallow: /travel/clk
171Disallow: /travel/lodging/clk
172Disallow: /travel/lodging/imp
173Disallow: /travel/entity
174Disallow: /travel/search
175Disallow: /travel/flights/booking
176Disallow: /travel/flights/s/
177Disallow: /travel/flights/search
178Disallow: /travel/hotels/stories
179Disallow: /travel/hotels/*/stories
180Disallow: /travel/story
181Disallow: /hotelfinder/rpc
182Disallow: /hotels/rpc
183Disallow: /evaluation/
184Disallow: /forms/perks/
185Disallow: /shopping/suppliers/search
186Disallow: /edu/cs4hs/
187Disallow: /trustedstores/s/
188Disallow: /trustedstores/tm2
189Disallow: /trustedstores/verify
190Disallow: /shopping?
191Disallow: /shopping/product/
192Disallow: /shopping/seller
193Disallow: /shopping/ratings/account/metrics
194Disallow: /shopping/ratings/merchant/immersivedetails
195Disallow: /shopping/reviewer
196Disallow: /shopping/search
197Disallow: /shopping/deals
198Allow: /shopping?udm=28$
199Disallow: /storefront
200Disallow: /storepicker
201Disallow: /about/careers/applications/candidate-prep
202Disallow: /about/careers/applications/connect-with-a-googler
203Disallow: /about/careers/applications/jobs/results?page=
204Disallow: /about/careers/applications/jobs/results/?page=
205Disallow: /about/careers/applications/jobs/results?*&page=
206Disallow: /about/careers/applications/jobs/results/?*&page=
207Disallow: /landing/signout.html
208Disallow: /gallery/
209Disallow: /landing/now/ontap/
210Allow: /maps/reserve
211Allow: /maps/reserve/partners
212Disallow: /maps/reserve/api/
213Disallow: /maps/reserve/search
214Disallow: /maps/reserve/bookings
215Disallow: /maps/reserve/settings
216Disallow: /maps/reserve/manage
217Disallow: /maps/reserve/payment
218Disallow: /maps/reserve/receipt
219Disallow: /maps/reserve/sellersignup
220Disallow: /maps/reserve/feedback
221Disallow: /maps/reserve/terms
222Disallow: /maps/reserve/m/
223Disallow: /maps/reserve/b/
224Disallow: /maps/reserve/partner-dashboard
225Disallow: /local/cars
226Disallow: /local/dealership/
227Disallow: /local/dining/
228Disallow: /local/place/products/
229Disallow: /local/place/reviews/
230Disallow: /local/place/rap/
231Disallow: /local/tab/
232Disallow: /localservices/
233Allow: /localservices/coupon/
234Disallow: /nonprofits/account/
235Disallow: /uviewer
236Disallow: /landing/cmsnext-root/
237
238# AdsBot
239User-agent: AdsBot-Google
240Disallow: /maps/api/js/
241Allow: /maps/api/js
242Disallow: /maps/api/place/js/
243Disallow: /maps/api/staticmap
244Disallow: /maps/api/streetview
245
246# New user agent groups must also have a user agent reference in the global (*)
247# group. See "Order of precedence" section in
248# https://goo.gle/rep#order-of-precedence-for-user-agents
249User-agent: Yandex
250Disallow: /about/careers/applications/jobs/results
251Disallow: /about/careers/applications-a/jobs/results
252
253# Crawlers of certain social media sites are allowed to access page markup when
254# google.com/imgres* links are shared. To learn more, please contact
255# images-robots-allowlist@google.com.
256User-agent: facebookexternalhit
257User-agent: Twitterbot
258Allow: /imgres
259Allow: /search
260Disallow: /groups
261Disallow: /hosted/images/
262Disallow: /m/
263
264Sitemap: https://www.google.com/sitemap.xml
265
What do the bots actually get?
This page reads robots.txt, which states what google.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on google.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does google.com block GPTBot?
No. google.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does google.com block ClaudeBot?
No. google.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does google.com block Google-Extended?
No. google.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does google.com block PerplexityBot?
No. google.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find google.com?
Yes. google.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.