Who blocks AI · #1032 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does lg.com block AI crawlers?
No. lg.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.lg.com/robots.txt · 291 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | line 5 · Allow: / under User-agent: * | |
Bingbot | Microsoft | Allowed | line 5 · Allow: / under User-agent: * |
DuckDuckBot | DuckDuckGo | Allowed | line 5 · Allow: / under User-agent: * |
Applebot | Apple | Allowed | line 5 · Allow: / under User-agent: * |
PetalBot | Huawei | Allowed | line 5 · Allow: / under User-agent: * |
YandexBot | Yandex | Allowed | line 5 · Allow: / under User-agent: * |
Baiduspider | Baidu | Allowed | line 5 · Allow: / under User-agent: * |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | line 5 · Allow: / under User-agent: * |
PerplexityBot | Perplexity | Allowed | line 5 · Allow: / under User-agent: * |
Claude-SearchBot | Anthropic | Allowed | line 5 · Allow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Allowed | line 5 · Allow: / under User-agent: * |
Amazonbot | Amazon | Allowed | line 5 · Allow: / under User-agent: * |
YouBot | You.com | Allowed | line 5 · Allow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | line 5 · Allow: / under User-agent: * |
ClaudeBot | Anthropic | Allowed | line 5 · Allow: / under User-agent: * |
Google-Extended | Allowed | line 5 · Allow: / under User-agent: * | |
CCBot | Common Crawl | Allowed | line 5 · Allow: / under User-agent: * |
Applebot-Extended | Apple | Allowed | line 5 · Allow: / under User-agent: * |
GoogleOther | Allowed | line 5 · Allow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Allowed | line 5 · Allow: / under User-agent: * |
Bytespider | ByteDance | Allowed | line 5 · Allow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | line 5 · Allow: / under User-agent: * |
Claude-User | Anthropic | Allowed | line 5 · Allow: / under User-agent: * |
Perplexity-User | Perplexity | Allowed | line 5 · Allow: / under User-agent: * |
MistralAI-User | Mistral | Allowed | line 5 · Allow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Allowed | line 5 · Allow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | line 5 · Allow: / under User-agent: * |
SemrushBot | Semrush | Allowed | line 5 · Allow: / under User-agent: * |
MJ12bot | Majestic | Allowed | line 5 · Allow: / under User-agent: * |
DotBot | Moz | Allowed | line 5 · Allow: / under User-agent: * |
View www.lg.com/robots.txt · 291 lines
1# LG.com robots.txt
2# Crawl policy is managed in the User-agent: * group.
3
4User-agent: *
5Allow: /
6Disallow: *?bvstate=pg*
7Disallow: /*nf-ec*
8Disallow: /*f-ec*
9Disallow: /*/search*?*
10Disallow: /*/*.ldwf*
11Disallow: /*/support/product-help/doc*
12Disallow: /*/support/error-pages
13Disallow: /error
14Disallow: /common/bug
15Disallow: /common/content_*
16Disallow: /common/error
17Disallow: /common/search/
18Disallow: /common/support
19Disallow: /templatedata
20Disallow: /war
21Disallow: /download/support/contactus
22Disallow: /download/support
23Disallow: /*.ajax$
24Disallow: /us/about-lg/investor-relations/
25Disallow: /*gclid
26Disallow: /us/includes/
27Disallow: /us/cms-templates/
28Disallow: /us/common/
29Disallow: /us/compare/
30Disallow: /us/sys/
31Disallow: /us/application-data
32Disallow: /us/secure
33Disallow: /us/*/*?bvdata=*
34Disallow: /us/*/*?bvroute*
35Disallow: /us/*/*?bvstate*
36Disallow: /us/campaign/*
37Disallow: /*/products/wtb*
38Disallow: /compare.lg?*
39Disallow: /*/*.swf$
40Disallow: */mylg/
41Disallow: /flash/
42Disallow: /*/*/reviews/product/*
43Disallow: /lgecs.downloadFile.ldwf?*
44Disallow: /es/HEProductExperiencePage
45Disallow: /es/HEProductExperiencePage/
46Disallow: /es/HEProductExperiencePage/tv/main
47Disallow: /*/srch*
48Disallow: /*?search=*
49Disallow: /*?bvrrp=*
50Disallow: /*_escaped_fragment*
51Disallow: /*?lgeurl=*
52Disallow: /*/response/*
53Disallow: /*/popup/*
54Disallow: /*/search.lg*
55Disallow: /au/pdfs/*
56Disallow: /nz/pdfs/*
57Disallow: /*.jsp?subCatId
58Disallow: /*.jsp?value
59Disallow: /*.jsp?contextPath
60Disallow: /*.jsp?ref_no
61Disallow: /*.jsp?s_kwcid
62Disallow: /*.jsp?cmpid
63Disallow: /jp/support/repair-warranty/locate-repair-center-detail/*
64Disallow: /jp/mobile-phones/download-page/*
65Disallow: /it/convenzioni-aziendali/corporate-benefit*
66Disallow: /de/edustore
67Disallow: /de/edustore/*
68Disallow: /de/partnerstore
69Disallow: /de/partnerstore/*
70Disallow: /au/partners
71Disallow: /au/partners/*
72Disallow: /au/small-medium-business
73Disallow: /au/small-medium-business/*
74Disallow: /br/partners
75Disallow: /br/partners/*
76Disallow: /br/small-medium-business
77Disallow: /br/small-medium-business/*
78Disallow: /ca_en/partners
79Disallow: /ca_en/partners/*
80Disallow: /ca_en/small-medium-business
81Disallow: /ca_en/small-medium-business/*
82Disallow: /ca_fr/partners
83Disallow: /ca_fr/partners/*
84Disallow: /ca_fr/small-medium-business
85Disallow: /ca_fr/small-medium-business/*
86Disallow: /in/partners
87Disallow: /in/partners/*
88Disallow: /in/small-medium-business
89Disallow: /in/small-medium-business/*
90Disallow: /it/partners
91Disallow: /it/partners/*
92Disallow: /it/small-medium-business
93Disallow: /it/small-medium-business/*
94Disallow: /mx/partners
95Disallow: /mx/partners/*
96Disallow: /mx/small-medium-business
97Disallow: /mx/small-medium-business/*
98Disallow: /nl/partners
99Disallow: /nl/partners/*
100Disallow: /nl/small-medium-business
101Disallow: /nl/small-medium-business/*
102Disallow: /pe/partners
103Disallow: /pe/partners/*
104Disallow: /pe/small-medium-business
105Disallow: /pe/small-medium-business/*
106Disallow: /pl/partners
107Disallow: /pl/partners/*
108Disallow: /pl/small-medium-business
109Disallow: /pl/small-medium-business/*
110Disallow: /sa_en/partners
111Disallow: /sa_en/partners/*
112Disallow: /sa_en/small-medium-business
113Disallow: /sa_en/small-medium-business/*
114Disallow: /th/partners
115Disallow: /th/partners/*
116Disallow: /th/small-medium-business
117Disallow: /th/small-medium-business/*
118Disallow: /tw/partners
119Disallow: /tw/partners/*
120Disallow: /tw/small-medium-business
121Disallow: /tw/small-medium-business/*
122Disallow: /uk/partners
123Disallow: /uk/partners/*
124Disallow: /uk/small-medium-business
125Disallow: /uk/small-medium-business/*
126Disallow: /vn/partners
127Disallow: /vn/partners/*
128Disallow: /vn/small-medium-business
129Disallow: /vn/small-medium-business/*
130Disallow: /portal/*
131Disallow: /lg5-common-gp/capcha/botdetectcaptcha
132Disallow: /WEB-INF/jsp/common/error/error404.jsp
133Disallow: /*?from=*
134Disallow: /*?awc=*
135Disallow: /*?category=*
136Disallow: /*?amp=*
137Disallow: /*?code=*
138Disallow: /*?cmpid=*
139Disallow: /*?sub_
140Disallow: /*?sa=*
141Disallow: /*?element=*
142Disallow: /*?model=*
143Disallow: /*?bizType=*
144Disallow: /*?cjevent=*
145Disallow: /*?csSalesCode=*
146Disallow: /*?emptyModel=*
147Disallow: /*?modelNumber=*
148Disallow: /tw/*?bannerLinkUrl=*
149Disallow: /ru/*web-survey
150Disallow: /ru/products/*.page
151Disallow: /*jcr:content
152Disallow: /*_jcr_content
153Disallow: /*jcr%3Acontent
154Disallow: /*/search*
155Disallow: /*/wtb?modelName=*
156Disallow: /*/my-support
157Disallow: /*/mylg-login
158Disallow: /*?cmpid
159Disallow: /*/player?videoId=
160Disallow: /*/embedded-player?videoId=
161Disallow: /adm
162Disallow: /Admin
163Disallow: /*locate-repair-center-detail/
164Disallow: /search-all?*
165
166User-agent: Mediapartners-Google
167Allow: /*.jsp?cmpid
168
169User-agent: AdsBot-Google
170Allow: /*.jsp?cmpid
171
172User-agent: AdsBot-Google-Mobile
173Allow: /*.jsp?cmpid
174
175Sitemap: https://www.lg.com/sitemap.xml
176
177# LLM-friendly content guide for GEO/AEO (Hub file)
178LLM: https://www.lg.com/llms.txt
179
180# Yandex
181Host: www.lg.com
182Clean-param: utm_source&utm_medium&utm_term&utm_campaign&utm_content /
183Clean-param: gclid /
184Clean-param: countryCd /*.jsp
185Clean-param: subCatId /*.jsp
186Clean-param: value /*.jsp
187Clean-param: contextPath /*.jsp
188Clean-param: ref_no /*.jspA
189Clean-param: s_kwcid /*.jsp
190Clean-param: cmpid /*.jsp
191Clean-param: element&model&bizType&cjevent&csSalesCode
192Clean-param: modelid&ref&code&&awc&from&category&cmpid&emptyModel&id /
193Clean-param: elqat
194Clean-param: _gl
195Clean-param: _lang
196Clean-param: action
197Clean-param: adobeSearchType
198Clean-param: bannerLinkUrl
199Clean-param: bannerOmniture
200Clean-param: blogger
201Clean-param: bvdata
202Clean-param: bvroute
203Clean-param: bvstate
204Clean-param: c
205Clean-param: categoryId
206Clean-param: categoryName
207Clean-param: componentID
208Clean-param: countryCode
209Clean-param: countryName
210Clean-param: csB2bFlag
211Clean-param: customerModelCode
212Clean-param: disableGlobalInfoCollect
213Clean-param: downTime
214Clean-param: ds_rl
215Clean-param: elqah
216Clean-param: elqaid
217Clean-param: elqTrackId
218Clean-param: emptyModel
219Clean-param: etext
220Clean-param: feed
221Clean-param: fromApp
222Clean-param: gbraid
223Clean-param: gclsrc
224Clean-param: gtm_debug
225Clean-param: hashTag
226Clean-param: headerTitle
227Clean-param: hl
228Clean-param: initialTab
229Clean-param: keyword
230Clean-param: lgcom_source
231Clean-param: limit
232Clean-param: linkId
233Clean-param: m
234Clean-param: menu
235Clean-param: modelCode
236Clean-param: modelName
237Clean-param: modelNum
238Clean-param: modelNumber
239Clean-param: msclkid
240Clean-param: name
241Clean-param: nm
242Clean-param: nofire
243Clean-param: nx
244Clean-param: obsBuynowFlag
245Clean-param: obsOnly
246Clean-param: oldTerm
247Clean-param: omniture
248Clean-param: ort
249Clean-param: p
250Clean-param: q
251Clean-param: quickMenuSuperCategoryId
252Clean-param: r
253Clean-param: realSearch
254Clean-param: rearr-factors
255Clean-param: receiptNumber
256Clean-param: redirectCheckUrl
257Clean-param: reSearch
258Clean-param: returnFlag
259Clean-param: rev
260Clean-param: s
261Clean-param: salesModel
262Clean-param: search
263Clean-param: searchChck
264Clean-param: searchResultFlag
265Clean-param: selectItem
266Clean-param: sid
267Clean-param: siteType
268Clean-param: sort
269Clean-param: source
270Clean-param: spm
271Clean-param: subcategory
272Clean-param: subCateId
273Clean-param: submit
274Clean-param: superCategoryId
275Clean-param: superCateId
276Clean-param: surveyPageChangeYn
277Clean-param: target
278Clean-param: tpclid
279Clean-param: twclid
280Clean-param: type
281Clean-param: u
282Clean-param: url
283Clean-param: viewSalesCode
284Clean-param: visit_id
285Clean-param: wbraid
286Clean-param: yandex_ad_client_id
287Clean-param: tabType&clickref¤tPage&sku&sub_id
288Clean-param: n_source&n_campaign&n_medium&n_content&success&social&sa&ved&DOC_ID&what&fromSystem&fileId&ORIGINAL_NAME_b1_a1&sqi&back&utm_channel&ranMID&ranEAID&ranSiteID&LSNSUBSITE&2599630&3488388&sv1&frm&sv_campaign&ppi&ppa&qwe&sdf&rtp&v
289Clean-param: sv_campaign_id&aku&elqak
290Clean-param: room&product&interior&simulator&popup
291
What do the bots actually get?
This page reads robots.txt, which states what lg.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on lg.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does lg.com block GPTBot?
No. lg.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does lg.com block ClaudeBot?
No. lg.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does lg.com block Google-Extended?
No. lg.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does lg.com block PerplexityBot?
No. lg.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find lg.com?
Yes. lg.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.