Who blocks AI · #969 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does att.com block AI crawlers?
No. att.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.att.com/robots.txt · 390 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 223 · Disallow: /gtm/ line 227 · Disallow: /search/ and 113 more | |
Bingbot | Microsoft | Allowed | line 8 · Allow: / under User-agent: * |
DuckDuckBot | DuckDuckGo | Allowed | line 8 · Allow: / under User-agent: * |
Applebot | Apple | Partly blocked | line 223 · Disallow: /gtm/ line 227 · Disallow: /search/ and 113 more |
PetalBot | Huawei | Allowed | line 8 · Allow: / under User-agent: * |
YandexBot | Yandex | Allowed | line 8 · Allow: / under User-agent: * |
Baiduspider | Baidu | Allowed | line 8 · Allow: / under User-agent: * |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | line 8 · Allow: / under User-agent: * |
PerplexityBot | Perplexity | Allowed | line 8 · Allow: / under User-agent: * |
Claude-SearchBot | Anthropic | Allowed | line 8 · Allow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Allowed | line 8 · Allow: / under User-agent: * |
Amazonbot | Amazon | Allowed | line 8 · Allow: / under User-agent: * |
YouBot | You.com | Allowed | line 8 · Allow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | line 8 · Allow: / under User-agent: * |
ClaudeBot | Anthropic | Allowed | line 8 · Allow: / under User-agent: * |
Google-Extended | Allowed | line 8 · Allow: / under User-agent: * | |
CCBot | Common Crawl | Allowed | line 8 · Allow: / under User-agent: * |
Applebot-Extended | Apple | Allowed | line 8 · Allow: / under User-agent: * |
GoogleOther | Allowed | line 8 · Allow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Allowed | line 8 · Allow: / under User-agent: * |
Bytespider | ByteDance | Allowed | line 8 · Allow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | line 8 · Allow: / under User-agent: * |
Claude-User | Anthropic | Allowed | line 8 · Allow: / under User-agent: * |
Perplexity-User | Perplexity | Allowed | line 8 · Allow: / under User-agent: * |
MistralAI-User | Mistral | Allowed | line 8 · Allow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Allowed | line 8 · Allow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | line 8 · Allow: / under User-agent: * |
SemrushBot | Semrush | Allowed | line 8 · Allow: / under User-agent: * |
MJ12bot | Majestic | Allowed | line 8 · Allow: / under User-agent: * |
DotBot | Moz | Blocked | line 3 · Disallow: / under User-agent: DotBot |
View www.att.com/robots.txt · 390 lines
1User-agent: DotBot
2User-agent: dotbot
3Disallow: /
4
5
6User-agent: *
7Content-Signal: search=yes, ai-input=yes, ai-train=no
8Allow: /
9
10# Public AI-agent discovery resources
11Allow: /.well-known/agent-card.json
12Allow: /.well-known/mcp/server-card.json
13Allow: /.well-known/agent-skills/index.json
14Allow: /.well-known/ai-catalog.json
15Allow: /.well-known/api-catalog
16Allow: /ai-instructions/shopping-assistant.md
17Allow: /ai-instructions/build-a-plan.md
18Allow: /ai-instructions/build-a-plan.html
19Allow: /ai.txt
20Allow: /llms.txt
21Allow: /llms-full.txt
22Allow: /trust.txt
23Allow: /openapi.json
24
25Disallow: /wireless/switch-and-save-b/
26Disallow: /redressG/
27Disallow: /redressGV/
28Disallow: /redressH/
29Disallow: /redressT/
30Disallow: /redressC/
31
32# edits 03.04.25 - end
33
34# edits 10.20.26 - start
35Disallow: /smart-home-manager/*
36# edits 10.20.26 - end
37
38# edits 07.29.25 - start
39
40# Allow exact homepage
41Allow: /es-us/$
42
43# Allow exact folder roots only (no subfolders/files)
44
45Allow: /es-us/local/$
46Allow: /es-us/local/internet/$
47Allow: /es-us/local/fiber/$
48Allow: /es-us/local/cell-phones/$
49Allow: /es-us/local/bundles/$
50Allow: /es-us/stores/$
51
52# Block all subfolders and files inside these folders
53
54Disallow: /es-us/local/*
55Disallow: /es-us/local/internet/*
56Disallow: /es-us/local/fiber/*
57Disallow: /es-us/local/cell-phones/*
58Disallow: /es-us/local/bundles/*
59Disallow: /es-us/stores/*
60
61# edits 07.29.25 - end
62
63# edits 09.24.25 - start
64
65Disallow: /ssaf/ssafc/v1/controllerdata
66
67# edits 09.24.25 - end
68
69# edits 02.09.25 - start
70
71Disallow: /buy/login
72
73# edits 02.09.25 - end
74
75# edits 05.01.24 - start
76
77Disallow: /search/
78Disallow: /cfd/
79Disallow: /*bvstate=pg*
80Disallow: /*/rest/clicktrack?*
81Disallow: /*/rest/getlist?*
82Disallow: /buy/postpaid/line-selection
83Disallow: /ssaf/ssafc/
84Disallow: /att-dgt-checkin-e/api/*
85Disallow: /*/sdattservicesinc/*
86
87# edits 01.25.25 - end
88
89Disallow: /gtm/
90
91# Consumer Wireless and Home
92
93Disallow: /support/showarticle*
94Disallow: /contactus/*
95Disallow: /esupport/index.html*
96Disallow: /esupport/index.jsp*
97Disallow: /esupport/article.html*
98Disallow: /esupport/article.jsp*
99Disallow: /esupport/main.html*
100Disallow: /esupport/main.jsp*
101Disallow: /esupport/u-verse-tv-troubleshooting/*
102Disallow: /esupport/billAndAccount.jsp*
103Disallow: /esupport/security.jsp*
104Disallow: /esupport/postmaster*
105Disallow: /esupport/online-services.jsp*
106Disallow: /esupport/smbindex.jsp*
107Disallow: /esupport/report-call-or-text.html*
108Disallow: /esupport/speed-calculator*
109Disallow: /esupport/data-calculator*
110Disallow: /OLAM_PROD_CMS/*
111Disallow: /*EDOCSSESSIONID*
112Disallow: /*SESSIONID*
113Disallow: /*es-us/esupport/article.jsp?pagebody*
114Disallow: /*es/legalterms.html
115Disallow: /*rex/recommendation/*
116Disallow: /Common/maintenance/maintenance.html
117Disallow: /Common/popup/
118Disallow: /Common/supern/
119Disallow: /Large-Files/
120Disallow: /apis/maps/v2/locator/search/query.json
121Disallow: /apis/maps/v2/locator/search/viewport.json
122Disallow: /billinserts/
123Disallow: /dsl/modals/
124Disallow: /dsl/shop/fragments/
125Disallow: /buy/broadband
126Disallow: /buy/directv
127Disallow: /buy/cart
128Disallow: /buy/wireless/
129Disallow: /buy/wireless/devices
130Disallow: /buy/byod/
131Disallow: /buy/byod/sim-selection/
132Disallow: /buy/bundles
133Disallow: /esupport/ref/showarticle.jsp*
134Disallow: /esupport/sender/
135Disallow: /esupport/sender/browseResults.jsp
136Disallow: /esupport/solutionDisplay.jsp
137Disallow: /gen/press-room?pid=19457&cdvn=news&newsarticleid=31735
138Disallow: /maintenance/maintenance.html
139Disallow: /maps/not-available.html
140Disallow: /privacy/policy/
141Disallow: /global-search/search*
142Disallow: /mktg/
143
144Allow: /mktg/v1/_next/
145
146Disallow: /olam/passthroughAction.myworld*
147Disallow: /olam/loginAction.olamexecute
148Disallow: /olam/IdentityFailureAction.olamexecute*
149Disallow: /olam/validateAccountForgotIdSlid.myworld
150Disallow: /olam/jsp/login/inProgress.jsp*
151
152# Small Business
153
154Disallow: /smallbusiness/404.jsp?
155Disallow: /smallbusiness/b?
156Disallow: /smallbusiness/search
157Disallow: /smallbusiness/wireless/buyFlowContainer.jsp
158Disallow: /smallbusiness/i9nvalidParamInput.jsp
159Disallow: /es-us/OLAM_PROD_CMS/*
160Disallow: /es-us/*EDOCSSESSIONID*
161Disallow: /es-us/*SESSIONID*
162Disallow: /es-us/*es-us/esupport/article.jsp?pagebody*
163Disallow: /es-us/*es/legalterms.html
164Disallow: /es-us/*rex/recommendation/*
165Disallow: /es-us/Common/maintenance/maintenance.html
166Disallow: /es-us/Common/popup/
167Disallow: /es-us/Common/supern/
168Disallow: /es-us/Large-Files/
169Disallow: /es-us/apis/maps/v2/locator/search/query.json
170Disallow: /es-us/apis/maps/v2/locator/search/viewport.json
171Disallow: /es-us/billinserts/
172Disallow: /es-us/dsl/modals/
173Disallow: /es-us/dsl/shop/fragments/
174Disallow: /es-us/buy/broadband
175Disallow: /es-us/buy/directv
176Disallow: /es-us/buy/bundles
177
178Allow: /es-us/buy/bundles?product_suite=NBB
179Allow: /es-us/mktg/v1/_next/
180
181Disallow: /es-us/buy/cart
182Disallow: /es-us/esupport/ref/showarticle.jsp*
183Disallow: /es-us/esupport/sender/
184Disallow: /es-us/esupport/sender/browseResults.jsp
185Disallow: /es-us/esupport/solutionDisplay.jsp
186Disallow: /es-us/gen/press-room?pid=19457&cdvn=news&newsarticleid=31735
187Disallow: /es-us/maintenance/maintenance.html
188Disallow: /es-us/maps/not-available.html
189Disallow: /es-us/privacy/policy/
190Disallow: /es-us/global-search/search*
191Disallow: /es-us/mktg/
192Disallow: /es-us/olam/passthroughAction.myworld*
193Disallow: /es-us/olam/loginAction.olamexecute
194Disallow: /es-us/olam/IdentityFailureAction.olamexecute*
195Disallow: /es-us/olam/validateAccountForgotIdSlid.myworld
196Disallow: /es-us/olam/jsp/login/inProgress.jsp*
197
198# Small Business
199
200Disallow: /es-us/smallbusiness/404.jsp?
201Disallow: /es-us/smallbusiness/b?
202Disallow: /es-us/smallbusiness/search
203Disallow: /es-us/smallbusiness/wireless/buyFlowContainer.jsp
204Disallow: /es-us/smallbusiness/i9nvalidParamInput.jsp
205
206User-agent: Googlebot
207Content-Signal: search=yes, ai-input=yes, ai-train=no
208
209Allow: /.well-known/agent-card.json
210Allow: /.well-known/mcp/server-card.json
211Allow: /.well-known/agent-skills/index.json
212Allow: /.well-known/ai-catalog.json
213Allow: /.well-known/api-catalog
214Allow: /ai-instructions/shopping-assistant.md
215Allow: /ai-instructions/build-a-plan.md
216Allow: /ai-instructions/build-a-plan.html
217Allow: /ai.txt
218Allow: /llms.txt
219Allow: /llms-full.txt
220Allow: /trust.txt
221Allow: /openapi.json
222
223Disallow: /gtm/
224
225# edits 05.01.24 - start
226
227Disallow: /search/
228Disallow: /cfd/
229Disallow: /*bvstate=pg*
230Disallow: /*/rest/clicktrack?*
231Disallow: /*/rest/getlist?*
232Disallow: /buy/postpaid/line-selection
233Disallow: /ssaf/ssafc/
234Disallow: /*/sdattservicesinc/*
235
236# edits 01.25.25 - end
237
238# Consumer Wireless and Home
239
240Disallow: /support/showarticle*
241Disallow: /contactus/*
242Disallow: /esupport/index.html*
243Disallow: /esupport/index.jsp*
244Disallow: /esupport/article.html*
245Disallow: /esupport/article.jsp*
246Disallow: /esupport/main.html*
247Disallow: /esupport/main.jsp*
248Disallow: /esupport/u-verse-tv-troubleshooting/*
249Disallow: /esupport/billAndAccount.jsp*
250Disallow: /esupport/security.jsp*
251Disallow: /esupport/postmaster*
252Disallow: /esupport/online-services.jsp*
253Disallow: /esupport/smbindex.jsp*
254Disallow: /esupport/report-call-or-text.html*
255Disallow: /esupport/speed-calculator*
256Disallow: /esupport/data-calculator*
257Disallow: /OLAM_PROD_CMS/*
258Disallow: /*EDOCSSESSIONID*
259Disallow: /*SESSIONID*
260Disallow: /*es-us/esupport/article.jsp?pagebody*
261Disallow: /*es/legalterms.html
262Disallow: /*rex/recommendation/*
263Disallow: /Common/maintenance/maintenance.html
264Disallow: /Common/popup/
265Disallow: /Common/supern/
266Disallow: /Large-Files/
267Disallow: /apis/maps/v2/locator/search/query.json
268Disallow: /apis/maps/v2/locator/search/viewport.json
269Disallow: /billinserts/
270Disallow: /dsl/modals/
271Disallow: /dsl/shop/fragments/
272Disallow: /buy/broadband
273Disallow: /buy/directv
274Disallow: /buy/cart
275Disallow: /buy/wireless/
276Disallow: /buy/wireless/devices
277Disallow: /buy/byod/
278Disallow: /buy/byod/sim-selection/
279Disallow: /buy/bundles
280Disallow: /esupport/ref/showarticle.jsp*
281Disallow: /esupport/sender/
282Disallow: /esupport/sender/browseResults.jsp
283Disallow: /esupport/solutionDisplay.jsp
284Disallow: /gen/press-room?pid=19457&cdvn=news&newsarticleid=31735
285Disallow: /maintenance/maintenance.html
286Disallow: /maps/not-available.html
287Disallow: /privacy/policy/
288Disallow: /global-search/search*
289Disallow: /mktg/
290
291Allow: /mktg/v1/_next/
292
293Disallow: /olam/passthroughAction.myworld*
294Disallow: /olam/loginAction.olamexecute
295Disallow: /olam/IdentityFailureAction.olamexecute*
296Disallow: /olam/validateAccountForgotIdSlid.myworld
297Disallow: /olam/jsp/login/inProgress.jsp*
298
299# Small Business
300
301Disallow: /smallbusiness/404.jsp?
302Disallow: /smallbusiness/b?
303Disallow: /smallbusiness/search
304Disallow: /smallbusiness/wireless/buyFlowContainer.jsp
305Disallow: /smallbusiness/i9nvalidParamInput.jsp
306Disallow: /es-us/OLAM_PROD_CMS/*
307Disallow: /es-us/*EDOCSSESSIONID*
308Disallow: /es-us/*SESSIONID*
309Disallow: /es-us/*es-us/esupport/article.jsp?pagebody*
310Disallow: /es-us/*es/legalterms.html
311Disallow: /es-us/*rex/recommendation/*
312Disallow: /es-us/Common/maintenance/maintenance.html
313Disallow: /es-us/Common/popup/
314Disallow: /es-us/Common/supern/
315Disallow: /es-us/Large-Files/
316Disallow: /es-us/apis/maps/v2/locator/search/query.json
317Disallow: /es-us/apis/maps/v2/locator/search/viewport.json
318Disallow: /es-us/billinserts/
319Disallow: /es-us/dsl/modals/
320Disallow: /es-us/dsl/shop/fragments/
321Disallow: /es-us/buy/broadband
322Disallow: /es-us/buy/directv
323Disallow: /es-us/buy/bundles
324
325Allow: /es-us/buy/bundles?product_suite=NBB
326
327Disallow: /es-us/buy/cart
328Disallow: /es-us/esupport/ref/showarticle.jsp*
329Disallow: /es-us/esupport/sender/
330Disallow: /es-us/esupport/sender/browseResults.jsp
331Disallow: /es-us/esupport/solutionDisplay.jsp
332Disallow: /es-us/gen/press-room?pid=19457&cdvn=news&newsarticleid=31735
333Disallow: /es-us/maintenance/maintenance.html
334Disallow: /es-us/maps/not-available.html
335Disallow: /es-us/privacy/policy/
336Disallow: /es-us/global-search/search*
337Disallow: /es-us/mktg/
338
339Allow: /es-us/mktg/v1/_next/
340
341Disallow: /es-us/olam/passthroughAction.myworld*
342Disallow: /es-us/olam/loginAction.olamexecute
343Disallow: /es-us/olam/IdentityFailureAction.olamexecute*
344Disallow: /es-us/olam/validateAccountForgotIdSlid.myworld
345Disallow: /es-us/olam/jsp/login/inProgress.jsp*
346
347# Small Business
348
349Disallow: /es-us/smallbusiness/404.jsp?
350Disallow: /es-us/smallbusiness/b?
351Disallow: /es-us/smallbusiness/search
352Disallow: /es-us/smallbusiness/wireless/buyFlowContainer.jsp
353Disallow: /es-us/smallbusiness/i9nvalidParamInput.jsp
354
355
356# PROD-34469
357
358Disallow: /u-verse/availability/pending-order.jsp
359Disallow: /u-verse/availability/check-customer-type.jsp
360
361
362# PROD-34470
363
364Disallow: /shop/unified/availability.html?destination_page=BYOB
365Disallow: /shop/unified/availability/results.html
366Disallow: /shop/unified/availability/calltoorder.html
367
368# AKA-6232
369
370Disallow: /myorders/*
371Disallow: /acctmgmt/*
372
373
374# Previous Update 8/14/2023
375# Last Update 10.15.2026
376
377# ====================================================================
378# Global pointers
379# ====================================================================
380
381Sitemap: https://www.att.com/sitemap.xml
382
383# Custom AI catalog directive
384Agentmap: https://www.att.com/.well-known/ai-catalog.json
385# Agent discovery resources are published through trust.txt and ai.txt.
386# https://www.att.com/.well-known/agent-card.json
387# https://www.att.com/.well-known/mcp/server-card.json
388# https://www.att.com/.well-known/agent-skills/index.json
389# https://www.att.com/ai-instructions/shopping-assistant.md
390
What do the bots actually get?
This page reads robots.txt, which states what att.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on att.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does att.com block GPTBot?
No. att.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does att.com block ClaudeBot?
No. att.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does att.com block Google-Extended?
No. att.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does att.com block PerplexityBot?
No. att.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find att.com?
Yes. att.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.