Who blocks AI · #972 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does today.com block AI crawlers?
Yes. today.com’s robots.txt blocks 17 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 14 more. It also blocks PetalBot among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.today.com/robots.txt · 443 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 362 · Disallow: / under User-agent: Petalbot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 315 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 351 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 136 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Blocked | line 22 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 428 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 208 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 123 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 197 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 101 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 44 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 242 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 90 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 112 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 136 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 351 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 277 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Blocked | line 254 · Disallow: / under User-agent: Meta-ExternalFetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 10 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.today.com/robots.txt · 443 lines
1User-agent: *
2Disallow: /search*
3Disallow: /xml/today/SitemapToday*.xml
4Disallow: /ajax*
5Disallow: /akam/
6Disallow: /mynewsapi/
7
8# Disallow Bots (Allow /shop only)
9User-agent: AhrefsBot
10Disallow: /
11Allow: /shop
12Allow: /shop/
13Allow: /about/today/today-digital-masthead-rcna155416
14Allow: /about/today-rcna226078
15Allow: /information/join-today-live-plaza-t250868
16Allow: /start-today/app/supplemental-terms
17Allow: /information/closed-captioning-t215383
18Allow: /information/-us-send-today-your-questions-comments-t215381
19
20User-agent: Amazonbot
21User-agent: AmazonBot
22Disallow: /
23Allow: /shop
24Allow: /shop/
25Allow: /about/today/today-digital-masthead-rcna155416
26Allow: /about/today-rcna226078
27Allow: /information/join-today-live-plaza-t250868
28Allow: /start-today/app/supplemental-terms
29Allow: /information/closed-captioning-t215383
30Allow: /information/-us-send-today-your-questions-comments-t215381
31
32User-agent: anthropic-ai
33Disallow: /
34Allow: /shop
35Allow: /shop/
36Allow: /about/today/today-digital-masthead-rcna155416
37Allow: /about/today-rcna226078
38Allow: /information/join-today-live-plaza-t250868
39Allow: /start-today/app/supplemental-terms
40Allow: /information/closed-captioning-t215383
41Allow: /information/-us-send-today-your-questions-comments-t215381
42
43User-agent: Applebot-Extended
44Disallow: /
45Allow: /shop
46Allow: /shop/
47Allow: /about/today/today-digital-masthead-rcna155416
48Allow: /about/today-rcna226078
49Allow: /information/join-today-live-plaza-t250868
50Allow: /start-today/app/supplemental-terms
51Allow: /information/closed-captioning-t215383
52Allow: /information/-us-send-today-your-questions-comments-t215381
53
54User-agent: Archive.org_bot
55Disallow: /
56Allow: /shop
57Allow: /shop/
58Allow: /about/today/today-digital-masthead-rcna155416
59Allow: /about/today-rcna226078
60Allow: /information/join-today-live-plaza-t250868
61Allow: /start-today/app/supplemental-terms
62Allow: /information/closed-captioning-t215383
63Allow: /information/-us-send-today-your-questions-comments-t215381
64
65User-agent: AwarioRssBot
66User-agent: AwarioSmartBot
67Disallow: /
68Allow: /shop
69Allow: /shop/
70Allow: /about/today/today-digital-masthead-rcna155416
71Allow: /about/today-rcna226078
72Allow: /information/join-today-live-plaza-t250868
73Allow: /start-today/app/supplemental-terms
74Allow: /information/closed-captioning-t215383
75Allow: /information/-us-send-today-your-questions-comments-t215381
76
77User-agent: Brightbot 1.0
78Disallow: /
79Allow: /shop
80Allow: /shop/
81Allow: /about/today/today-digital-masthead-rcna155416
82Allow: /about/today-rcna226078
83Allow: /information/join-today-live-plaza-t250868
84Allow: /start-today/app/supplemental-terms
85Allow: /information/closed-captioning-t215383
86Allow: /information/-us-send-today-your-questions-comments-t215381
87
88User-agent: Bytespider
89User-agent: ByteSpider Desktop
90Disallow: /
91Allow: /shop
92Allow: /shop/
93Allow: /about/today/today-digital-masthead-rcna155416
94Allow: /about/today-rcna226078
95Allow: /information/join-today-live-plaza-t250868
96Allow: /start-today/app/supplemental-terms
97Allow: /information/closed-captioning-t215383
98Allow: /information/-us-send-today-your-questions-comments-t215381
99
100User-agent: CCBot
101Disallow: /
102Allow: /shop
103Allow: /shop/
104Allow: /about/today/today-digital-masthead-rcna155416
105Allow: /about/today-rcna226078
106Allow: /information/join-today-live-plaza-t250868
107Allow: /start-today/app/supplemental-terms
108Allow: /information/closed-captioning-t215383
109Allow: /information/-us-send-today-your-questions-comments-t215381
110
111User-agent: ChatGPT-User
112Disallow: /
113Allow: /shop
114Allow: /shop/
115Allow: /about/today/today-digital-masthead-rcna155416
116Allow: /about/today-rcna226078
117Allow: /information/join-today-live-plaza-t250868
118Allow: /start-today/app/supplemental-terms
119Allow: /information/closed-captioning-t215383
120Allow: /information/-us-send-today-your-questions-comments-t215381
121
122User-agent: ClaudeBot
123Disallow: /
124Allow: /shop
125Allow: /shop/
126Allow: /about/today/today-digital-masthead-rcna155416
127Allow: /about/today-rcna226078
128Allow: /information/join-today-live-plaza-t250868
129Allow: /start-today/app/supplemental-terms
130Allow: /information/closed-captioning-t215383
131Allow: /information/-us-send-today-your-questions-comments-t215381
132
133User-agent: Claude-Web
134User-agent: Claude-User
135User-agent: Claude-SearchBot
136Disallow: /
137Allow: /shop
138Allow: /shop/
139Allow: /about/today/today-digital-masthead-rcna155416
140Allow: /about/today-rcna226078
141Allow: /information/join-today-live-plaza-t250868
142Allow: /start-today/app/supplemental-terms
143Allow: /information/closed-captioning-t215383
144Allow: /information/-us-send-today-your-questions-comments-t215381
145
146User-agent: cohere-ai
147Disallow: /
148Allow: /shop
149Allow: /shop/
150Allow: /about/today/today-digital-masthead-rcna155416
151Allow: /about/today-rcna226078
152Allow: /information/join-today-live-plaza-t250868
153Allow: /start-today/app/supplemental-terms
154Allow: /information/closed-captioning-t215383
155Allow: /information/-us-send-today-your-questions-comments-t215381
156
157User-agent: DataForSeoBot
158Disallow: /
159Allow: /shop
160Allow: /shop/
161Allow: /about/today/today-digital-masthead-rcna155416
162Allow: /about/today-rcna226078
163Allow: /information/join-today-live-plaza-t250868
164Allow: /start-today/app/supplemental-terms
165Allow: /information/closed-captioning-t215383
166Allow: /information/-us-send-today-your-questions-comments-t215381
167
168User-agent: Diffbot
169Disallow: /
170Allow: /shop
171Allow: /shop/
172Allow: /about/today/today-digital-masthead-rcna155416
173Allow: /about/today-rcna226078
174Allow: /information/join-today-live-plaza-t250868
175Allow: /start-today/app/supplemental-terms
176Allow: /information/closed-captioning-t215383
177Allow: /information/-us-send-today-your-questions-comments-t215381
178
179User-agent: ExaBot
180User-agent: ExaSearchBot
181Disallow: /
182Allow: /shop
183Allow: /shop/
184Allow: /about/today/today-digital-masthead-rcna155416
185Allow: /about/today-rcna226078
186Allow: /information/join-today-live-plaza-t250868
187Allow: /start-today/app/supplemental-terms
188Allow: /information/closed-captioning-t215383
189Allow: /information/-us-send-today-your-questions-comments-t215381
190
191User-agent: FacebookBot
192Disallow: /
193Allow: /shop
194Allow: /shop/
195
196User-agent: Google-Extended
197Disallow: /
198Allow: /shop
199Allow: /shop/
200Allow: /about/today/today-digital-masthead-rcna155416
201Allow: /about/today-rcna226078
202Allow: /information/join-today-live-plaza-t250868
203Allow: /start-today/app/supplemental-terms
204Allow: /information/closed-captioning-t215383
205Allow: /information/-us-send-today-your-questions-comments-t215381
206
207User-agent: GPTBot
208Disallow: /
209Allow: /shop
210Allow: /shop/
211Allow: /about/today/today-digital-masthead-rcna155416
212Allow: /about/today-rcna226078
213Allow: /information/join-today-live-plaza-t250868
214Allow: /start-today/app/supplemental-terms
215Allow: /information/closed-captioning-t215383
216Allow: /information/-us-send-today-your-questions-comments-t215381
217
218User-agent: ImageBot/1.0
219Disallow: /
220Allow: /shop
221Allow: /shop/
222Allow: /about/today/today-digital-masthead-rcna155416
223Allow: /about/today-rcna226078
224Allow: /information/join-today-live-plaza-t250868
225Allow: /start-today/app/supplemental-terms
226Allow: /information/closed-captioning-t215383
227Allow: /information/-us-send-today-your-questions-comments-t215381
228
229User-agent: magpie-crawler
230Disallow: /
231Allow: /shop
232Allow: /shop/
233Allow: /about/today/today-digital-masthead-rcna155416
234Allow: /about/today-rcna226078
235Allow: /information/join-today-live-plaza-t250868
236Allow: /start-today/app/supplemental-terms
237Allow: /information/closed-captioning-t215383
238Allow: /information/-us-send-today-your-questions-comments-t215381
239
240User-agent: Meta-ExternalAgent
241User-agent: meta-externalagent
242Disallow: /
243Allow: /shop
244Allow: /shop/
245Allow: /about/today/today-digital-masthead-rcna155416
246Allow: /about/today-rcna226078
247Allow: /information/join-today-live-plaza-t250868
248Allow: /start-today/app/supplemental-terms
249Allow: /information/closed-captioning-t215383
250Allow: /information/-us-send-today-your-questions-comments-t215381
251
252User-agent: Meta-ExternalFetcher
253User-agent: meta-externalfetcher
254Disallow: /
255Allow: /shop
256Allow: /shop/
257Allow: /about/today/today-digital-masthead-rcna155416
258Allow: /about/today-rcna226078
259Allow: /information/join-today-live-plaza-t250868
260Allow: /start-today/app/supplemental-terms
261Allow: /information/closed-captioning-t215383
262Allow: /information/-us-send-today-your-questions-comments-t215381
263
264User-agent: Meta-WebIndexer
265User-agent: meta-webindexer/1.1
266Disallow: /
267Allow: /shop
268Allow: /shop/
269Allow: /about/today/today-digital-masthead-rcna155416
270Allow: /about/today-rcna226078
271Allow: /information/join-today-live-plaza-t250868
272Allow: /start-today/app/supplemental-terms
273Allow: /information/closed-captioning-t215383
274Allow: /information/-us-send-today-your-questions-comments-t215381
275
276User-agent: MistralAI-User
277Disallow: /
278Allow: /shop
279Allow: /shop/
280Allow: /about/today/today-digital-masthead-rcna155416
281Allow: /about/today-rcna226078
282Allow: /information/join-today-live-plaza-t250868
283Allow: /start-today/app/supplemental-terms
284Allow: /information/closed-captioning-t215383
285Allow: /information/-us-send-today-your-questions-comments-t215381
286
287User-agent: NagetBot
288Disallow: /
289Allow: /shop
290Allow: /shop/
291Allow: /about/today/today-digital-masthead-rcna155416
292Allow: /about/today-rcna226078
293Allow: /information/join-today-live-plaza-t250868
294Allow: /start-today/app/supplemental-terms
295Allow: /information/closed-captioning-t215383
296Allow: /information/-us-send-today-your-questions-comments-t215381
297
298User-agent: NewsNow
299Disallow: /
300Allow: /shop
301Allow: /shop/
302
303User-agent: news-please
304Disallow: /
305Allow: /shop
306Allow: /shop/
307Allow: /about/today/today-digital-masthead-rcna155416
308Allow: /about/today-rcna226078
309Allow: /information/join-today-live-plaza-t250868
310Allow: /start-today/app/supplemental-terms
311Allow: /information/closed-captioning-t215383
312Allow: /information/-us-send-today-your-questions-comments-t215381
313
314User-agent: OAI-SearchBot
315Disallow: /
316Allow: /shop
317Allow: /shop/
318Allow: /about/today/today-digital-masthead-rcna155416
319Allow: /about/today-rcna226078
320Allow: /information/join-today-live-plaza-t250868
321Allow: /start-today/app/supplemental-terms
322Allow: /information/closed-captioning-t215383
323Allow: /information/-us-send-today-your-questions-comments-t215381
324
325User-agent: omgili
326User-agent: omgilibot
327Disallow: /
328Allow: /shop
329Allow: /shop/
330Allow: /about/today/today-digital-masthead-rcna155416
331Allow: /about/today-rcna226078
332Allow: /information/join-today-live-plaza-t250868
333Allow: /start-today/app/supplemental-terms
334Allow: /information/closed-captioning-t215383
335Allow: /information/-us-send-today-your-questions-comments-t215381
336
337User-agent: peer39_crawler
338User-agent: peer39_crawler/1.0
339Disallow: /
340Allow: /shop
341Allow: /shop/
342Allow: /about/today/today-digital-masthead-rcna155416
343Allow: /about/today-rcna226078
344Allow: /information/join-today-live-plaza-t250868
345Allow: /start-today/app/supplemental-terms
346Allow: /information/closed-captioning-t215383
347Allow: /information/-us-send-today-your-questions-comments-t215381
348
349User-agent: PerplexityBot
350User-agent: Perplexity-User
351Disallow: /
352Allow: /shop
353Allow: /shop/
354Allow: /about/today/today-digital-masthead-rcna155416
355Allow: /about/today-rcna226078
356Allow: /information/join-today-live-plaza-t250868
357Allow: /start-today/app/supplemental-terms
358Allow: /information/closed-captioning-t215383
359Allow: /information/-us-send-today-your-questions-comments-t215381
360
361User-agent: Petalbot
362Disallow: /
363Allow: /shop
364Allow: /shop/
365Allow: /about/today/today-digital-masthead-rcna155416
366Allow: /about/today-rcna226078
367Allow: /information/join-today-live-plaza-t250868
368Allow: /start-today/app/supplemental-terms
369Allow: /information/closed-captioning-t215383
370Allow: /information/-us-send-today-your-questions-comments-t215381
371
372User-agent: QuillBot
373User-agent: quillbot.com
374Disallow: /
375Allow: /shop
376Allow: /shop/
377Allow: /about/today/today-digital-masthead-rcna155416
378Allow: /about/today-rcna226078
379Allow: /information/join-today-live-plaza-t250868
380Allow: /start-today/app/supplemental-terms
381Allow: /information/closed-captioning-t215383
382Allow: /information/-us-send-today-your-questions-comments-t215381
383
384User-agent: Scrapy
385Disallow: /
386Allow: /shop
387Allow: /shop/
388
389User-agent: ShapBot
390Disallow: /
391Allow: /shop
392Allow: /shop/
393Allow: /about/today/today-digital-masthead-rcna155416
394Allow: /about/today-rcna226078
395Allow: /information/join-today-live-plaza-t250868
396Allow: /start-today/app/supplemental-terms
397Allow: /information/closed-captioning-t215383
398Allow: /information/-us-send-today-your-questions-comments-t215381
399
400User-agent: TaraGroup Intelligent Bot
401Disallow: /
402Allow: /shop
403Allow: /shop/
404Allow: /about/today/today-digital-masthead-rcna155416
405Allow: /about/today-rcna226078
406Allow: /information/join-today-live-plaza-t250868
407Allow: /start-today/app/supplemental-terms
408Allow: /information/closed-captioning-t215383
409Allow: /information/-us-send-today-your-questions-comments-t215381
410
411User-agent: TurnitinBot
412Disallow: /
413Allow: /shop
414Allow: /shop/
415
416User-agent: ViennaTinyBot
417Disallow: /
418Allow: /shop
419Allow: /shop/
420Allow: /about/today/today-digital-masthead-rcna155416
421Allow: /about/today-rcna226078
422Allow: /information/join-today-live-plaza-t250868
423Allow: /start-today/app/supplemental-terms
424Allow: /information/closed-captioning-t215383
425Allow: /information/-us-send-today-your-questions-comments-t215381
426
427User-agent: YouBot
428Disallow: /
429Allow: /shop
430Allow: /shop/
431Allow: /about/today/today-digital-masthead-rcna155416
432Allow: /about/today-rcna226078
433Allow: /information/join-today-live-plaza-t250868
434Allow: /start-today/app/supplemental-terms
435Allow: /information/closed-captioning-t215383
436Allow: /information/-us-send-today-your-questions-comments-t215381
437
438# Sitemaps
439Sitemap: https://www.today.com/sitemap/today/sitemap-index
440Sitemap: https://www.today.com/sitemap/today/sitemap-news
441Sitemap: https://www.today.com/sitemap/today/sitemap-curations
442Sitemap: https://www.today.com/sitemap/today/sitemap-shop.xml
443
What do the bots actually get?
This page reads robots.txt, which states what today.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on today.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
- 26 October 2026: started blocking Meta-ExternalFetcher.
Questions
Does today.com block GPTBot?
Yes. today.com’s robots.txt disallows GPTBot from the whole site (line 208: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does today.com block ClaudeBot?
Yes. today.com’s robots.txt disallows ClaudeBot from the whole site (line 123: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does today.com block Google-Extended?
Yes. today.com’s robots.txt disallows Google-Extended from the whole site (line 197: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does today.com block PerplexityBot?
Yes. today.com’s robots.txt disallows PerplexityBot from the whole site (line 351: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find today.com?
Not through its own crawler. today.com’s robots.txt disallows OAI-SearchBot from the whole site (line 315: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.