Who blocks AI · #425 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does newsweek.com block AI crawlers?
Yes. newsweek.com’s robots.txt blocks 8 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, PerplexityBot and 5 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedAllowedCCBotBlocked- robots.txt
- www.newsweek.com/robots.txt · 120 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Blocked | line 54 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 38 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 26 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Blocked | line 22 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 14 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Blocked | line 20 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 24 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Blocked | line 56 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.newsweek.com/robots.txt · 120 lines
1User-agent: *
2Disallow: /feeds/*
3Disallow: /system/
4Disallow: /addineyeV2l
5Disallow: /eyeblaster
6Disallow: /doubleclick/
7Disallow: /ads/sponsored
8Disallow: /_fs-ch-1T1wmsGaOgGaSxcX/
9
10# Disallow Bots
11User-agent: anthropic-ai
12Disallow: /
13User-agent: Applebot-Extended
14Disallow: /
15User-agent: AwarioRssBot
16Disallow: /
17User-agent: AwarioSmartBot
18Disallow: /
19User-agent: Bytespider
20Disallow: /
21User-agent: CCBot
22Disallow: /
23User-agent: ChatGPT-User
24Disallow: /
25User-agent: ClaudeBot
26Disallow: /
27User-agent: Claude-Web
28Disallow: /
29User-agent: cohere-ai
30Disallow: /
31User-agent: DataForSeoBot
32Disallow: /
33User-agent: Diffbot
34Disallow: /
35User-agent: FacebookBot
36Disallow: /
37User-agent: GPTBot
38Disallow: /
39User-agent: magpie-crawler
40Disallow: /
41User-agent: NewsNow
42Disallow: /
43User-agent: news-please
44Disallow: /
45User-agent: omgili
46Disallow: /
47User-agent: omgilibot
48Disallow: /
49User-agent: peer39_crawler
50Disallow: /
51User-agent: peer39_crawler/1.0
52Disallow: /
53User-agent: PerplexityBot
54Disallow: /
55User-agent: Perplexity-User
56Disallow: /
57User-agent: Scrapy
58Disallow: /
59User-agent: ShapBot
60Disallow: /
61User-agent: Shap-User
62Disallow: /
63User-agent: TurnitinBot
64Disallow: /
65
66Sitemap: https://www.newsweek.com/sitemap.xml
67Sitemap: https://www.newsweek.com/googlenews.xml
68Sitemap: https://www.newsweek.com/sitemap-2026.xml
69Sitemap: https://www.newsweek.com/sitemap-2025.xml
70Sitemap: https://www.newsweek.com/sitemap-2024.xml
71Sitemap: https://www.newsweek.com/sitemap-2023.xml
72Sitemap: https://www.newsweek.com/sitemap-2022.xml
73Sitemap: https://www.newsweek.com/sitemap-2021.xml
74Sitemap: https://www.newsweek.com/sitemap-2020.xml
75Sitemap: https://www.newsweek.com/sitemap-2019.xml
76Sitemap: https://www.newsweek.com/sitemap-2018.xml
77Sitemap: https://www.newsweek.com/sitemap-2017.xml
78Sitemap: https://www.newsweek.com/sitemap-2016.xml
79Sitemap: https://www.newsweek.com/sitemap-2015.xml
80Sitemap: https://www.newsweek.com/sitemap-2014.xml
81Sitemap: https://www.newsweek.com/sitemap-2013.xml
82Sitemap: https://www.newsweek.com/sitemap-2012.xml
83Sitemap: https://www.newsweek.com/sitemap-2011.xml
84Sitemap: https://www.newsweek.com/sitemap-2010.xml
85Sitemap: https://www.newsweek.com/sitemap-2009.xml
86Sitemap: https://www.newsweek.com/sitemap-2008.xml
87Sitemap: https://www.newsweek.com/sitemap-2007.xml
88Sitemap: https://www.newsweek.com/sitemap-2006.xml
89Sitemap: https://www.newsweek.com/sitemap-2005.xml
90Sitemap: https://www.newsweek.com/sitemap-2004.xml
91Sitemap: https://www.newsweek.com/sitemap-2003.xml
92Sitemap: https://www.newsweek.com/sitemap-2002.xml
93Sitemap: https://www.newsweek.com/sitemap-2001.xml
94Sitemap: https://www.newsweek.com/sitemap-2000.xml
95Sitemap: https://www.newsweek.com/sitemap-1999.xml
96Sitemap: https://www.newsweek.com/sitemap-1998.xml
97Sitemap: https://www.newsweek.com/sitemap-1997.xml
98Sitemap: https://www.newsweek.com/sitemap-1996.xml
99Sitemap: https://www.newsweek.com/sitemap-1995.xml
100Sitemap: https://www.newsweek.com/sitemap-1994.xml
101Sitemap: https://www.newsweek.com/sitemap-1993.xml
102Sitemap: https://www.newsweek.com/sitemap-1992.xml
103Sitemap: https://www.newsweek.com/sitemap-1991.xml
104Sitemap: https://www.newsweek.com/sitemap-1990.xml
105Sitemap: https://www.newsweek.com/sitemap-1988.xml
106Sitemap: https://www.newsweek.com/sitemap-1987.xml
107Sitemap: https://www.newsweek.com/sitemap-1986.xml
108Sitemap: https://www.newsweek.com/sitemap-1985.xml
109Sitemap: https://www.newsweek.com/sitemap-1984.xml
110Sitemap: https://www.newsweek.com/sitemap-1983.xml
111Sitemap: https://www.newsweek.com/sitemap-1982.xml
112Sitemap: https://www.newsweek.com/sitemap-1981.xml
113Sitemap: https://www.newsweek.com/sitemap-1980.xml
114Sitemap: https://www.newsweek.com/sitemap-1979.xml
115Sitemap: https://www.newsweek.com/sitemap-1975.xml
116Sitemap: https://www.newsweek.com/sitemap-1970.xml
117Sitemap: https://www.newsweek.com/sitemap-1969.xml
118Sitemap: https://www.newsweek.com/sitemap-1968.xml
119Sitemap: https://www.newsweek.com/sitemap-1964.xml
120Sitemap: https://www.newsweek.com/sitemap-1961.xml
What do the bots actually get?
This page reads robots.txt, which states what newsweek.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on newsweek.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
- 2 November 2026: stopped blocking Amazonbot; started blocking Perplexity-User.
Questions
Does newsweek.com block GPTBot?
Yes. newsweek.com’s robots.txt disallows GPTBot from the whole site (line 38: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does newsweek.com block ClaudeBot?
Yes. newsweek.com’s robots.txt disallows ClaudeBot from the whole site (line 26: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does newsweek.com block Google-Extended?
No. newsweek.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does newsweek.com block PerplexityBot?
Yes. newsweek.com’s robots.txt disallows PerplexityBot from the whole site (line 54: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find newsweek.com?
Yes. newsweek.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.