Who blocks AI · #399 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does usnews.com block AI crawlers?

Yes. usnews.com’s robots.txt blocks 15 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 12 more. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: www.usnews.com/robots.txt · 228 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Allowed	no rule matches the home page
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 75 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 132 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Allowed	no rule matches the home page
`DuckAssistBot`	DuckDuckGo	Allowed	no rule matches the home page
`Amazonbot`	Amazon	Blocked	line 25 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Blocked	line 138 · `Disallow: /` under `User-agent: YouBot`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 59 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 83 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Blocked	line 109 · `Disallow: /` under `User-agent: Google-Extended`
`CCBot`	Common Crawl	Blocked	line 51 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 159 · `Disallow: /` under `User-agent: Applebot-Extended`
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Blocked	line 98 · `Disallow: /` under `User-agent: meta-externalagent`
`Bytespider`	ByteDance	Blocked	line 48 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 67 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Allowed	no rule matches the home page
`Perplexity-User`	Perplexity	Blocked	line 135 · `Disallow: /` under `User-agent: Perplexity-User`
`MistralAI-User`	Mistral	Blocked	line 165 · `Disallow: /` under `User-agent: MistralAI-User`
`Meta-ExternalFetcher`	Meta	Blocked	line 101 · `Disallow: /` under `User-agent: meta-externalfetcher`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	no rule matches the home page
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View www.usnews.com/robots.txt · 228 lines

1User-agent: *

2Disallow: /search

4User-agent: *

5Disallow: /emailprefs

7User-agent: Googlebot-News

8Disallow: /sponsored/

10User-agent: Applebot

11Allow: /brandfuse/articles/lemonade/how-to-create-a-home-inventory-checklist-for-renters-insurance-claims

12Allow: /brandfuse/articles/lemonade/how-renters-insurance-works-for-students-living-off-campus

13Allow: /brandfuse/articles/lemonade/what-renters-insurance-policies-commonly-exclude

14Allow: /brandfuse/articles/lemonade/5-high-value-items-your-renters-insurance-probably-doesnt-fully-cover

15Allow: /brandfuse/articles/lemonade/8-steps-to-evaluate-a-renters-insurance-policy-before-signing

17User-agent: Bingbot

18Allow: /brandfuse/articles/lemonade/how-to-create-a-home-inventory-checklist-for-renters-insurance-claims

19Allow: /brandfuse/articles/lemonade/how-renters-insurance-works-for-students-living-off-campus

20Allow: /brandfuse/articles/lemonade/what-renters-insurance-policies-commonly-exclude

21Allow: /brandfuse/articles/lemonade/5-high-value-items-your-renters-insurance-probably-doesnt-fully-cover

22Allow: /brandfuse/articles/lemonade/8-steps-to-evaluate-a-renters-insurance-policy-before-signing

24User-agent: Amazonbot

25Disallow: /

27User-agent: amazon-kendra

28Disallow: /

30User-agent: bedrockbot

31Disallow: /

33User-agent: Anthropic-ai

34Allow: /brandfuse/articles/lemonade/how-to-create-a-home-inventory-checklist-for-renters-insurance-claims

35Allow: /brandfuse/articles/lemonade/how-renters-insurance-works-for-students-living-off-campus

36Allow: /brandfuse/articles/lemonade/what-renters-insurance-policies-commonly-exclude

37Allow: /brandfuse/articles/lemonade/5-high-value-items-your-renters-insurance-probably-doesnt-fully-cover

38Allow: /brandfuse/articles/lemonade/8-steps-to-evaluate-a-renters-insurance-policy-before-signing

39Disallow: /

41User-agent: AwarioRssBot

42Disallow: /

44User-agent: AwarioSmartBot

45Disallow: /

47User-agent: Bytespider

48Disallow: /

50User-agent: CCBot

51Disallow: /

53User-agent: GPTBot

54Allow: /brandfuse/articles/lemonade/how-to-create-a-home-inventory-checklist-for-renters-insurance-claims

55Allow: /brandfuse/articles/lemonade/how-renters-insurance-works-for-students-living-off-campus

56Allow: /brandfuse/articles/lemonade/what-renters-insurance-policies-commonly-exclude

57Allow: /brandfuse/articles/lemonade/5-high-value-items-your-renters-insurance-probably-doesnt-fully-cover

58Allow: /brandfuse/articles/lemonade/8-steps-to-evaluate-a-renters-insurance-policy-before-signing

59Disallow: /

61User-agent: ChatGPT-User

62Allow: /brandfuse/articles/lemonade/how-to-create-a-home-inventory-checklist-for-renters-insurance-claims

63Allow: /brandfuse/articles/lemonade/how-renters-insurance-works-for-students-living-off-campus

64Allow: /brandfuse/articles/lemonade/what-renters-insurance-policies-commonly-exclude

65Allow: /brandfuse/articles/lemonade/5-high-value-items-your-renters-insurance-probably-doesnt-fully-cover

66Allow: /brandfuse/articles/lemonade/8-steps-to-evaluate-a-renters-insurance-policy-before-signing

67Disallow: /

69User-agent: OAI-SearchBot

70Allow: /brandfuse/articles/lemonade/how-to-create-a-home-inventory-checklist-for-renters-insurance-claims

71Allow: /brandfuse/articles/lemonade/how-renters-insurance-works-for-students-living-off-campus

72Allow: /brandfuse/articles/lemonade/what-renters-insurance-policies-commonly-exclude

73Allow: /brandfuse/articles/lemonade/5-high-value-items-your-renters-insurance-probably-doesnt-fully-cover

74Allow: /brandfuse/articles/lemonade/8-steps-to-evaluate-a-renters-insurance-policy-before-signing

75Disallow: /

77User-agent: ClaudeBot

78Allow: /brandfuse/articles/lemonade/how-to-create-a-home-inventory-checklist-for-renters-insurance-claims

79Allow: /brandfuse/articles/lemonade/how-renters-insurance-works-for-students-living-off-campus

80Allow: /brandfuse/articles/lemonade/what-renters-insurance-policies-commonly-exclude

81Allow: /brandfuse/articles/lemonade/5-high-value-items-your-renters-insurance-probably-doesnt-fully-cover

82Allow: /brandfuse/articles/lemonade/8-steps-to-evaluate-a-renters-insurance-policy-before-signing

83Disallow: /

85User-agent: Claude-Web

86Disallow: /

88User-agent: Cohere-ai

89Disallow: /

91User-agent: DataForSeoBot

92Disallow: /

94User-agent: FacebookBot

95Disallow: /

97User-agent: meta-externalagent

98Disallow: /

100User-agent: meta-externalfetcher

101Disallow: /

102

103User-agent: Google-Extended

104Allow: /brandfuse/articles/lemonade/how-to-create-a-home-inventory-checklist-for-renters-insurance-claims

105Allow: /brandfuse/articles/lemonade/how-renters-insurance-works-for-students-living-off-campus

106Allow: /brandfuse/articles/lemonade/what-renters-insurance-policies-commonly-exclude

107Allow: /brandfuse/articles/lemonade/5-high-value-items-your-renters-insurance-probably-doesnt-fully-cover

108Allow: /brandfuse/articles/lemonade/8-steps-to-evaluate-a-renters-insurance-policy-before-signing

109Disallow: /

110

111User-agent: Gemini-Deep-Research

112Disallow: /

113

114User-agent: ImagesiftBot

115Disallow: /

116

117User-agent: Magpie-crawler

118Disallow: /

119

120User-agent: Omgili

121Disallow: /

122

123User-agent: Omgilibot

124Disallow: /

125

126User-agent: PerplexityBot

127Allow: /brandfuse/articles/lemonade/how-to-create-a-home-inventory-checklist-for-renters-insurance-claims

128Allow: /brandfuse/articles/lemonade/how-renters-insurance-works-for-students-living-off-campus

129Allow: /brandfuse/articles/lemonade/what-renters-insurance-policies-commonly-exclude

130Allow: /brandfuse/articles/lemonade/5-high-value-items-your-renters-insurance-probably-doesnt-fully-cover

131Allow: /brandfuse/articles/lemonade/8-steps-to-evaluate-a-renters-insurance-policy-before-signing

132Disallow: /

133

134User-agent: Perplexity-User

135Disallow: /

136

137User-agent: YouBot

138Disallow: /

139

140User-agent: huggingface

141Disallow: /

142

143User-agent: NewsNow

144Disallow: /

145

146User-agent: news-please

147Disallow: /

148

149User-agent: Diffbot

150Disallow: /

151

152User-agent: Scrapy

153Disallow: /

154

155User-agent: TurnitinBot

156Disallow: /

157

158User-agent: Applebot-Extended

159Disallow: /

160

161User-agent: AI2Bot

162Disallow: /

163

164User-agent: MistralAI-User

165Disallow: /

166

167User-agent: ChatGLM-Spider

168Disallow: /

169

170User-agent: Cotoyogi

171Disallow: /

172

173User-agent: DeepSeekBot

174Disallow: /

175

176User-agent: EchoboxBot

177Disallow: /

178

179User-agent: iaskspider

180Disallow: /

181

182User-agent: ICC-Crawler

183Disallow: /

184

185User-agent: LinerBot

186Disallow: /

187

188User-agent: PanguBot

189Disallow: /

190

191User-agent: ProRataInc

192Disallow: /

193

194User-agent: QualifiedBot

195Disallow: /

196

197User-agent: SeekrBot

198Disallow: /

199

200User-agent: VelenPublicWebCrawler

201Disallow: /

202

203User-agent: WARDBot

204Disallow: /

205

206User-agent: WPBot

207Disallow: /

208

209User-agent: WRTNBot

210Disallow: /

211

212User-agent: ZanistaBot

213Disallow: /

214

215User-agent: KimiBot

216Disallow: /

217

218User-agent: Kimi-SearchBot

219Disallow: /

220

221User-agent: Kimi-User

222Disallow: /

223

224User-Agent: *

225sitemap: https://www.usnews.com/news/editorial-sitemap.xml

226sitemap: https://www.usnews.com/news-sitemap.xml

227sitemap: https://deals.usnews.com/sitemap.xml

228

What do the bots actually get?

This page reads robots.txt, which states what usnews.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on usnews.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does usnews.com block GPTBot?

Yes. usnews.com’s robots.txt disallows GPTBot from the whole site (line 59: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does usnews.com block ClaudeBot?

Yes. usnews.com’s robots.txt disallows ClaudeBot from the whole site (line 83: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does usnews.com block Google-Extended?

Yes. usnews.com’s robots.txt disallows Google-Extended from the whole site (line 109: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does usnews.com block PerplexityBot?

Yes. usnews.com’s robots.txt disallows PerplexityBot from the whole site (line 132: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find usnews.com?

Not through its own crawler. usnews.com’s robots.txt disallows OAI-SearchBot from the whole site (line 75: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.