Who blocks AI · #915 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does lifehacker.com block AI crawlers?

Yes. lifehacker.com’s robots.txt blocks 17 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, PerplexityBot and 14 more. It also blocks Applebot and PetalBot among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedAllowed CCBotBlocked

robots.txt: lifehacker.com/robots.txt · 219 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	line 184 · `Allow: /` under `User-agent: *`
`Bingbot`	Microsoft	Allowed	line 184 · `Allow: /` under `User-agent: *`
`DuckDuckBot`	DuckDuckGo	Allowed	line 184 · `Allow: /` under `User-agent: *`
`Applebot`	Apple	Blocked	line 166 · `Disallow: /` under `User-agent: Applebot`
`PetalBot`	Huawei	Blocked	line 166 · `Disallow: /` under `User-agent: PetalBot`
`YandexBot`	Yandex	Allowed	line 184 · `Allow: /` under `User-agent: *`
`Baiduspider`	Baidu	Allowed	line 184 · `Allow: /` under `User-agent: *`

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 166 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 166 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Blocked	line 166 · `Disallow: /` under `User-agent: Claude-SearchBot`
`DuckAssistBot`	DuckDuckGo	Blocked	line 166 · `Disallow: /` under `User-agent: DuckAssistBot`
`Amazonbot`	Amazon	Blocked	line 166 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Blocked	line 166 · `Disallow: /` under `User-agent: YouBot`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 166 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 166 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Allowed	line 184 · `Allow: /` under `User-agent: *`
`CCBot`	Common Crawl	Blocked	line 166 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 166 · `Disallow: /` under `User-agent: Applebot-extended`
`GoogleOther`	Google	Allowed	line 184 · `Allow: /` under `User-agent: *`
`Meta-ExternalAgent`	Meta	Blocked	line 166 · `Disallow: /` under `User-agent: meta-externalagent`
`Bytespider`	ByteDance	Blocked	line 166 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 166 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Blocked	line 166 · `Disallow: /` under `User-agent: Claude-User`
`Perplexity-User`	Perplexity	Blocked	line 166 · `Disallow: /` under `User-agent: Perplexity-User`
`MistralAI-User`	Mistral	Blocked	line 166 · `Disallow: /` under `User-agent: MistralAI-User`
`Meta-ExternalFetcher`	Meta	Blocked	line 166 · `Disallow: /` under `User-agent: meta-externalfetcher`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	line 184 · `Allow: /` under `User-agent: *`
`SemrushBot`	Semrush	Allowed	line 184 · `Allow: /` under `User-agent: *`
`MJ12bot`	Majestic	Allowed	line 184 · `Allow: /` under `User-agent: *`
`DotBot`	Moz	Allowed	line 184 · `Allow: /` under `User-agent: *`

View lifehacker.com/robots.txt · 219 lines

1# Ziff Davis content is made available for your non-commercial use subject to our

2# Terms of Use here: https://www.ziffdavis.com/terms-of-use

3# Use of any robot, crawler, or other tool to scrape, harvest, extract, or retrieve any content on

4# this website using automated means is prohibited without written permission from Ziff Davis.

5# Prohibited uses include but are not limited to:

6# (1) text and data mining under Art. 4 of the EU Directive on Copyright in the Digital Single

7# Market;

8# (2) development or operation of artificial intelligence or machine learning software or

9# databases, including by training, fine-tuning, embedding, and retrieval-augmented generation;

10# (3) creating data sets containing our content or sharing it with others; and

11# (4) any commercial purposes.

12# Contact licensing@ziffdavis.com for assistance.

14# Bot Disallows

15# Block AI and other unwanted bots

17User-agent: AddSearchBot

18User-agent: AgentDataBot

19User-agent: AI2Bot

20User-agent: Ai2Bot-DeepResearchEval

21User-agent: Ai2Bot-Dolma

22User-agent: AIWebIndex

23User-agent: AIWebIndex-Agent

24User-agent: amazon-QBusiness

25User-agent: AmazonBuyForMe

26User-agent: Amazonbot

27User-agent: Amzn-SearchBot

28User-agent: Amzn-User

29User-agent: Anchor Browser

30User-agent: AndiAgent

31User-agent: anthropic-ai

32User-agent: ApifyBot

33User-agent: ApifyWebsiteContentCrawler

34User-agent: Applebot

35User-agent: Applebot-extended

36User-agent: atlassian-bot

37User-agent: AutoRAG

38User-agent: AwarioSmartBot

39User-agent: AzureAI-SearchBot

40User-agent: BeansLLM-CorpusBot

41User-agent: Big Sur AI

42User-agent: bigsur.ai

43User-agent: BixelBot

44User-agent: Brandwatch

45User-agent: Bravebot

46User-agent: Brightbot

47User-agent: Bytespider

48User-agent: CCBot

49User-agent: Channel3Bot

50User-agent: ChatGLM-Spider

51User-agent: ChatGPT-User

52User-agent: Claude-Code

53User-agent: Claude-SearchBot

54User-agent: Claude-User

55User-agent: Claude-Web

56User-agent: ClaudeBot

57User-agent: Cloudflare-AutoRAG

58User-agent: CloudflareBrowserRenderingCrawler

59User-agent: Code

60User-agent: cohere-ai

61User-agent: cohere-training-data-crawler

62User-agent: Cotoyogi

63User-agent: Crawl4AI

64User-agent: Cursor

65User-agent: Datenbank Crawler

66User-agent: Datenbank-Crawler

67User-agent: DeepSeekBot

68User-agent: DesearchBot

69User-agent: Devin

70User-agent: Diffbot

71User-agent: Direqt Anomura

72User-agent: Dola-SearchBot

73User-agent: DuckAssistBot

74User-agent: Echobot Bot

75User-agent: ExaBot

76User-agent: ExaSearchBot

77User-agent: FacebookBot

78User-agent: Factset_spyderbot

79User-agent: FirecrawlAgent

80User-agent: GeistHaus-PageFetcher

81User-agent: GPTBot

82User-agent: grok-agent

83User-agent: GrokBot

84User-agent: Hunyuan

85User-agent: iAsk

86User-agent: iAskBot

87User-agent: iaskspider

88User-agent: ICC Crawler

89User-agent: ICC-Crawler

90User-agent: ImageBot

91User-agent: ImageSiftBot

92User-agent: imageSpider

93User-agent: IsonSearchBot

94User-agent: kagi-fetcher

95User-agent: Kangaroo Bot

96User-agent: Kangaroo-Bot

97User-agent: Keenable-User

98User-agent: KeenableBot

99User-agent: Kimi-Agent

100User-agent: Kimi-User

101User-agent: KlaviyoAIBot

102User-agent: Kunato

103User-agent: KunatoCrawler

104User-agent: laion-huggingface-processor

105User-agent: LCC

106User-agent: LinerBot

107User-agent: LinkupBot

108User-agent: LINER Bot

109User-agent: Manus-User

110User-agent: meta-externalagent

111User-agent: meta-externalfetcher

112User-agent: meta-webindexer

113User-agent: mistral.ai

114User-agent: MistralAI-User

115User-agent: netEstate Imprint Crawler

116User-agent: Novellum

117User-agent: Novellum AI Crawl

118User-agent: NovaAct

119User-agent: OAI-SearchBot

120User-agent: omgili

121User-agent: opencode

122User-agent: PanguBot

123User-agent: Perplexity-User

124User-agent: PerplexityBot

125User-agent: PeopleInc-DCipher-Scraper/1.1.0

126User-agent: PetalBot

127User-agent: PhindBot

128User-agent: Poggio-Citations

129User-agent: qodercli

130User-agent: QualifiedBot

131User-agent: recraft-image-dataset

132User-agent: Reflectionbot

133User-agent: ScryBot

134User-agent: SeekrBot

135User-agent: SemrushBot-OCOB

136User-agent: SemrushBotSwa

137User-agent: Shap-User

138User-agent: ShapBot

139User-agent: SBIntuitionsBot

140User-agent: SofyaBot

141User-agent: SomethinggBot

142User-agent: Spider

143User-agent: TalarionFrontierCrawler

144User-agent: TalarionSearchBot

145User-agent: TerraCotta

146User-agent: Timpibot

147User-agent: TavilyBot

148User-agent: TongyiBot

149User-agent: Trae

150User-agent: TwinAgent

151User-agent: UseAI

152User-agent: Velen Crawler

153User-agent: VelenPublicWebCrawler

154User-agent: Webzio-Extended

155User-agent: Wrtn

156User-agent: WRTNBot

157User-agent: xAI-Grok

158User-agent: YandexAdditionalBot

159User-agent: YepBot

160User-agent: YiBot

161User-agent: YiyanBot

162User-agent: YouBot

163User-agent: ZanistaBot

164Allow: /ad/tech/how-to-actually-get-good-videos-at-a-concert-with-the-samsung-galaxy-s26

165Allow: /ad/tech/the-samsung-galaxy-s26-features-that-save-you-the-most-time-every-day

166Disallow: /

167

168# END Ziff Managed/Shared content

169

170# Ziff Davis content is made available for your non-commercial use subject to our

171# Terms of Use here: https://www.ziffdavis.com/terms-of-use

172# Use of any robot, crawler, or other tool to scrape, harvest, extract, or retrieve any content on

173# this website using automated means is prohibited without written permission from Ziff Davis.

174# Prohibited uses include but are not limited to:

175# (1) text and data mining under Art. 4 of the EU Directive on Copyright in the Digital Single

176# Market;

177# (2) development or operation of artificial intelligence or machine learning software or

178# databases, including by training, fine-tuning, embedding, and retrieval-augmented generation;

179# (3) creating data sets containing our content or sharing it with others; and

180# (4) any commercial purposes.

181# Contact licensing@ziffdavis.com for assistance.

182User-agent: *

183Disallow: /index.xml

184Allow: /

185

186User-agent: "008"

187User-agent: Ai2Bot-Dolma

188User-agent: Amazonbot

189User-agent: anthropic-ai

190User-agent: Applebot

191User-agent: Applebot-extended

192User-agent: Bytespider

193User-agent: CCBot

194User-agent: ChatGPT-User

195User-agent: Claude-Web

196User-agent: ClaudeBot

197User-agent: cohere-ai

198User-agent: DDM-DCipher/1.0.7

199User-agent: Diffbot

200User-agent: DuckAssistBot

201User-agent: FacebookBot

202User-agent: GPTBot

203User-agent: HTTrack

204User-agent: meta-externalagent

205User-agent: meta-externalfetcher

206User-agent: Nutch

207User-agent: OAI-SearchBot

208User-agent: Offline Explorer

209User-agent: omgili

210User-agent: Perplexity-User

211User-agent: PerplexityBot

212User-agent: Scrapy

213User-agent: Timpibot

214User-agent: YouBot

215Disallow: /

216

217Sitemap: https://lifehacker.com/sitemap-index.xml

218Sitemap: https://lifehacker.com/sitemap-google-news-0.xml

219

What do the bots actually get?

This page reads robots.txt, which states what lifehacker.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on lifehacker.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does lifehacker.com block GPTBot?

Yes. lifehacker.com’s robots.txt disallows GPTBot from the whole site (line 166: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does lifehacker.com block ClaudeBot?

Yes. lifehacker.com’s robots.txt disallows ClaudeBot from the whole site (line 166: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does lifehacker.com block Google-Extended?

No. lifehacker.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does lifehacker.com block PerplexityBot?

Yes. lifehacker.com’s robots.txt disallows PerplexityBot from the whole site (line 166: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find lifehacker.com?

Not through its own crawler. lifehacker.com’s robots.txt disallows OAI-SearchBot from the whole site (line 166: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.