Who blocks AI · #504 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does mashable.com block AI crawlers?

Yes. mashable.com’s robots.txt blocks 17 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, PerplexityBot and 14 more. It also blocks Applebot and PetalBot among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedAllowed CCBotBlocked

robots.txt: mashable.com/robots.txt · 209 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Blocked	line 169 · `Disallow: /` under `User-agent: Applebot`
`PetalBot`	Huawei	Blocked	line 169 · `Disallow: /` under `User-agent: PetalBot`
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 169 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 169 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Blocked	line 169 · `Disallow: /` under `User-agent: Claude-SearchBot`
`DuckAssistBot`	DuckDuckGo	Blocked	line 169 · `Disallow: /` under `User-agent: DuckAssistBot`
`Amazonbot`	Amazon	Blocked	line 169 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Blocked	line 169 · `Disallow: /` under `User-agent: YouBot`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 169 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 169 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Allowed	no rule matches the home page
`CCBot`	Common Crawl	Blocked	line 169 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 169 · `Disallow: /` under `User-agent: Applebot-extended`
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Blocked	line 169 · `Disallow: /` under `User-agent: meta-externalagent`
`Bytespider`	ByteDance	Blocked	line 169 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 169 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Blocked	line 169 · `Disallow: /` under `User-agent: Claude-User`
`Perplexity-User`	Perplexity	Blocked	line 169 · `Disallow: /` under `User-agent: Perplexity-User`
`MistralAI-User`	Mistral	Blocked	line 169 · `Disallow: /` under `User-agent: MistralAI-User`
`Meta-ExternalFetcher`	Meta	Blocked	line 169 · `Disallow: /` under `User-agent: meta-externalfetcher`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	no rule matches the home page
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View mashable.com/robots.txt · 209 lines

1# Ziff Davis content is made available for your non-commercial use subject to our

2# Terms of Use here: https://www.ziffdavis.com/terms-of-use

3# Use of any robot, crawler, or other tool to scrape, harvest, extract, or retrieve any content on

4# this website using automated means is prohibited without written permission from Ziff Davis.

5# Prohibited uses include but are not limited to:

6# (1) text and data mining under Art. 4 of the EU Directive on Copyright in the Digital Single

7# Market;

8# (2) development or operation of artificial intelligence or machine learning software or

9# databases, including by training, fine-tuning, embedding, and retrieval-augmented generation;

10# (3) creating data sets containing our content or sharing it with others; and

11# (4) any commercial purposes.

12# Contact licensing@ziffdavis.com for assistance.

14# Bot Disallows

15# Block AI and other unwanted bots

17User-agent: AddSearchBot

18User-agent: AgentDataBot

19User-agent: AI2Bot

20User-agent: Ai2Bot-DeepResearchEval

21User-agent: Ai2Bot-Dolma

22User-agent: AIWebIndex

23User-agent: AIWebIndex-Agent

24User-agent: amazon-QBusiness

25User-agent: AmazonBuyForMe

26User-agent: Amazonbot

27User-agent: Amzn-SearchBot

28User-agent: Amzn-User

29User-agent: Anchor Browser

30User-agent: AndiAgent

31User-agent: anthropic-ai

32User-agent: ApifyBot

33User-agent: ApifyWebsiteContentCrawler

34User-agent: Applebot

35User-agent: Applebot-extended

36User-agent: atlassian-bot

37User-agent: AutoRAG

38User-agent: AwarioSmartBot

39User-agent: AzureAI-SearchBot

40User-agent: BeansLLM-CorpusBot

41User-agent: Big Sur AI

42User-agent: bigsur.ai

43User-agent: BixelBot

44User-agent: Brandwatch

45User-agent: Bravebot

46User-agent: Brightbot

47User-agent: Bytespider

48User-agent: CCBot

49User-agent: Channel3Bot

50User-agent: ChatGLM-Spider

51User-agent: ChatGPT-User

52User-agent: Claude-Code

53User-agent: Claude-SearchBot

54User-agent: Claude-User

55User-agent: Claude-Web

56User-agent: ClaudeBot

57User-agent: Cloudflare-AutoRAG

58User-agent: CloudflareBrowserRenderingCrawler

59User-agent: Code

60User-agent: cohere-ai

61User-agent: cohere-training-data-crawler

62User-agent: Cotoyogi

63User-agent: Crawl4AI

64User-agent: Cursor

65User-agent: Datenbank Crawler

66User-agent: Datenbank-Crawler

67User-agent: DeepSeekBot

68User-agent: DesearchBot

69User-agent: Devin

70User-agent: Diffbot

71User-agent: Direqt Anomura

72User-agent: Dola-SearchBot

73User-agent: DuckAssistBot

74User-agent: Echobot Bot

75User-agent: ExaBot

76User-agent: ExaSearchBot

77User-agent: FacebookBot

78User-agent: Factset_spyderbot

79User-agent: FirecrawlAgent

80User-agent: GeistHaus-PageFetcher

81User-agent: GPTBot

82User-agent: grok-agent

83User-agent: GrokBot

84User-agent: Hunyuan

85User-agent: iAsk

86User-agent: iAskBot

87User-agent: iaskspider

88User-agent: ICC Crawler

89User-agent: ICC-Crawler

90User-agent: ImageBot

91User-agent: ImageSiftBot

92User-agent: imageSpider

93User-agent: IsonSearchBot

94User-agent: kagi-fetcher

95User-agent: Kangaroo Bot

96User-agent: Kangaroo-Bot

97User-agent: Keenable-User

98User-agent: KeenableBot

99User-agent: Kimi-Agent

100User-agent: Kimi-User

101User-agent: KlaviyoAIBot

102User-agent: Kunato

103User-agent: KunatoCrawler

104User-agent: laion-huggingface-processor

105User-agent: LCC

106User-agent: LinerBot

107User-agent: LinkupBot

108User-agent: LINER Bot

109User-agent: Manus-User

110User-agent: meta-externalagent

111User-agent: meta-externalfetcher

112User-agent: meta-webindexer

113User-agent: mistral.ai

114User-agent: MistralAI-User

115User-agent: netEstate Imprint Crawler

116User-agent: Novellum AI Crawl

117User-agent: Novellum

118User-agent: NovaAct

119User-agent: OAI-SearchBot

120User-agent: omgili

121User-agent: opencode

122User-agent: PanguBot

123User-agent: Perplexity-User

124User-agent: PerplexityBot

125User-agent: PeopleInc-DCipher-Scraper/1.1.0

126User-agent: PetalBot

127User-agent: PhindBot

128User-agent: Poggio-Citations

129User-agent: qodercli

130User-agent: QualifiedBot

131User-agent: recraft-image-dataset

132User-agent: Reflectionbot

133User-agent: ScryBot

134User-agent: SeekrBot

135User-agent: SemrushBot-OCOB

136User-agent: SemrushBotSwa

137User-agent: Shap-User

138User-agent: ShapBot

139User-agent: SBIntuitionsBot

140User-agent: SofyaBot

141User-agent: SomethinggBot

142User-agent: Spider

143User-agent: TalarionFrontierCrawler

144User-agent: TalarionSearchBot

145User-agent: TerraCotta

146User-agent: Timpibot

147User-agent: TavilyBot

148User-agent: TongyiBot

149User-agent: Trae

150User-agent: TwinAgent

151User-agent: UseAI

152User-agent: Velen Crawler

153User-agent: VelenPublicWebCrawler

154User-agent: Webzio-Extended

155User-agent: Wrtn

156User-agent: WRTNBot

157User-agent: xAI-Grok

158User-agent: YandexAdditionalBot

159User-agent: YepBot

160User-agent: YiBot

161User-agent: YiyanBot

162User-agent: YouBot

163User-agent: ZanistaBot

164Allow: /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-your-new-samsung-galaxy

165Allow: /ad/tech/how-the-samsung-galaxy-s26-ultra-turns-your-next-festival-into-content

166Allow: /ad/tech/msi-laptops-are-backpack-approved-for-gaming-ai-projects-and-bright-futures

167Allow: /ad/life/how-to-pick-the-right-wellness-gifts-for-everyone-on-your-holiday-list

168Allow: /ad/life/what-to-wear-to-a-holiday-party-without-overthinking-it

169Disallow: /

170

171# END Ziff Managed/Shared content

172

173# Ziff Davis content is made available for your non-commercial use subject to our

174# Terms of Use here: https://www.ziffdavis.com/terms-of-use

175# Use of any robot, crawler, or other tool to scrape, harvest, extract, or retrieve any content on

176# this website using automated means is prohibited without written permission from Ziff Davis.

177# Prohibited uses include but are not limited to:

178# (1) text and data mining under Art. 4 of the EU Directive on Copyright in the Digital Single

179# Market;

180# (2) development or operation of artificial intelligence or machine learning software or

181# databases, including by training, fine-tuning, embedding, and retrieval-augmented generation;

182# (3) creating data sets containing our content or sharing it with others; and

183# (4) any commercial purposes.

184# Contact licensing@ziffdavis.com for assistance.

185

186User-agent: *

187Disallow: /search

188Disallow: /archive/

189Disallow: /cdn-cgi/

190Allow: /*?page=[0-9]

191

192User-agent: "008"

193User-agent: Applebot

194User-agent: DDM-DCipher/1.0.7

195User-agent: GPTBot

196User-agent: HTTrack

197User-agent: Nutch

198User-agent: Offline Explorer

199User-agent: Scrapy

200Allow: /ad/article/samsung-galaxy-ai-changing-smartphone-photography

201Allow: /ad/article/how-samsung-galaxy-ai-can-upgrade-your-smartphone

202Allow: /ad/article/samsung-galaxy-ai-automates-everyday-tasks

203Disallow: /

204

205Sitemap: https://mashable.com/sitemap-index.xml

206Sitemap: https://mashable.com/sitemap-news-0.xml

207Sitemap: https://mashable.com/sitemap-live-blogs-0.xml

208

209

What do the bots actually get?

This page reads robots.txt, which states what mashable.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on mashable.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does mashable.com block GPTBot?

Yes. mashable.com’s robots.txt disallows GPTBot from the whole site (line 169: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does mashable.com block ClaudeBot?

Yes. mashable.com’s robots.txt disallows ClaudeBot from the whole site (line 169: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does mashable.com block Google-Extended?

No. mashable.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does mashable.com block PerplexityBot?

Yes. mashable.com’s robots.txt disallows PerplexityBot from the whole site (line 169: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find mashable.com?

Not through its own crawler. mashable.com’s robots.txt disallows OAI-SearchBot from the whole site (line 169: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.