Who blocks AI · #200 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does sagepub.com block AI crawlers?

Yes. sagepub.com’s robots.txt blocks 4 of the 19 AI crawlers we track, including CCBot, Meta-ExternalAgent, Bytespider and Amazonbot. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotAllowed OAI-SearchBotAllowed ClaudeBotAllowed PerplexityBotAllowed Google-ExtendedAllowed CCBotBlocked

robots.txt: www.sagepub.com/robots.txt · 248 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Allowed	no rule matches the home page
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Allowed	no rule matches the home page
`PerplexityBot`	Perplexity	Allowed	no rule matches the home page
`Claude-SearchBot`	Anthropic	Allowed	no rule matches the home page
`DuckAssistBot`	DuckDuckGo	Allowed	no rule matches the home page
`Amazonbot`	Amazon	Blocked	line 228 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Allowed	no rule matches the home page

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Allowed	no rule matches the home page
`ClaudeBot`	Anthropic	Allowed	no rule matches the home page
`Google-Extended`	Google	Allowed	no rule matches the home page
`CCBot`	Common Crawl	Blocked	line 234 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Allowed	no rule matches the home page
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Blocked	line 242 · `Disallow: /` under `User-agent: meta-externalagent`
`Bytespider`	ByteDance	Blocked	line 231 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Allowed	no rule matches the home page
`Claude-User`	Anthropic	Allowed	no rule matches the home page
`Perplexity-User`	Perplexity	Allowed	no rule matches the home page
`MistralAI-User`	Mistral	Allowed	no rule matches the home page
`Meta-ExternalFetcher`	Meta	Allowed	no rule matches the home page

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	no rule matches the home page
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View www.sagepub.com/robots.txt · 248 lines

1# _____ _____ _ _____ _ _

2# / ____| | __ \ | | | __ \ | | | |

3# | (___ __ _ __ _ ___ | |__) | _| |__ | |__) |___ | |__ ___ | |_ ___

4# \___ \ / _` |/ _` |/ _ \ | ___/ | | | '_ \ | _ // _ \| '_ \ / _ \| __/ __|

5# ____) | (_| | (_| | __/ | | | |_| | |_) | | | \ \ (_) | |_) | (_) | |_\__ \

6# |_____/ \__,_|\__, |\___| |_| \__,_|_.__/ |_| \_\___/|_.__/ \___/ \__|___/

7# __/ |

8# |___/

10#

11#

12# This file is to prevent the crawling and indexing of certain parts

13# of your site by web crawlers and spiders run by sites like Yahoo!

14# and Google. By telling these "robots" where not to go on your site,

15# you save bandwidth and server resources.

16#

17# This file will be ignored unless it is at the root of your host:

18# Used: http://example.com/robots.txt

19# Ignored: http://example.com/site/robots.txt

20#

21# For more information about the robots.txt standard, see:

22# http://www.robotstxt.org/robotstxt.html

23#

24#

25#############################################

26## GENERAL ALLOW RULES

27#############################################

28#

30User-agent: *

31# CSS, JS, Images

32Allow: /misc/*.css$

33Allow: /misc/*.css?

34Allow: /misc/*.js$

35Allow: /misc/*.js?

36Allow: /misc/*.gif

37Allow: /misc/*.jpg

38Allow: /misc/*.jpeg

39Allow: /misc/*.png

40Allow: /modules/*.css$

41Allow: /modules/*.css?

42Allow: /modules/*.js$

43Allow: /modules/*.js?

44Allow: /modules/*.gif

45Allow: /modules/*.jpg

46Allow: /modules/*.jpeg

47Allow: /modules/*.png

48Allow: /profiles/*.css$

49Allow: /profiles/*.css?

50Allow: /profiles/*.js$

51Allow: /profiles/*.js?

52Allow: /profiles/*.gif

53Allow: /profiles/*.jpg

54Allow: /profiles/*.jpeg

55Allow: /profiles/*.png

56Allow: /themes/*.css$

57Allow: /themes/*.css?

58Allow: /themes/*.js$

59Allow: /themes/*.js?

60Allow: /themes/*.gif

61Allow: /themes/*.jpg

62Allow: /themes/*.jpeg

63Allow: /themes/*.png

65#############################################

66## GENERAL DISALLOW RULES

67#############################################

69## Directories

70User-agent: *

71Disallow: /sitefinity/

72Disallow: /Sitefinity/

73Disallow: /misc/

74Disallow: /modules/

75Disallow: /profiles/

76Disallow: /scripts/

77Disallow: /themes/

79## Files (legacy)

80Disallow: /CHANGELOG.txt

81Disallow: /cron.php

82Disallow: /INSTALL.mysql.txt

83Disallow: /INSTALL.pgsql.txt

84Disallow: /INSTALL.sqlite.txt

85Disallow: /install.php

86Disallow: /INSTALL.txt

87Disallow: /LICENSE.txt

88Disallow: /MAINTAINERS.txt

89Disallow: /update.php

90Disallow: /UPGRADE.txt

91Disallow: /xmlrpc.php

93## Paths (clean URLs)

94Disallow: /admin/

95Disallow: /comment/reply/

96Disallow: /filter/tips/

97Disallow: /node/add/

98Disallow: /search/

99Disallow: /user/register/

100Disallow: /user/password/

101Disallow: /user/login/

102Disallow: /user/logout/

103

104## Paths (no clean URLs)

105Disallow: /?q=admin/

106Disallow: /?q=comment/reply/

107Disallow: /?q=filter/tips/

108Disallow: /?q=node/add/

109Disallow: /?q=search/

110Disallow: /?q=user/password/

111Disallow: /?q=user/register/

112Disallow: /?q=user/login/

113Disallow: /?q=user/logout/

114

115

116#############################################

117# GENERAL CRAWLER DISALLOW RULES

118# We don't know about their intentions, so block 'em

119#############################################

120User-agent: Robozilla

121Disallow: /

122User-agent: Gigabot

123Disallow: /

124User-agent: Nutch

125Disallow: /

126User-agent: naverbot

127Disallow: /

128User-agent: yeti

129Disallow: /

130User-agent: Sogou web spider

131Disallow: /

132User-agent: YisouSpider

133Disallow: /

134

135#############################################

136## SPECIFIC CRAWLER RULES

137#############################################

138

139# Apple

140User-agent: Applebot

141Disallow:

142

143# Ask

144User-agent: Teoma

145Disallow:

146

147# Baidu

148User-agent: baiduspider

149Disallow:

150Crawl-delay: 10

151

152# Google

153User-agent: Googlebot

154Disallow:

155User-agent: Googlebot-Image

156Disallow:

157User-agent: Googlebot-Mobile

158Disallow:

159User-agent: Googlebot-News

160Disallow:

161User-agent: Googlebot-Video

162Disallow:

163User-agent: Storebot-Google

164Disallow:

165User-agent: Mediapartners-Google

166Disallow:

167User-agent: AdsBot-Google

168Disallow:

169

170# Microsoft

171User-agent: bingbot

172Disallow:

173Crawl-delay: 10

174User-agent: msnbot

175Disallow:

176Crawl-delay: 10

177User-agent: msnbot-media

178Disallow:

179Crawl-delay: 10

180User-agent: adidxbot

181Disallow:

182Crawl-delay: 10

183

184# Yahoo

185User-agent: Slurp

186Disallow:

187Crawl-delay: 10

188User-agent: yahoo-mmcrawler

189Disallow:

190Crawl-delay: 10

191User-agent: yandex

192Disallow:

193Crawl-delay: 10

194

195# Wayback

196User-agent: ia_archiver

197Disallow:

198Crawl-delay: 10

199

200#############################################

201## KNOWN AI CRAWLER RULES

202#############################################

203## ALLOWED

204User-agent: Applebot-Extended

205Disallow:

206User-agent: Google-Extended

207Disallow:

208User-agent: GPTBot

209Disallow:

210User-Agent: ChatGPT-User

211Disallow:

212

213## DISALLOWED

214# General

215User-Agent: Omgilibot

216Disallow: /

217User-Agent: Omgili

218Disallow: /

219User-Agent: Diffbot

220Disallow: /

221User-Agent: ImagesiftBot

222Disallow: /

223User-Agent: cohere-ai

224Disallow: /

225

226# Amazon

227User-agent: Amazonbot

228Disallow: /

229# TikTok

230User-agent: Bytespider

231Disallow: /

232# Common Crawl

233User-agent: CCBot

234Disallow: /

235# Claude Anthropic

236User-Agent: anthropic-ai

237Disallow:

238User-agent: ClaudeBot

239Disallow:

240# Meta Facebook

241User-agent: meta-externalagent

242Disallow: /

243

244#############################################

245## SITEMAP

246#############################################

247

248Sitemap: https://www.sagepub.com/sitemap/sitemap-index.xml

What do the bots actually get?

This page reads robots.txt, which states what sagepub.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on sagepub.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does sagepub.com block GPTBot?

No. sagepub.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does sagepub.com block ClaudeBot?

No. sagepub.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does sagepub.com block Google-Extended?

No. sagepub.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does sagepub.com block PerplexityBot?

No. sagepub.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find sagepub.com?

Yes. sagepub.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.