Who blocks AI · #1227 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does unicode.org block AI crawlers?

Yes. unicode.org’s robots.txt blocks 1 of the 19 AI crawlers we track, including CCBot. It also blocks Bingbot and Baiduspider among search engines.

GPTBotAllowed OAI-SearchBotAllowed ClaudeBotAllowed PerplexityBotAllowed Google-ExtendedAllowed CCBotBlocked

robots.txt: unicode.org/robots.txt · 448 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Blocked	line 447 · `Disallow: /` under `User-agent: msnbot`
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Allowed	no rule matches the home page
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Blocked	line 428 · `Disallow: /` under `User-agent: Baiduspider`

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Allowed	no rule matches the home page
`PerplexityBot`	Perplexity	Allowed	no rule matches the home page
`Claude-SearchBot`	Anthropic	Allowed	no rule matches the home page
`DuckAssistBot`	DuckDuckGo	Allowed	no rule matches the home page
`Amazonbot`	Amazon	Allowed	no rule matches the home page
`YouBot`	You.com	Allowed	no rule matches the home page

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Allowed	no rule matches the home page
`ClaudeBot`	Anthropic	Allowed	no rule matches the home page
`Google-Extended`	Google	Allowed	no rule matches the home page
`CCBot`	Common Crawl	Blocked	line 428 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Allowed	no rule matches the home page
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Allowed	no rule matches the home page
`Bytespider`	ByteDance	Allowed	no rule matches the home page

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Allowed	no rule matches the home page
`Claude-User`	Anthropic	Allowed	no rule matches the home page
`Perplexity-User`	Perplexity	Allowed	no rule matches the home page
`MistralAI-User`	Mistral	Allowed	no rule matches the home page
`Meta-ExternalFetcher`	Meta	Allowed	no rule matches the home page

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 428 · `Disallow: /` under `User-agent: AhrefsBot`
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Blocked	line 447 · `Disallow: /` under `User-agent: dotbot`

View unicode.org/robots.txt · 448 lines

1User-agent: *

2Disallow: /draft/ # not official even if visible

5# Begin updates for 2025 - Release 17.0

6Disallow: /Public/16.0.0/ # obsolete data

7Disallow: /charts/PDF/Unicode-16.0 # obsolete version

8Disallow: /reports/tr46/tr46-33.html

9Disallow: /reports/tr14/tr14-53.html

10Disallow: /reports/tr15/tr15-56.html

11Disallow: /reports/tr9/tr9-50.html

12Disallow: /reports/tr51/tr51-27.html

13Disallow: /reports/tr34/tr34-31.html

14Disallow: /reports/tr35/tr35-74/ # obsolete version

15Disallow: /reports/tr25/tr25-14.pdf

16# End updates for 2025 - Release 17.0

18Disallow: /Public/1.1-Update/ # obsolete data

19Disallow: /Public/2.1-Update2/ # obsolete data

20Disallow: /Public/3.0-Update/ # obsolete data

21Disallow: /Public/3.0-Update/ # obsolete data

22Disallow: /Public/2.1-Update3/ # obsolete data

23Disallow: /Public/2.1-Update/ # obsolete data

24Disallow: /Public/2.1-Update4/ # obsolete data

25Disallow: /Public/3.0-Update1/ # obsolete data

26Disallow: /Public/3.1-Update/ # obsolete data

27Disallow: /Public/3.1-Update1/ # obsolete data

28Disallow: /Public/3.2-Update/ # obsolete data

29Disallow: /Public/4.0-Update/ # obsolete data

30Disallow: /Public/4.0-Update1/ # obsolete data

31Disallow: /Public/4.1.0/ # obsolete data

32Disallow: /Public/5.0.0/ # obsolete data

33Disallow: /Public/5.1.0/ # obsolete data

34Disallow: /Public/5.2.0/ # obsolete data

35Disallow: /Public/6.0.0/ # obsolete data

36Disallow: /Public/6.1.0/ # obsolete data

37Disallow: /Public/6.2.0/ # obsolete data

38Disallow: /Public/6.3.0/ # obsolete data

39Disallow: /Public/7.0.0/ # obsolete data

40Disallow: /Public/8.0.0/ # obsolete data

41Disallow: /Public/9.0.0/ # obsolete data

42Disallow: /Public/10.0.0/ # obsolete data

43Disallow: /Public/11.0.0/ # obsolete data

44Disallow: /Public/12.0.0/ # obsolete data

45Disallow: /Public/13.0.0/ # obsolete data

46Disallow: /Public/14.0.0/ # obsolete data

47Disallow: /Public/15.0.0/ # obsolete data

48Disallow: /Public/15.1.0/ # obsolete data

49Disallow: /fr/ # obsolete pages and charts

50Disallow: /versions/Unicode2.0.0/ # obsolete version

51Disallow: /versions/Unicode2.1.0/ # obsolete version

52Disallow: /versions/Unicode3.0.0/ # obsolete version

53Disallow: /versions/Unicode3.0.1/ # obsolete version

54Disallow: /versions/Unicode3.1.0/ # obsolete version

55Disallow: /versions/Unicode3.1.1/ # obsolete version

56Disallow: /versions/Unicode3.2.0/ # obsolete version

57Disallow: /versions/Unicode4.0.0/ # obsolete version

58Disallow: /versions/Unicode4.0.1/ # obsolete version

59Disallow: /versions/Unicode4.1.0/ # obsolete version

60Disallow: /versions/Unicode5.0.0/ # obsolete version

61Disallow: /versions/Unicode5.1.0/ # obsolete version

62Disallow: /versions/Unicode5.2.0/ # obsolete version

63Disallow: /versions/Unicode6.0.0/ # obsolete version

64Disallow: /versions/Unicode6.1.0/ # obsolete version

65Disallow: /versions/Unicode6.2.0/ # obsolete version

66Disallow: /versions/Unicode6.3.0/ # obsolete version

67Disallow: /versions/Unicode7.0.0/ # obsolete version

68Disallow: /versions/Unicode8.0.0/ # obsolete version

69Disallow: /versions/Unicode9.0.0/ # obsolete version

70Disallow: /versions/Unicode10.0.0/ # obsolete version

71Disallow: /versions/Unicode11.0.0/ # obsolete version

72Disallow: /versions/Unicode12.0.0/ # obsolete version

73Disallow: /versions/Unicode13.0.0/ # obsolete version

74Disallow: /versions/Unicode14.0.0/ # obsolete version

75Disallow: /versions/Unicode15.0.0/ # obsolete version

76Disallow: /versions/Unicode15.1.0/ # obsolete version

77Disallow: /charts/PDF/Unicode-3.1 # obsolete version

78Disallow: /charts/PDF/Unicode-3.2 # obsolete version

79Disallow: /charts/PDF/Unicode-4.0 # obsolete version

80Disallow: /charts/PDF/Unicode-4.1 # obsolete version

81Disallow: /charts/PDF/Unicode-5.0 # obsolete version

82Disallow: /charts/PDF/Unicode-5.1 # obsolete version

83Disallow: /charts/PDF/Unicode-5.2 # obsolete version

84Disallow: /charts/PDF/Unicode-6.0 # obsolete version

85Disallow: /charts/PDF/Unicode-6.1 # obsolete version

86Disallow: /charts/PDF/Unicode-6.2 # obsolete version

87Disallow: /charts/PDF/Unicode-6.3 # obsolete version

88Disallow: /charts/PDF/Unicode-7.0 # obsolete version

89Disallow: /charts/PDF/Unicode-8.0 # obsolete version

90Disallow: /charts/PDF/Unicode-9.0 # obsolete version

91Disallow: /charts/PDF/Unicode-10.0 # obsolete version

92Disallow: /charts/PDF/Unicode-11.0 # obsolete version

93Disallow: /charts/PDF/Unicode-12.0 # obsolete version

94Disallow: /charts/PDF/Unicode-13.0 # obsolete version

95Disallow: /charts/PDF/Unicode-14.0 # obsolete version

96Disallow: /charts/PDF/Unicode-15.0 # obsolete version

97Disallow: /charts/PDF/Unicode-15.1 # obsolete version

98Disallow: /charts/beta/ # not permanent content

99Disallow: /reports/tr1.html # obsolete TR

100Disallow: /reports/tr2.html # obsolete TR

101Disallow: /reports/tr3.html # obsolete TR

102Disallow: /reports/tr6-10.html # obsolete TR draft

103Disallow: /reports/tr1/ # obsolete TR

104Disallow: /reports/tr2/ # obsolete TR

105Disallow: /reports/tr3/ # obsolete TR

106Disallow: /reports/tr7/ # obsolete TR

107Disallow: /reports/tr12/ # obsolete TR

108Disallow: /reports/tr13/ # obsolete TR

109Disallow: /reports/tr19/ # obsolete TR

110Disallow: /reports/tr21/ # obsolete TR

111Disallow: /reports/tr30/ # obsolete TR

112Disallow: /reports/tr32/ # obsolete TR

113Disallow: /reports/tr40/ # obsolete TR

114Disallow: /reports/tr47/ # obsolete TR

115Disallow: /reports/tr49/ # obsolete TR

116Disallow: /reports/tr52/ # obsolete TR

117Disallow: /anon-ftp/ # same as Public, different path

118Disallow: /Public/MAPPINGS/OBSOLETE/ # obsolete data

119Disallow: /unicode/members/

120Disallow: /members/

121Disallow: /repository/ # dynamic

122Disallow: /cldr/repository/ # dynamic

123Disallow: /cldr/utility/ # dynamic

124Disallow: /cgi-bin/ # dynamic

125Disallow: /cldr/data/diff/ # dynamic

126Disallow: /cldr/trac/ # dynamic

127Disallow: /edcom/bugtrack/ # dynamic

128Disallow: /uli/trac/ # dynamic

129Disallow: /~srloomis/ut/trac/ #dynamic

130Disallow: /repos/ # dynamic

131Disallow: /repos/cldr-tmp/ # dynamic

132Disallow: /utility/trac/ # dynamic

133Disallow: /utility/ # dynamic

134Disallow: /forum/ # dynamic

135Disallow: /forum/viewtopic.php # dynamic, discontinued

136Allow: /cldr/data/common/

137Allow: /cldr/data/docs/

138Allow: /cldr/data/tools/

139Disallow: /cldr/data/ # everything else

140Disallow: /cldr/dropbox/ # not really public

141Disallow: /~ecartis # not really public

142Disallow: /unibook/oldversions/ # obsolete data

143Disallow: /reports/tr46/tr46-1.html

144Disallow: /reports/tr46/tr46-2.html

145Disallow: /reports/tr46/tr46-3.html

146Disallow: /reports/tr46/tr46-4.html

147Disallow: /reports/tr46/tr46-5.html

148Disallow: /reports/tr46/tr46-6.html

149Disallow: /reports/tr46/tr46-7.html

150Disallow: /reports/tr46/tr46-8.html

151Disallow: /reports/tr46/tr46-9.html

152Disallow: /reports/tr46/tr46-10.html

153Disallow: /reports/tr46/tr46-11.html

154Disallow: /reports/tr46/tr46-12.html

155Disallow: /reports/tr46/tr46-13.html

156Disallow: /reports/tr46/tr46-14.html

157Disallow: /reports/tr46/tr46-15.html

158Disallow: /reports/tr46/tr46-16.html

159Disallow: /reports/tr46/tr46-17.html

160Disallow: /reports/tr46/tr46-18.html

161Disallow: /reports/tr46/tr46-19.html

162Disallow: /reports/tr46/tr46-20.html

163Disallow: /reports/tr46/tr46-21.html

164Disallow: /reports/tr46/tr46-22.html

165Disallow: /reports/tr46/tr46-23.html

166Disallow: /reports/tr46/tr46-24.html

167Disallow: /reports/tr46/tr46-25.html

168Disallow: /reports/tr46/tr46-26.html

169Disallow: /reports/tr46/tr46-27.html

170Disallow: /reports/tr46/tr46-28.html

171Disallow: /reports/tr46/tr46-29.html

172Disallow: /reports/tr46/tr46-30.html

173Disallow: /reports/tr46/tr46-31.html

174Disallow: /reports/tr46/tr46-32.html

175Disallow: /reports/tr15/tr15-7.html

176Disallow: /reports/tr15/tr15-8.html

177Disallow: /reports/tr15/tr15-9.html

178Disallow: /reports/tr15/tr15-10.html

179Disallow: /reports/tr15/tr15-11.html

180Disallow: /reports/tr15/tr15-12.html

181Disallow: /reports/tr15/tr15-13.html

182Disallow: /reports/tr15/tr15-14.html

183Disallow: /reports/tr15/tr15-15.html

184Disallow: /reports/tr15/tr15-16.html

185Disallow: /reports/tr15/tr15-17.html

186Disallow: /reports/tr15/tr15-18.html

187Disallow: /reports/tr15/tr15-19.html

188Disallow: /reports/tr15/tr15-20.html

189Disallow: /reports/tr15/tr15-21.html

190Disallow: /reports/tr15/tr15-22.html

191Disallow: /reports/tr15/tr15-23.html

192Disallow: /reports/tr15/tr15-24.html

193Disallow: /reports/tr15/tr15-25.html

194Disallow: /reports/tr15/tr15-26.html

195Disallow: /reports/tr15/tr15-20.html

196Disallow: /reports/tr15/tr15-21.html

197Disallow: /reports/tr15/tr15-22.html

198Disallow: /reports/tr15/tr15-23.html

199Disallow: /reports/tr15/tr15-24.html

200Disallow: /reports/tr15/tr15-25.html

201Disallow: /reports/tr15/tr15-26.html

202Disallow: /reports/tr15/tr15-27.html

203Disallow: /reports/tr15/tr15-28.html

204Disallow: /reports/tr15/tr15-29.html

205Disallow: /reports/tr15/tr15-30.html

206Disallow: /reports/tr15/tr15-31.html

207Disallow: /reports/tr15/tr15-32.html

208Disallow: /reports/tr15/tr15-33.html

209Disallow: /reports/tr15/tr15-34.html

210Disallow: /reports/tr15/tr15-35.html

211Disallow: /reports/tr15/tr15-36.html

212Disallow: /reports/tr15/tr15-37.html

213Disallow: /reports/tr15/tr15-38.html

214Disallow: /reports/tr15/tr15-39.html

215Disallow: /reports/tr15/tr15-40.html

216Disallow: /reports/tr15/tr15-40.html

217Disallow: /reports/tr15/tr15-41.html

218Disallow: /reports/tr15/tr15-42.html

219Disallow: /reports/tr15/tr15-43.html

220Disallow: /reports/tr15/tr15-44.html

221Disallow: /reports/tr15/tr15-45.html

222Disallow: /reports/tr15/tr15-46.html

223Disallow: /reports/tr15/tr15-47.html

224Disallow: /reports/tr15/tr15-48.html

225Disallow: /reports/tr15/tr15-49.html

226Disallow: /reports/tr15/tr15-50.html

227Disallow: /reports/tr15/tr15-51.html

228Disallow: /reports/tr15/tr15-52.html

229Disallow: /reports/tr15/tr15-53.html

230Disallow: /reports/tr15/tr15-54.html

231Disallow: /reports/tr15/tr15-55.html

232Disallow: /reports/tr14/tr14-15.html

233Disallow: /reports/dtr14-03.html

234Disallow: /reports/tr14-4/index.html

235Disallow: /reports/tr14-5/index.html

236Disallow: /reports/tr9/tr9-10.html

237Disallow: /reports/tr9/tr9-11.html

238Disallow: /reports/tr9/tr9-12.html

239Disallow: /reports/tr9/tr9-13.html

240Disallow: /reports/tr9/tr9-14.html

241Disallow: /reports/tr9/tr9-15.html

242Disallow: /reports/tr9/tr9-16.html

243Disallow: /reports/tr9/tr9-17.html

244Disallow: /reports/tr9/tr9-18.html

245Disallow: /reports/tr9/tr9-19.html

246Disallow: /reports/tr9/tr9-20.html

247Disallow: /reports/tr9/tr9-21.html

248Disallow: /reports/tr9/tr9-22.html

249Disallow: /reports/tr9/tr9-23.html

250Disallow: /reports/tr9/tr9-24.html

251Disallow: /reports/tr9/tr9-25.html

252Disallow: /reports/tr9/tr9-26.html

253Disallow: /reports/tr9/tr9-20.html

254Disallow: /reports/tr9/tr9-21.html

255Disallow: /reports/tr9/tr9-22.html

256Disallow: /reports/tr9/tr9-23.html

257Disallow: /reports/tr9/tr9-24.html

258Disallow: /reports/tr9/tr9-25.html

259Disallow: /reports/tr9/tr9-26.html

260Disallow: /reports/tr9/tr9-27.html

261Disallow: /reports/tr9/tr9-28.html

262Disallow: /reports/tr9/tr9-29.html

263Disallow: /reports/tr9/tr9-30.html

264Disallow: /reports/tr9/tr9-31.html

265Disallow: /reports/tr9/tr9-32.html

266Disallow: /reports/tr9/tr9-33.html

267Disallow: /reports/tr9/tr9-34.html

268Disallow: /reports/tr9/tr9-35.html

269Disallow: /reports/tr9/tr9-36.html

270Disallow: /reports/tr9/tr9-37.html

271Disallow: /reports/tr9/tr9-38.html

272Disallow: /reports/tr9/tr9-39.html

273Disallow: /reports/tr9/tr9-40.html

274Disallow: /reports/tr9/tr9-40.html

275Disallow: /reports/tr9/tr9-41.html

276Disallow: /reports/tr9/tr9-42.html

277Disallow: /reports/tr9/tr9-43.html

278Disallow: /reports/tr9/tr9-44.html

279Disallow: /reports/tr9/tr9-45.html

280Disallow: /reports/tr9/tr9-46.html

281Disallow: /reports/tr9/tr9-47.html

282Disallow: /reports/tr9/tr9-48.html

283Disallow: /reports/tr9/tr9-49.html

284Disallow: /reports/tr51/tr51-1.html

285Disallow: /reports/tr51/tr51-1-archive.html

286Disallow: /reports/tr51/tr51-2.html

287Disallow: /reports/tr51/tr51-3.html

288Disallow: /reports/tr51/tr51-4.html

289Disallow: /reports/tr51/tr51-5.html

290Disallow: /reports/tr51/tr51-6.html

291Disallow: /reports/tr51/tr51-7.html

292Disallow: /reports/tr51/tr51-8.html

293Disallow: /reports/tr51/tr51-9.html

294Disallow: /reports/tr51/tr51-10.html

295Disallow: /reports/tr51/tr51-11.html

296Disallow: /reports/tr51/tr51-12.html

297Disallow: /reports/tr51/tr51-13.html

298Disallow: /reports/tr51/tr51-14.html

299Disallow: /reports/tr51/tr51-15.html

300Disallow: /reports/tr51/tr51-16.html

301Disallow: /reports/tr51/tr51-17.html

302Disallow: /reports/tr51/tr51-18.html

303Disallow: /reports/tr51/tr51-19.html

304Disallow: /reports/tr51/tr51-20.html

305Disallow: /reports/tr51/tr51-21.html

306Disallow: /reports/tr51/tr51-22.html

307Disallow: /reports/tr51/tr51-23.html

308Disallow: /reports/tr51/tr51-24.html

309Disallow: /reports/tr51/tr51-25.html

310Disallow: /reports/tr51/tr51-26.html

311Disallow: /reports/tr34/tr34-1.html

312Disallow: /reports/tr34/tr34-2.html

313Disallow: /reports/tr34/tr34-3.html

314Disallow: /reports/tr34/tr34-4.html

315Disallow: /reports/tr34/tr34-5.html

316Disallow: /reports/tr34/tr34-6.html

317Disallow: /reports/tr34/tr34-7.html

318Disallow: /reports/tr34/tr34-8.html

319Disallow: /reports/tr34/tr34-9.html

320Disallow: /reports/tr34/tr34-10.html

321Disallow: /reports/tr34/tr34-11.html

322Disallow: /reports/tr34/tr34-12.html

323Disallow: /reports/tr34/tr34-13.html

324Disallow: /reports/tr34/tr34-14.html

325Disallow: /reports/tr34/tr34-15.html

326Disallow: /reports/tr34/tr34-16.html

327Disallow: /reports/tr34/tr34-17.html

328Disallow: /reports/tr34/tr34-18.html

329Disallow: /reports/tr34/tr34-19.html

330Disallow: /reports/tr34/tr34-20.html

331Disallow: /reports/tr34/tr34-21.html

332Disallow: /reports/tr34/tr34-22.html

333Disallow: /reports/tr34/tr34-23.html

334Disallow: /reports/tr34/tr34-24.html

335Disallow: /reports/tr34/tr34-25.html

336Disallow: /reports/tr34/tr34-26.html

337Disallow: /reports/tr34/tr34-27.html

338Disallow: /reports/tr34/tr34-28.html

339Disallow: /reports/tr34/tr34-29.html

340Disallow: /reports/tr34/tr34-30.html

341Disallow: /reports/tr35/tr35-1.html # obsolete version

342Disallow: /reports/tr35/tr35-2.html # obsolete version

343Disallow: /reports/tr35/tr35-3.html # obsolete version

344Disallow: /reports/tr35/tr35-4.html # obsolete version

345Disallow: /reports/tr35/tr35-5.html # obsolete version

346Disallow: /reports/tr35/tr35-6.html # obsolete version

347Disallow: /reports/tr35/tr35-7.html # obsolete version

348Disallow: /reports/tr35/tr35-8.html # obsolete version

349Disallow: /reports/tr35/tr35-9.html # obsolete version

350Disallow: /reports/tr35/tr35-10.html # obsolete version

351Disallow: /reports/tr35/tr35-11.html # obsolete version

352Disallow: /reports/tr35/tr35-12.html # obsolete version

353Disallow: /reports/tr35/tr35-13.html # obsolete version

354Disallow: /reports/tr35/tr35-14d1.html # obsolete version

355Disallow: /reports/tr35/tr35-15.html # obsolete version

356Disallow: /reports/tr35/tr35-16.html # obsolete version

357Disallow: /reports/tr35/tr35-17.html # obsolete version

358Disallow: /reports/tr35/tr35-18.html # obsolete version

359Disallow: /reports/tr35/tr35-19.html # obsolete version

360Disallow: /reports/tr35/tr35-20.html # obsolete version

361Disallow: /reports/tr35/tr35-21.html # obsolete version

362Disallow: /reports/tr35/tr35-22.html # obsolete version

363Disallow: /reports/tr35/tr35-23.html # obsolete version

364Disallow: /reports/tr35/tr35-25.html # obsolete version

365Disallow: /reports/tr35/tr35-27.html # obsolete version

366Disallow: /reports/tr35/tr35-29.html # obsolete version

367Disallow: /reports/tr35/tr35-30.html # obsolete version

368Disallow: /reports/tr35/tr35-30/ # obsolete version

369Disallow: /reports/tr35/tr35-31/ # obsolete version

370Disallow: /reports/tr35/tr35-32/ # obsolete version

371Disallow: /reports/tr35/tr35-33/ # obsolete version

372Disallow: /reports/tr35/tr35-34/ # obsolete version

373Disallow: /reports/tr35/tr35-35/ # obsolete version

374Disallow: /reports/tr35/tr35-36/ # obsolete version

375Disallow: /reports/tr35/tr35-37/ # obsolete version

376Disallow: /reports/tr35/tr35-38/ # obsolete version

377Disallow: /reports/tr35/tr35-39/ # obsolete version

378Disallow: /reports/tr35/tr35-40/ # obsolete version

379Disallow: /reports/tr35/tr35-41/ # obsolete version

380Disallow: /reports/tr35/tr35-42/ # obsolete version

381Disallow: /reports/tr35/tr35-43/ # obsolete version

382Disallow: /reports/tr35/tr35-44/ # obsolete version

383Disallow: /reports/tr35/tr35-45/ # obsolete version

384Disallow: /reports/tr35/tr35-46/ # obsolete version

385Disallow: /reports/tr35/tr35-47/ # obsolete version

386Disallow: /reports/tr35/tr35-48/ # obsolete version

387Disallow: /reports/tr35/tr35-49/ # obsolete version

388Disallow: /reports/tr35/tr35-50/ # obsolete version

389Disallow: /reports/tr35/tr35-51/ # obsolete version

390Disallow: /reports/tr35/tr35-52/ # obsolete version

391Disallow: /reports/tr35/tr35-53/ # obsolete version

392Disallow: /reports/tr35/tr35-54/ # obsolete version

393Disallow: /reports/tr35/tr35-55/ # obsolete version

394Disallow: /reports/tr35/tr35-56/ # obsolete version

395Disallow: /reports/tr35/tr35-57/ # obsolete version

396Disallow: /reports/tr35/tr35-58/ # obsolete version

397Disallow: /reports/tr35/tr35-59/ # obsolete version

398Disallow: /reports/tr35/tr35-60/ # obsolete version

399Disallow: /reports/tr35/tr35-61/ # obsolete version

400Disallow: /reports/tr35/tr35-62/ # obsolete version

401Disallow: /reports/tr35/tr35-63/ # obsolete version

402Disallow: /reports/tr35/tr35-64/ # obsolete version

403Disallow: /reports/tr35/tr35-65/ # obsolete version

404Disallow: /reports/tr35/tr35-66/ # obsolete version

405Disallow: /reports/tr35/tr35-67/ # obsolete version

406Disallow: /reports/tr35/tr35-68/ # obsolete version

407Disallow: /reports/tr25/tr25-1.html

408Disallow: /reports/tr25/tr25-2.html

409Disallow: /reports/tr25/tr25-3.html

410Disallow: /reports/tr25/tr25-4.html

411Disallow: /reports/tr25/tr25-5.html

412Disallow: /reports/tr25/tr25-6.html

413Disallow: /reports/tr25/tr25-7.html

414Disallow: /reports/tr25/tr25-8.html

415Disallow: /reports/tr25/tr25-9.html

416Disallow: /reports/tr25/tr25-10.pdf

417Disallow: /reports/tr25/tr25-11.pdf

418Disallow: /reports/tr25/tr25-12.pdf

419Disallow: /reports/tr25/tr25-13.pdf

420Disallow: /reports/tr25/tr25-14.pdf

421User-Agent: AhrefsBot

422Crawl-Delay: 60

423User-Agent: Baiduspider

424Crawl-Delay: 60

425User-agent: CCBot

426Crawl-Delay: 60

427User-agent: Gigabot

428Disallow: /

429User-agent: betaBot

430Disallow: /

431User-agent: CCBot

432Crawl-Delay: 60

433User-agent: AhrefsBot

434Crawl-Delay: 60

435User-agent: dotbot

436Crawl-Delay: 60

437User-agent: MJ12bot

438Crawl-Delay: 60

439User-agent: msnbot

440Crawl-delay: 60

441User-agent: BLEXBot

442Crawl-delay: 60

443User-agent: SeznamBot

444Request-rate: 1/1m

445Crawl-delay: 60

446User-agent: Viber/18.3.2.0

447Disallow: /

448

What do the bots actually get?

This page reads robots.txt, which states what unicode.org asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on unicode.org Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does unicode.org block GPTBot?

No. unicode.org’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does unicode.org block ClaudeBot?

No. unicode.org’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does unicode.org block Google-Extended?

No. unicode.org’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does unicode.org block PerplexityBot?

No. unicode.org’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find unicode.org?

Yes. unicode.org’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.