Who blocks AI · #859 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does 1688.com block AI crawlers?

No. 1688.com’s robots.txt lets all 19 AI crawlers we track crawl the site. It also blocks Bingbot among search engines.

GPTBotAllowed OAI-SearchBotAllowed ClaudeBotAllowed PerplexityBotAllowed Google-ExtendedAllowed CCBotAllowed

robots.txt: www.1688.com/robots.txt · 368 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	line 166 · `Allow: /$` under `User-agent: Googlebot`
`Bingbot`	Microsoft	Blocked	line 208 · `Disallow: /` under `User-agent: bingbot`
`DuckDuckBot`	DuckDuckGo	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`Applebot`	Apple	Allowed	line 166 · `Allow: /$` under `User-agent: Googlebot`
`PetalBot`	Huawei	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`YandexBot`	Yandex	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`Baiduspider`	Baidu	Partly blocked	line 98 · `Disallow: /ask/` line 99 · `Disallow: /?*` and 64 more

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`PerplexityBot`	Perplexity	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`Claude-SearchBot`	Anthropic	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`DuckAssistBot`	DuckDuckGo	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`Amazonbot`	Amazon	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`YouBot`	You.com	Allowed	line 7 · `Allow: /$` under `User-agent: *`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`ClaudeBot`	Anthropic	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`Google-Extended`	Google	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`CCBot`	Common Crawl	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`Applebot-Extended`	Apple	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`GoogleOther`	Google	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`Meta-ExternalAgent`	Meta	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`Bytespider`	ByteDance	Allowed	line 7 · `Allow: /$` under `User-agent: *`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`Claude-User`	Anthropic	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`Perplexity-User`	Perplexity	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`MistralAI-User`	Mistral	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`Meta-ExternalFetcher`	Meta	Allowed	line 7 · `Allow: /$` under `User-agent: *`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`SemrushBot`	Semrush	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`MJ12bot`	Majestic	Allowed	line 7 · `Allow: /$` under `User-agent: *`
`DotBot`	Moz	Allowed	line 7 · `Allow: /$` under `User-agent: *`

View www.1688.com/robots.txt · 368 lines

1## -----------------------------------------------------------------------------

2## author shenyong modify by shengjie.gsj 2026.10.21

3## fileEncoding = UTF-8

4##

5## -----------------------------------------------------------------------------

6User-agent: *

7Allow: /$

8Allow: /?

9Allow: /wiki/

10Allow: /*/product-search/

11Allow: /chanpin/index.html?

12Allow: /chanpin/--*.html

13Allow: /topic/index.html?

14Allow: /member/profile.htm?memberId=

15Disallow: /tw/*/

16Disallow: /chanpin/*?*

17Disallow: /gongsi/*?*

18Disallow: /qiye/*?*

19Disallow: /similar/*?*

20Disallow: /store/*?*

21Disallow: /new/*?*

22Disallow: /jiage/*?*

23Disallow: /pic/*?*

24Disallow: /yijiandaifa/*?*

25Disallow: /*?*

26Disallow: /chanpin/-*-0*.html

27Disallow: /chanpin/-*-1*.html

28Disallow: /chanpin/-*-2*.html

29Disallow: /chanpin/-*-3*.html

30Disallow: /chanpin/-*-4*.html

31Disallow: /chanpin/-*-5*.html

32Disallow: /chanpin/-*-6*.html

33Disallow: /chanpin/-*-7*.html

34Disallow: /chanpin/-*-8*.html

35Disallow: /chanpin/-*-9*.html

36Disallow: /gongsi/-*-0*.html

37Disallow: /gongsi/-*-1*.html

38Disallow: /gongsi/-*-2*.html

39Disallow: /gongsi/-*-3*.html

40Disallow: /gongsi/-*-4*.html

41Disallow: /gongsi/-*-5*.html

42Disallow: /gongsi/-*-6*.html

43Disallow: /gongsi/-*-7*.html

44Disallow: /gongsi/-*-8*.html

45Disallow: /gongsi/-*-9*.html

46Disallow: /*?tracelog=

47Disallow: /offerdetail/view_large_pics_*.html

48Disallow: /company/offerlist/

49Disallow: /company/*?

50Disallow: /company/turbine/

51Disallow: /company/post_company.htm

52Disallow: /company/my_company.htm

53Disallow: /company/company_contact.htm

54Disallow: /company/company_intro_detail.htm

55Disallow: /company/buyer/

56Disallow: /buyer/turbine/

57Disallow: /member/

58Disallow: /member/ajax/

59Disallow: /seller/

60Disallow: /offer/

61Disallow: /catalog/

62Disallow: /athena/

63Disallow: /alitalk/

64Disallow: /album/

65Disallow: /ali/news/

66Disallow: /bin/

67Disallow: /myalibaba/

68Disallow: /work/

69Disallow: /loan/

70Disallow: /product/

71Disallow: /ims/

72Disallow: /community/

73Disallow: /message/

74Disallow: /others/

75Disallow: /bizexpress/

76Disallow: /domain/

77Disallow: /misc/

78Disallow: /flash/

79Disallow: /index1.html

80Disallow: /message

81Disallow: /taoshihui/

82Disallow: /pifa/

84User-agent: Baiduspider

85Allow: /$

86Allow: /?

87Allow: /aiseo-sitemap/

88Allow: /wiki/

89Allow: /*/product-search/

90Allow: /chanpin/index.html?

91Allow: /chanpin/--*.html

92Allow: /topic/index.html?

93Allow: /member/profile.htm?memberId=

94Allow: /bidding/

95Allow: /enterprise/

96Allow: /chandi/

97Allow: /hotbid/

98Disallow: /ask/*

99Disallow: /*?*

100Disallow: /tw/*/

101Disallow: /chanpin/*?

102Disallow: /gongsi/*?

103Disallow: /qiye/*?

104Disallow: /similar/*?

105Disallow: /chanpin/-*-0*.html

106Disallow: /chanpin/-*-1*.html

107Disallow: /chanpin/-*-2*.html

108Disallow: /chanpin/-*-3*.html

109Disallow: /chanpin/-*-4*.html

110Disallow: /chanpin/-*-5*.html

111Disallow: /chanpin/-*-6*.html

112Disallow: /chanpin/-*-7*.html

113Disallow: /chanpin/-*-8*.html

114Disallow: /chanpin/-*-9*.html

115Disallow: /gongsi/-*-0*.html

116Disallow: /gongsi/-*-1*.html

117Disallow: /gongsi/-*-2*.html

118Disallow: /gongsi/-*-3*.html

119Disallow: /gongsi/-*-4*.html

120Disallow: /gongsi/-*-5*.html

121Disallow: /gongsi/-*-6*.html

122Disallow: /gongsi/-*-7*.html

123Disallow: /gongsi/-*-8*.html

124Disallow: /gongsi/-*-9*.html

125Disallow: /*?tracelog=

126Disallow: /offerdetail/view_large_pics_*.html

127Disallow: /company/offerlist/

128Disallow: /company/*?

129Disallow: /company/turbine/

130Disallow: /company/post_company.htm

131Disallow: /company/my_company.htm

132Disallow: /company/company_contact.htm

133Disallow: /company/company_intro_detail.htm

134Disallow: /company/buyer/

135Disallow: /buyer/turbine/

136Disallow: /member/

137Disallow: /member/ajax/

138Disallow: /seller/

139Disallow: /offer/

140Disallow: /catalog/

141Disallow: /athena/

142Disallow: /alitalk/

143Disallow: /album/

144Disallow: /ali/news/

145Disallow: /bin/

146Disallow: /myalibaba/

147Disallow: /work/

148Disallow: /loan/

149Disallow: /product/

150Disallow: /ims/

151Disallow: /community/

152Disallow: /message/

153Disallow: /others/

154Disallow: /bizexpress/

155Disallow: /domain/

156Disallow: /misc/

157Disallow: /flash/

158Disallow: /index1.html

159Disallow: /message

160Disallow: /taoshihui/

161Disallow: /qingqu/*

162Disallow: /video/*

163Disallow: /top/*

164

165User-agent: Googlebot

166Allow: /$

167Allow: /?

168Allow: /aiseo-sitemap/

169Allow: /wiki/

170Allow: /*/product-search/

171Allow: /xunjia/

172Allow: /seorush-sitemap/

173Allow: /baojia/

174Allow: /howmuch/

175Allow: /pingjia/

176Allow: /dingzhi/

177Allow: /jiagong/

178Allow: /changhuo/

179Allow: /yangpin/

180Allow: /gongchang/

181Allow: /changjia/

182Allow: /bangdan/

183Allow: /factory/

184Allow: /bidding/

185Allow: /enterprise/

186Allow: /chandi/

187Allow: /hotbid/

188

189User-agent: bingbot

190Allow: /$

191Allow: /?

192Allow: /aiseo-sitemap/

193Allow: /wiki/

194Allow: /*/product-search/

195Allow: /xunjia/

196Allow: /seorush-sitemap/

197Allow: /baojia/

198Allow: /howmuch/

199Allow: /pingjia/

200Allow: /dingzhi/

201Allow: /jiagong/

202Allow: /changhuo/

203Allow: /yangpin/

204Allow: /gongchang/

205Allow: /changjia/

206Allow: /bangdan/

207Allow: /factory/

208Disallow: /

209

210User-agent: MSNBot

211Allow: /$

212Allow: /?

213Allow: /aiseo-sitemap/

214Allow: /wiki/

215Allow: /*/product-search/

216Allow: /chanpin/index.html?

217Allow: /chanpin/--*.html

218Allow: /topic/index.html?

219Allow: /member/profile.htm?memberId=

220Allow: /tw/*/

221Allow: /xunjia/

222Allow: /seorush-sitemap/

223Allow: /baojia/

224Allow: /howmuch/

225Allow: /pingjia/

226Allow: /dingzhi/

227Allow: /jiagong/

228Allow: /changhuo/

229Allow: /yangpin/

230Allow: /gongchang/

231Allow: /changjia/

232Allow: /bangdan/

233Allow: /factory/

234Allow: /bidding/

235Allow: /enterprise/

236Allow: /chandi/

237Allow: /hotbid/

238Disallow: /chanpin/*.html

239Disallow: /gongsi/*.html

240Disallow: /qiye/*?

241Disallow: /similar/*?

242Disallow: /qiye/

243Disallow: /cluster/

244Disallow: /zhuanti/

245Disallow: /2yuandian/

246Disallow: /*?tracelog=

247Disallow: /offerdetail/view_large_pics_*.html

248Disallow: /company/offerlist/

249Disallow: /company/*?

250Disallow: /company/turbine/

251Disallow: /company/post_company.htm

252Disallow: /company/my_company.htm

253Disallow: /company/company_contact.htm

254Disallow: /company/company_intro_detail.htm

255Disallow: /company/buyer/

256Disallow: /buyer/turbine/

257Disallow: /member/

258Disallow: /member/ajax/

259Disallow: /seller/

260Disallow: /offer/

261Disallow: /catalog/

262Disallow: /athena/

263Disallow: /alitalk/

264Disallow: /album/

265Disallow: /ali/news/

266Disallow: /bin/

267Disallow: /myalibaba/

268Disallow: /work/

269Disallow: /loan/

270Disallow: /product/

271Disallow: /ims/

272Disallow: /community/

273Disallow: /message/

274Disallow: /others/

275Disallow: /bizexpress/

276Disallow: /domain/

277Disallow: /misc/

278Disallow: /flash/

279Disallow: /index1.html

280Disallow: /message

281Disallow: /taoshihui/

282Disallow: /chanpin/-*-0*.html

283Disallow: /chanpin/-*-1*.html

284Disallow: /chanpin/-*-2*.html

285Disallow: /chanpin/-*-3*.html

286Disallow: /chanpin/-*-4*.html

287Disallow: /chanpin/-*-5*.html

288Disallow: /chanpin/-*-6*.html

289Disallow: /chanpin/-*-7*.html

290Disallow: /chanpin/-*-8*.html

291Disallow: /chanpin/-*-9*.html

292

293User-agent: 360Spider

294Allow: /$

295Allow: /?

296Allow: /aiseo-sitemap/

297Allow: /wiki/

298Allow: /*/product-search/

299Allow: /chanpin/index.html?

300Allow: /chanpin/--*.html

301Allow: /topic/index.html?

302Allow: /member/profile.htm?memberId=

303Allow: /tw/*/

304Allow: /bidding/

305Allow: /enterprise/

306Allow: /chandi/

307Allow: /hotbid/

308Disallow: /*?*

309Disallow: /chanpin/*?

310Disallow: /gongsi/*?

311Disallow: /qiye/*?

312Disallow: /similar/*?

313Disallow: /chanpin/-*-0*.html

314Disallow: /chanpin/-*-1*.html

315Disallow: /chanpin/-*-2*.html

316Disallow: /chanpin/-*-3*.html

317Disallow: /chanpin/-*-4*.html

318Disallow: /chanpin/-*-5*.html

319Disallow: /chanpin/-*-6*.html

320Disallow: /chanpin/-*-7*.html

321Disallow: /chanpin/-*-8*.html

322Disallow: /chanpin/-*-9*.html

323Disallow: /gongsi/-*-0*.html

324Disallow: /gongsi/-*-1*.html

325Disallow: /gongsi/-*-2*.html

326Disallow: /gongsi/-*-3*.html

327Disallow: /gongsi/-*-4*.html

328Disallow: /gongsi/-*-5*.html

329Disallow: /gongsi/-*-6*.html

330Disallow: /gongsi/-*-7*.html

331Disallow: /gongsi/-*-8*.html

332Disallow: /gongsi/-*-9*.html

333Disallow: /*?tracelog=

334Disallow: /offerdetail/view_large_pics_*.html

335Disallow: /company/offerlist/

336Disallow: /company/*?

337Disallow: /company/turbine/

338Disallow: /company/post_company.htm

339Disallow: /company/my_company.htm

340Disallow: /company/company_contact.htm

341Disallow: /company/company_intro_detail.htm

342Disallow: /company/buyer/

343Disallow: /buyer/turbine/

344Disallow: /member/

345Disallow: /member/ajax/

346Disallow: /seller/

347Disallow: /offer/

348Disallow: /catalog/

349Disallow: /athena/

350Disallow: /alitalk/

351Disallow: /album/

352Disallow: /ali/news/

353Disallow: /bin/

354Disallow: /myalibaba/

355Disallow: /work/

356Disallow: /loan/

357Disallow: /product/

358Disallow: /ims/

359Disallow: /community/

360Disallow: /message/

361Disallow: /others/

362Disallow: /bizexpress/

363Disallow: /domain/

364Disallow: /misc/

365Disallow: /flash/

366Disallow: /index1.html

367Disallow: /message

368Disallow: /taoshihui/

What do the bots actually get?

This page reads robots.txt, which states what 1688.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on 1688.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does 1688.com block GPTBot?

No. 1688.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does 1688.com block ClaudeBot?

No. 1688.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does 1688.com block Google-Extended?

No. 1688.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does 1688.com block PerplexityBot?

No. 1688.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find 1688.com?

Yes. 1688.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.