Who blocks AI · #117 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does weibo.com block AI crawlers?

Yes. weibo.com’s robots.txt blocks all 19 AI crawlers we track, from GPTBot and ClaudeBot to PerplexityBot. It also blocks DuckDuckBot, PetalBot and YandexBot among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: weibo.com/robots.txt · 45 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`YandexBot`	Yandex	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 38 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Claude-SearchBot`	Anthropic	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`DuckAssistBot`	DuckDuckGo	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Amazonbot`	Amazon	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`YouBot`	You.com	Blocked	line 43 · `Disallow: /` under `User-agent: *`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`ClaudeBot`	Anthropic	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Google-Extended`	Google	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`CCBot`	Common Crawl	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Applebot-Extended`	Apple	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`GoogleOther`	Google	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Meta-ExternalAgent`	Meta	Blocked	line 38 · `Disallow: /` under `User-agent: meta-externalagent`
`Bytespider`	ByteDance	Blocked	line 38 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 38 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Perplexity-User`	Perplexity	Blocked	line 38 · `Disallow: /` under `User-agent: Perplexity-User`
`MistralAI-User`	Mistral	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`Meta-ExternalFetcher`	Meta	Blocked	line 43 · `Disallow: /` under `User-agent: *`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`SemrushBot`	Semrush	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`MJ12bot`	Majestic	Blocked	line 43 · `Disallow: /` under `User-agent: *`
`DotBot`	Moz	Blocked	line 43 · `Disallow: /` under `User-agent: *`

View weibo.com/robots.txt · 45 lines

1Sitemap: https://weibo.com/sitemap.xml

3User-Agent: Baiduspider

4User-Agent: 360Spider

5User-Agent: Googlebot

6User-Agent: Sogou web spider

7User-Agent: bingbot

8User-Agent: smspider

9User-Agent: HaosouSpider

10User-Agent: YisouSpider

11Content-Signal: search=yes, ai-input=yes, ai-train=no

12Crawl-delay: 1

13Disallow:

15User-Agent: Google-adstxt

16Content-Signal: search=yes, ai-input=yes, ai-train=no

17Crawl-delay: 1

18Allow: /app-ads.txt

19Disallow: /

21User-Agent: Perplexity-User

22User-Agent: DeepSeek Chat

23User-Agent: DeepSeekBot

24User-Agent: Bytespider

25User-Agent: DeepSeek

26User-Agent: OAI-SearchBot

27User-Agent: ChatGPT-User

28User-Agent: Kimi

29User-Agent: Doubao

30User-Agent: meta-externalagent

31User-Agent: ChatGLM

32Content-Signal: search=yes, ai-input=yes, ai-train=no

33Crawl-delay: 1

34Allow: /sitemap.xml

35Allow: /sitemap/

36Allow: /2/detail/

37Allow: /ttarticle/p/show

38Disallow: /

40User-Agent: *

41Content-Signal: search=yes, ai-input=yes, ai-train=no

42Crawl-delay: 1

43Disallow: /

44Allow: /ads.txt

What do the bots actually get?

This page reads robots.txt, which states what weibo.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on weibo.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does weibo.com block GPTBot?

Yes. weibo.com’s robots.txt disallows GPTBot from the whole site (line 43: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does weibo.com block ClaudeBot?

Yes. weibo.com’s robots.txt disallows ClaudeBot from the whole site (line 43: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does weibo.com block Google-Extended?

Yes. weibo.com’s robots.txt disallows Google-Extended from the whole site (line 43: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does weibo.com block PerplexityBot?

Yes. weibo.com’s robots.txt disallows PerplexityBot from the whole site (line 43: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find weibo.com?

Not through its own crawler. weibo.com’s robots.txt disallows OAI-SearchBot from the whole site (line 38: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.