Who blocks AI · #391 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does zhihu.com block AI crawlers?

Yes. zhihu.com’s robots.txt blocks all 19 AI crawlers we track, from GPTBot and ClaudeBot to PerplexityBot. It also blocks DuckDuckBot, PetalBot and YandexBot among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: www.zhihu.com/robots.txt · 116 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Partly blocked	line 3 · `Disallow: /appview/` line 4 · `Disallow: /login` and 9 more
`Bingbot`	Microsoft	Partly blocked	line 30 · `Disallow: /appview/` line 31 · `Disallow: /login` and 9 more
`DuckDuckBot`	DuckDuckGo	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`Applebot`	Apple	Partly blocked	line 3 · `Disallow: /appview/` line 4 · `Disallow: /login` and 9 more
`PetalBot`	Huawei	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`YandexBot`	Yandex	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`Baiduspider`	Baidu	Partly blocked	line 59 · `Disallow: /appview/` line 60 · `Disallow: /login` and 10 more

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`PerplexityBot`	Perplexity	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`Claude-SearchBot`	Anthropic	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`DuckAssistBot`	DuckDuckGo	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`Amazonbot`	Amazon	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`YouBot`	You.com	Blocked	line 115 · `Disallow: /` under `User-agent: *`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`ClaudeBot`	Anthropic	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`Google-Extended`	Google	Blocked	line 111 · `Disallow: /` under `User-agent: Google-Extended`
`CCBot`	Common Crawl	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`Applebot-Extended`	Apple	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`GoogleOther`	Google	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`Meta-ExternalAgent`	Meta	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`Bytespider`	ByteDance	Blocked	line 115 · `Disallow: /` under `User-agent: *`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`Claude-User`	Anthropic	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`Perplexity-User`	Perplexity	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`MistralAI-User`	Mistral	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`Meta-ExternalFetcher`	Meta	Blocked	line 115 · `Disallow: /` under `User-agent: *`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`SemrushBot`	Semrush	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`MJ12bot`	Majestic	Blocked	line 115 · `Disallow: /` under `User-agent: *`
`DotBot`	Moz	Blocked	line 115 · `Disallow: /` under `User-agent: *`

View www.zhihu.com/robots.txt · 116 lines

1User-agent: Googlebot

2Allow: /tardis/jm

3Disallow: /appview/

4Disallow: /login

5Disallow: /logout

6Disallow: /resetpassword

7Disallow: /terms

8Disallow: /search

9Disallow: /notifications

10Disallow: /settings

11Disallow: /inbox

12Disallow: /admin_inbox

13Disallow: /*?guide*

15User-agent: Googlebot-Image

16Disallow: /appview/

17Disallow: /login

18Disallow: /logout

19Disallow: /resetpassword

20Disallow: /terms

21Disallow: /search

22Disallow: /notifications

23Disallow: /settings

24Disallow: /inbox

25Disallow: /admin_inbox

26Disallow: /*?guide*

28User-agent: bingbot

29Allow: /tardis/jm

30Disallow: /appview/

31Disallow: /login

32Disallow: /logout

33Disallow: /resetpassword

34Disallow: /terms

35Disallow: /search

36Disallow: /notifications

37Disallow: /settings

38Disallow: /inbox

39Disallow: /admin_inbox

40Disallow: /*?guide*

42User-agent: Baiduspider-news

43Allow: /tardis/jm

44Disallow: /appview/

45Disallow: /login

46Disallow: /logout

47Disallow: /resetpassword

48Disallow: /terms

49Disallow: /search

50Disallow: /notifications

51Disallow: /settings

52Disallow: /inbox

53Disallow: /admin_inbox

54Disallow: /*?guide*

55Disallow: /en/

57User-agent: Baiduspider

58Allow: /tardis/jm

59Disallow: /appview/

60Disallow: /login

61Disallow: /logout

62Disallow: /resetpassword

63Disallow: /terms

64Disallow: /search

65Disallow: /notifications

66Disallow: /settings

67Disallow: /inbox

68Disallow: /admin_inbox

69Disallow: /*?guide*

70Disallow: /en/

72User-agent: Baiduspider-render

73Allow: /tardis/jm

74Disallow: /appview/

75Disallow: /login

76Disallow: /logout

77Disallow: /resetpassword

78Disallow: /terms

79Disallow: /search

80Disallow: /notifications

81Disallow: /settings

82Disallow: /inbox

83Disallow: /admin_inbox

84Disallow: /*?guide*

85Disallow: /en/

87User-agent: Baiduspider-image

88Disallow: /appview/

89Disallow: /login

90Disallow: /logout

91Disallow: /resetpassword

92Disallow: /terms

93Disallow: /search

94Disallow: /notifications

95Disallow: /settings

96Disallow: /inbox

97Disallow: /admin_inbox

98Disallow: /*?guide*

99Disallow: /en/

100

101User-agent: AdsBot-Google

102Allow: /en/

103Disallow: /

104

105User-agent: Sogou web spider

106Allow: /tardis/sogou/

107Disallow: /

108

109User-Agent: Google-Extended

110Allow: /tardis/jm

111Disallow: /

112

113User-Agent: *

114Allow: /tardis/jm

115Disallow: /

116

What do the bots actually get?

This page reads robots.txt, which states what zhihu.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on zhihu.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does zhihu.com block GPTBot?

Yes. zhihu.com’s robots.txt disallows GPTBot from the whole site (line 115: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does zhihu.com block ClaudeBot?

Yes. zhihu.com’s robots.txt disallows ClaudeBot from the whole site (line 115: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does zhihu.com block Google-Extended?

Yes. zhihu.com’s robots.txt disallows Google-Extended from the whole site (line 111: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does zhihu.com block PerplexityBot?

Yes. zhihu.com’s robots.txt disallows PerplexityBot from the whole site (line 115: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find zhihu.com?

Not through its own crawler. zhihu.com’s robots.txt disallows OAI-SearchBot from the whole site (line 115: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.