Who blocks AI · #890 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does nhk.or.jp block AI crawlers?

Yes. nhk.or.jp’s robots.txt blocks 10 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 7 more. It also blocks PetalBot and Baiduspider among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: www.nhk.or.jp/robots.txt · 142 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Blocked	line 19 · `Disallow: /` under `User-agent: PetalBot`
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Blocked	line 19 · `Disallow: /` under `User-agent: baiduspider`

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 14 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 14 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Allowed	no rule matches the home page
`DuckAssistBot`	DuckDuckGo	Allowed	no rule matches the home page
`Amazonbot`	Amazon	Allowed	no rule matches the home page
`YouBot`	You.com	Allowed	no rule matches the home page

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 14 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 14 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Blocked	line 14 · `Disallow: /` under `User-agent: Google-Extended`
`CCBot`	Common Crawl	Blocked	line 14 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 14 · `Disallow: /` under `User-agent: Applebot-Extended`
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Blocked	line 14 · `Disallow: /` under `User-agent: Meta-ExternalAgent`
`Bytespider`	ByteDance	Blocked	line 14 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 14 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Allowed	no rule matches the home page
`Perplexity-User`	Perplexity	Allowed	no rule matches the home page
`MistralAI-User`	Mistral	Allowed	no rule matches the home page
`Meta-ExternalFetcher`	Meta	Allowed	no rule matches the home page

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	no rule matches the home page
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View www.nhk.or.jp/robots.txt · 142 lines

1User-agent: GPTBot

2User-agent: ChatGPT-User

3User-agent: ClaudeBot

4User-agent: Google-Extended

5User-agent: Applebot-Extended

6User-agent: anthropic-ai

7User-agent: cohere-ai

8User-agent: CCBot

9User-agent: ICC-Crawler

10User-agent: Bytespider

11User-agent: OAI-SearchBot

12User-agent: PerplexityBot

13User-agent: Meta-ExternalAgent

14Disallow: /

16User-agent: PetalBot

17User-agent: baiduspider

18User-agent: baiduimagespider

19Disallow: /

21User-agent: Googlebot-Image

22User-agent: Googlebot-Video

23Disallow: /

24Disallow: /archives/k/

25Disallow: /archives/r/

26Disallow: /das/k/

27Disallow: /das/r/

28Disallow: /gendai/k/

29Disallow: /gendai/r/

30Disallow: /heart-net/k/

31Disallow: /heart-net/r/

32Disallow: /kenko/k/

33Disallow: /kenko/r/

34Disallow: /learning/k/

35Disallow: /learning/r/

36Disallow: /learning-blog/r/

37Disallow: /news/k/

38Disallow: /news/r/

39Disallow: /nhkworld/k/

40Disallow: /nhkworld/r/

41Disallow: /politics/k/

42Disallow: /politics/r/

43Disallow: /recipe/k/

44Disallow: /recipe/r/

45Disallow: /shutoken/k/

46Disallow: /shutoken/r/

47Allow: /robots.txt

48Allow: /favicon.ico

49Allow: /*/lreport/

50Allow: /archives/

51Allow: /assets/

52Allow: /common/

53Allow: /das/

54Allow: /gendai/

55Allow: /heart-net/

56Allow: /kenko/

57Allow: /learning/

58Allow: /learning-blog/

59Allow: /news/

60Allow: /nhkworld/

61Allow: /politics/

62Allow: /prog/

63Allow: /program/

64Allow: /recipe/

65Allow: /shutoken/

66Allow: /akita/info/

67Allow: /aomori/info/

68Allow: /chiba/info/

69Allow: /fukui/info/

70Allow: /fukuoka/info/

71Allow: /fukushima/info/

72Allow: /gifu/info/

73Allow: /hiroshima/info/

74Allow: /hokkaido/info/

75Allow: /kagoshima/info/

76Allow: /kanazawa/info/

77Allow: /kitakyushu/info/

78Allow: /kobe/info/

79Allow: /kochi/info/

80Allow: /kofu/info/

81Allow: /kumamoto/info/

82Allow: /kyoto/info/

83Allow: /maebashi/info/

84Allow: /matsue/info/

85Allow: /matsuyama/info/

86Allow: /mito/info/

87Allow: /miyazaki/info/

88Allow: /morioka/info/

89Allow: /nagano/info/

90Allow: /nagasaki/info/

91Allow: /nagoya/info/

92Allow: /nara/info/

93Allow: /niigata/info/

94Allow: /oita/info/

95Allow: /okayama/info/

96Allow: /okinawa/info/

97Allow: /osaka/info/

98Allow: /otsu/info/

99Allow: /saga/info/

100Allow: /saitama/info/

101Allow: /sendai/info/

102Allow: /shizuoka/info/

103Allow: /takamatsu/info/

104Allow: /tokushima/info/

105Allow: /tottori/info/

106Allow: /toyama/info/

107Allow: /tsu/info/

108Allow: /utsunomiya/info/

109Allow: /wakayama/info/

110Allow: /yamagata/info/

111Allow: /yamaguchi/info/

112Allow: /yokohama/info/

113

114User-agent: Twitterbot

115Disallow: /*/r/

116

117User-agent: *

118Disallow: /*.xml$

119Disallow: /*.json$

120Disallow: /*.cgi$

121Disallow: /*.cgi?

122Disallow: /*/api/

123Disallow: /*/hc/

124Disallow: /*/k/

125Disallow: /*/r/

126Disallow: /*/shv/

127Disallow: /cgiblog/

128Disallow: /cgisearch/

129Disallow: /chronicle/

130Disallow: /error/

131Disallow: /hc-*

132Disallow: /hensei/

133Disallow: /saigai/*/dl/

134Disallow: /saigai/*/dn/

135Disallow: /saigai/*/ev/

136Disallow: /saigai/*/ss/

137Disallow: /saigai/f/

138Disallow: /topepg/

139Disallow: /toppage/errors/

140

141Sitemap: https://www.nhk.or.jp/sitemap.xml

142

What do the bots actually get?

This page reads robots.txt, which states what nhk.or.jp asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on nhk.or.jp Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does nhk.or.jp block GPTBot?

Yes. nhk.or.jp’s robots.txt disallows GPTBot from the whole site (line 14: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does nhk.or.jp block ClaudeBot?

Yes. nhk.or.jp’s robots.txt disallows ClaudeBot from the whole site (line 14: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does nhk.or.jp block Google-Extended?

Yes. nhk.or.jp’s robots.txt disallows Google-Extended from the whole site (line 14: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does nhk.or.jp block PerplexityBot?

Yes. nhk.or.jp’s robots.txt disallows PerplexityBot from the whole site (line 14: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find nhk.or.jp?

Not through its own crawler. nhk.or.jp’s robots.txt disallows OAI-SearchBot from the whole site (line 14: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.