Who blocks AI · #987 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does codeberg.org block AI crawlers?

Yes. codeberg.org’s robots.txt blocks 12 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 9 more. It also blocks PetalBot among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: codeberg.org/robots.txt · 104 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Blocked	line 103 · `Disallow: /` under `User-agent: PetalBot`
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 103 · `Disallow: /` under `User-agent: OAI-SearchBot`
`PerplexityBot`	Perplexity	Blocked	line 103 · `Disallow: /` under `User-agent: PerplexityBot`
`Claude-SearchBot`	Anthropic	Allowed	no rule matches the home page
`DuckAssistBot`	DuckDuckGo	Allowed	no rule matches the home page
`Amazonbot`	Amazon	Blocked	line 103 · `Disallow: /` under `User-agent: Amazonbot`
`YouBot`	You.com	Blocked	line 103 · `Disallow: /` under `User-agent: YouBot`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 103 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Blocked	line 103 · `Disallow: /` under `User-agent: ClaudeBot`
`Google-Extended`	Google	Blocked	line 103 · `Disallow: /` under `User-agent: Google-Extended`
`CCBot`	Common Crawl	Blocked	line 103 · `Disallow: /` under `User-agent: CCBot`
`Applebot-Extended`	Apple	Blocked	line 103 · `Disallow: /` under `User-agent: Applebot-Extended`
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Blocked	line 103 · `Disallow: /` under `User-agent: meta-externalagent`
`Bytespider`	ByteDance	Blocked	line 103 · `Disallow: /` under `User-agent: Bytespider`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 103 · `Disallow: /` under `User-agent: ChatGPT-User`
`Claude-User`	Anthropic	Allowed	no rule matches the home page
`Perplexity-User`	Perplexity	Allowed	no rule matches the home page
`MistralAI-User`	Mistral	Allowed	no rule matches the home page
`Meta-ExternalFetcher`	Meta	Allowed	no rule matches the home page

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	no rule matches the home page
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View codeberg.org/robots.txt · 104 lines

1User-agent: *

2Disallow: /api/

3Disallow: /avatars/

4Disallow: /user/

5Disallow: /swagger.*.json

6Disallow: /explore/*?*

8Disallow: /repo/create

9Disallow: /repo/migrate

10Disallow: /org/create

11Disallow: /*/*/fork

13Disallow: /*/*/watchers

14Disallow: /*/*/stargazers

15Disallow: /*/*/forks

17Disallow: /*/*/src/

18Disallow: /*/*/blame/

19Disallow: /*/*/commit/

20Disallow: /*/*/commits/

21Disallow: /*/*/raw/

22Disallow: /*/*/media/

23Disallow: /*/*/tags

24Disallow: /*/*/graph

25Disallow: /*/*/branches

26Disallow: /*/*/compare

27Disallow: /*/*/lastcommit/

28Disallow: /*/*/rss/branch/

29Disallow: /*/*/atom/branch/

31Disallow: /*/*/activity

32Disallow: /*/*/activity_author_data

34Disallow: /*/*/actions

35Disallow: /*/*/projects

36Disallow: /*/*/labels

37Disallow: /*/*/milestones

39Disallow: /*/*/find/

40Disallow: /*/*/tree-list/

41Disallow: /*/*/search/

42Disallow: /*/-/code

44Disallow: /*/*/issues/new

45Disallow: /*/*/pulls/*/files

46Disallow: /*/*/pulls/*/commits

48Disallow: /attachments/

49Disallow: /*/*/attachments/

50Disallow: /*/*/issues/*/attachments/

51Disallow: /*/*/pulls/*/attachments/

52Disallow: /*/*/releases/attachments

53Disallow: /*/*/releases/download

55Disallow: /*/*/archive/

56Disallow: /*.bundle$

57Disallow: /*.patch$

58Disallow: /*.diff$

59Disallow: /*.atom$

60Disallow: /*.rss$

62Disallow: /*lang=*

63Disallow: /*redirect_to=*

64Disallow: /*tab=*

65Disallow: /*q=*

66Disallow: /*sort=*

67Disallow: /*repo-search-archived=*

69# Codeberg-specific changes

70Disallow: /Codeberg/*/*/Imprint.md

71Disallow: /mirror ## huge linux mirror, pointless to index

73Crawl-delay: 2

75User-agent: Amazonbot

76User-agent: anthropic-ai

77User-agent: Applebot-Extended

78User-agent: Bytespider

79User-agent: CCBot

80User-agent: ChatGPT-User

81User-agent: ClaudeBot

82User-agent: Claude-Web

83User-agent: cohere-ai

84User-agent: Diffbot

85User-agent: FacebookBot

86User-agent: facebookexternalhit

87User-agent: FriendlyCrawler

88User-agent: Google-Extended

89User-agent: GPTBot

90User-agent: ICC-Crawler

91User-agent: ImagesiftBot

92User-agent: img2dataset

93User-agent: meta-externalagent

94User-agent: OAI-SearchBot

95User-agent: Omgili

96User-agent: Omgilibot

97User-agent: PerplexityBot

98User-agent: PetalBot

99User-agent: Scrapy

100User-agent: Timpibot

101User-agent: VelenPublicWebCrawler

102User-agent: YouBot

103Disallow: /

104

What do the bots actually get?

This page reads robots.txt, which states what codeberg.org asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on codeberg.org Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does codeberg.org block GPTBot?

Yes. codeberg.org’s robots.txt disallows GPTBot from the whole site (line 103: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does codeberg.org block ClaudeBot?

Yes. codeberg.org’s robots.txt disallows ClaudeBot from the whole site (line 103: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does codeberg.org block Google-Extended?

Yes. codeberg.org’s robots.txt disallows Google-Extended from the whole site (line 103: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does codeberg.org block PerplexityBot?

Yes. codeberg.org’s robots.txt disallows PerplexityBot from the whole site (line 103: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find codeberg.org?

Not through its own crawler. codeberg.org’s robots.txt disallows OAI-SearchBot from the whole site (line 103: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.