Who blocks AI · #5 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does twitter.com block AI crawlers?

Yes. twitter.com’s robots.txt blocks all 19 AI crawlers we track, from GPTBot and ClaudeBot to PerplexityBot. It also blocks DuckDuckBot, PetalBot and 2 more among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: twitter.com/robots.txt · 123 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Partly blocked	line 16 · `Disallow: /*?lang=en-ss` line 18 · `Disallow: /search/realtime` and 16 more
`Bingbot`	Microsoft	Partly blocked	line 16 · `Disallow: /*?lang=en-ss` line 18 · `Disallow: /search/realtime` and 16 more
`DuckDuckBot`	DuckDuckGo	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`Applebot`	Apple	Partly blocked	line 16 · `Disallow: /*?lang=en-ss` line 18 · `Disallow: /search/realtime` and 16 more
`PetalBot`	Huawei	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`YandexBot`	Yandex	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`Baiduspider`	Baidu	Blocked	line 108 · `Disallow: /` under `User-agent: *`

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`PerplexityBot`	Perplexity	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`Claude-SearchBot`	Anthropic	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`DuckAssistBot`	DuckDuckGo	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`Amazonbot`	Amazon	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`YouBot`	You.com	Blocked	line 108 · `Disallow: /` under `User-agent: *`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`ClaudeBot`	Anthropic	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`Google-Extended`	Google	Blocked	line 76 · `Disallow: *` under `User-agent: Google-Extended`
`CCBot`	Common Crawl	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`Applebot-Extended`	Apple	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`GoogleOther`	Google	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`Meta-ExternalAgent`	Meta	Blocked	line 90 · `Disallow: *` under `User-agent: meta-externalagent`
`Bytespider`	ByteDance	Blocked	line 108 · `Disallow: /` under `User-agent: *`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`Claude-User`	Anthropic	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`Perplexity-User`	Perplexity	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`MistralAI-User`	Mistral	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`Meta-ExternalFetcher`	Meta	Blocked	line 96 · `Disallow: *` under `User-agent: meta-externalfetcher`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`SemrushBot`	Semrush	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`MJ12bot`	Majestic	Blocked	line 108 · `Disallow: /` under `User-agent: *`
`DotBot`	Moz	Blocked	line 108 · `Disallow: /` under `User-agent: *`

View twitter.com/robots.txt · 123 lines

1# Google / Bing Search Engine Robots

2# ==================================

3# Shared group so Googlebot and Bingbot always get the same rules.

4User-agent: Googlebot

5User-agent: Bingbot

7Allow: /*?s=

8Allow: /*?t=

9Allow: /*?ref_src=

10Allow: /hashtag/*?src=

11Allow: /search?q=%23

12Allow: /i/api/

13# Same length as the /*/likes and /*/media rules; RFC 9309 breaks the tie in

14# favor of Allow, so /i/api/ fetches stay crawlable.

15Allow: /i/api/*

16Disallow: /*?lang=en-ss

17Allow: /*?lang=

18Disallow: /search/realtime

19Disallow: /search/users

20Disallow: /search/*/grid

22Disallow: /*/analytics

23Disallow: /*/followers

24Disallow: /*/following

25Disallow: /*/verified_followers

27Disallow: /account/deactivated

28Disallow: /settings/deactivated

30# Only `*` and `$` are metacharacters here (RFC 9309); regex classes match

31# literally. /photo is anchored so /status/*/photo/1 media pages stay open.

32Disallow: /*/likes

33Disallow: /*/likes?

34Disallow: /*/retweets

35Disallow: /*/retweets?

36Disallow: /*/media

37Disallow: /*/media?

38Disallow: /*/photo$

39Disallow: /*/photo?

40Allow: /*?

42User-agent: facebookexternalhit

44Allow: /*?lang=

45Allow: /*?s=

46Allow: /*&s=

47Allow: /*?t=

48Allow: /*?ref_src=

49Allow: /hashtag/*?src=

50Allow: /search?*cashtagRestId=

51Allow: /i/api/

52Allow: /i/api/*

53Disallow: /search?q=

54Disallow: /search/realtime

55Disallow: /search/users

56Disallow: /search/*/grid

58Disallow: /*?

59Disallow: /*/followers

60Disallow: /*/following

61Disallow: /*/verified_followers

63Disallow: /account/deactivated

64Disallow: /settings/deactivated

66Disallow: /*/likes

67Disallow: /*/likes?

68Disallow: /*/retweets

69Disallow: /*/retweets?

70Disallow: /*/media

71Disallow: /*/media?

72Disallow: /*/photo$

73Disallow: /*/photo?

75User-Agent: Google-Extended

76Disallow: *

78User-Agent: FacebookBot

79Disallow: *

81User-agent: Discordbot

82Disallow: *

84# Meta AI / product crawlers (product tokens; versions like /1.1 are ignored

85# under RFC 9309 token matching).

86User-agent: meta-webindexer

87Disallow: *

89User-agent: meta-externalagent

90Disallow: *

92User-agent: meta-externalads

93Disallow: *

95User-agent: meta-externalfetcher

96Disallow: *

98# Shared AI training preference (IETF draft-kelley-aipref-training-preference-signal),

99# honored by Microsoft, Google, OpenAI, and Anthropic. This is a usage policy

100# token, not a crawler: it does not change crawl access, and `User-agent: *`

101# never matches it.

102User-agent: __usage_ai-training

103Disallow: /

104

105# Every bot that might possibly read and respect this file

106# ========================================================

107User-agent: *

108Disallow: /

109

110

111# WHAT-4882 - Keep notification-email links (/i/u) out of search results.

112# Named crawlers stay un-blocked on purpose: they must crawl /i/u to see its

113# X-Robots-Tag noindex (the robots.txt Noindex directive died in 2019).

114Disallow: /i/u

115

116# Wait 1 second between successive requests. See ONBOARD-2698 for details.

117Crawl-delay: 1

118

119# Independent of user agent. Links in the sitemap are full URLs using https://

120# and need to match the protocol of the sitemap.

121Sitemap: https://x.com/sitemap.xml

122Sitemap: https://x.com/sitemaps/finance0

123

What do the bots actually get?

This page reads robots.txt, which states what twitter.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on twitter.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does twitter.com block GPTBot?

Yes. twitter.com’s robots.txt disallows GPTBot from the whole site (line 108: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does twitter.com block ClaudeBot?

Yes. twitter.com’s robots.txt disallows ClaudeBot from the whole site (line 108: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does twitter.com block Google-Extended?

Yes. twitter.com’s robots.txt disallows Google-Extended from the whole site (line 76: Disallow: *). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does twitter.com block PerplexityBot?

Yes. twitter.com’s robots.txt disallows PerplexityBot from the whole site (line 108: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find twitter.com?

Not through its own crawler. twitter.com’s robots.txt disallows OAI-SearchBot from the whole site (line 108: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.