Who blocks AI · #576 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does congress.gov block AI crawlers?
Yes. congress.gov’s robots.txt blocks 18 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 15 more. It also blocks PetalBot among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.congress.gov/robots.txt · 146 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 145 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 145 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 145 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 145 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 145 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 145 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 145 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 145 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 145 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 145 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 145 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Blocked | line 145 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Blocked | line 145 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 145 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 145 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 145 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 145 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 145 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Blocked | line 145 · Disallow: / under User-agent: meta-externalfetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.congress.gov/robots.txt · 146 lines
1User-agent: *
2Crawl-delay: 2
3Disallow: /search
4Disallow: /quick-search
5Disallow: /advanced-search
6Disallow: /*?q=*
7Disallow: /*&q=*
8Disallow: /*?r=*
9Disallow: /*&r=*
10Disallow: /*?s=*
11Disallow: /*&s=*
12Disallow: /account
13Disallow: /lac
14
15Sitemap: https://www.congress.gov/sitemap/index.xml
16
17User-agent: AddSearchBot
18User-agent: AI2Bot
19User-agent: AI2Bot-DeepResearchEval
20User-agent: Ai2Bot-Dolma
21User-agent: aiHitBot
22User-agent: amazon-kendra
23User-agent: Amazonbot
24User-agent: AmazonBuyForMe
25User-agent: Andibot
26User-agent: Anomura
27User-agent: anthropic-ai
28User-agent: atlassian-bot
29User-agent: Awario
30User-agent: bedrockbot
31User-agent: bigsur.ai
32User-agent: Bravebot
33User-agent: Brightbot 1.0
34User-agent: BuddyBot
35User-agent: Bytespider
36User-agent: CCBot
37User-agent: Channel3Bot
38User-agent: ChatGLM-Spider
39User-agent: ChatGPT Agent
40User-agent: ChatGPT-User
41User-agent: Claude-SearchBot
42User-agent: Claude-User
43User-agent: Claude-Web
44User-agent: ClaudeBot
45User-agent: Cloudflare-AutoRAG
46User-agent: CloudVertexBot
47User-agent: cohere-ai
48User-agent: cohere-training-data-crawler
49User-agent: Cotoyogi
50User-agent: Crawl4AI
51User-agent: Crawlspace
52User-agent: Datenbank Crawler
53User-agent: DeepSeekBot
54User-agent: Devin
55User-agent: Diffbot
56User-agent: DuckAssistBot
57User-agent: Echobot Bot
58User-agent: EchoboxBot
59User-agent: FacebookBot
60User-agent: facebookexternalhit
61User-agent: Factset_spyderbot
62User-agent: FirecrawlAgent
63User-agent: FriendlyCrawler
64User-agent: Gemini-Deep-Research
65User-agent: Google-CloudVertexBot
66User-agent: Google-Extended
67User-agent: Google-Firebase
68User-agent: Google-NotebookLM
69User-agent: GoogleAgent-Mariner
70User-agent: GoogleOther
71User-agent: GoogleOther-Image
72User-agent: GoogleOther-Video
73User-agent: GPTBot
74User-agent: iAskBot
75User-agent: iaskspider
76User-agent: iaskspider/2.0
77User-agent: IbouBot
78User-agent: ICC-Crawler
79User-agent: ImagesiftBot
80User-agent: imageSpider
81User-agent: img2dataset
82User-agent: ISSCyberRiskCrawler
83User-agent: Kangaroo Bot
84User-agent: KlaviyoAIBot
85User-agent: KunatoCrawler
86User-agent: laion-huggingface-processor
87User-agent: LAIONDownloader
88User-agent: LCC
89User-agent: LinerBot
90User-agent: Linguee Bot
91User-agent: LinkupBot
92User-agent: Manus-User
93User-agent: meta-externalagent
94User-agent: Meta-ExternalAgent
95User-agent: meta-externalfetcher
96User-agent: Meta-ExternalFetcher
97User-agent: meta-webindexer
98User-agent: MistralAI-User
99User-agent: MistralAI-User/1.0
100User-agent: MyCentralAIScraperBot
101User-agent: netEstate Imprint Crawler
102User-agent: NotebookLM
103User-agent: NovaAct
104User-agent: OAI-SearchBot
105User-agent: omgili
106User-agent: omgilibot
107User-agent: OpenAI
108User-agent: Operator
109User-agent: PanguBot
110User-agent: Panscient
111User-agent: panscient.com
112User-agent: Perplexity-User
113User-agent: PerplexityBot
114User-agent: PetalBot
115User-agent: PhindBot
116User-agent: Poggio-Citations
117User-agent: Poseidon Research Crawler
118User-agent: QualifiedBot
119User-agent: QuillBot
120User-agent: quillbot.com
121User-agent: SBIntuitionsBot
122User-agent: Scrapy
123User-agent: SemrushBot-OCOB
124User-agent: SemrushBot-SWA
125User-agent: ShapBot
126User-agent: Sidetrade indexer bot
127User-agent: Spider
128User-agent: TavilyBot
129User-agent: TerraCotta
130User-agent: Thinkbot
131User-agent: TikTokSpider
132User-agent: Timpibot
133User-agent: TwinAgent
134User-agent: VelenPublicWebCrawler
135User-agent: WARDBot
136User-agent: Webzio-Extended
137User-agent: webzio-extended
138User-agent: wpbot
139User-agent: WRTNBot
140User-agent: YaK
141User-agent: YandexAdditional
142User-agent: YandexAdditionalBot
143User-agent: YouBot
144User-agent: ZanistaBot
145Disallow: /
146
What do the bots actually get?
This page reads robots.txt, which states what congress.gov asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on congress.gov Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does congress.gov block GPTBot?
Yes. congress.gov’s robots.txt disallows GPTBot from the whole site (line 145: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does congress.gov block ClaudeBot?
Yes. congress.gov’s robots.txt disallows ClaudeBot from the whole site (line 145: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does congress.gov block Google-Extended?
Yes. congress.gov’s robots.txt disallows Google-Extended from the whole site (line 145: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does congress.gov block PerplexityBot?
Yes. congress.gov’s robots.txt disallows PerplexityBot from the whole site (line 145: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find congress.gov?
Not through its own crawler. congress.gov’s robots.txt disallows OAI-SearchBot from the whole site (line 145: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.