Who blocks AI · #813 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does brookings.edu block AI crawlers?
No. brookings.edu’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.brookings.edu/robots.txt · 303 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.brookings.edu/robots.txt · 303 lines
1User-agent: *
2Disallow: /?*qry
3Disallow: /?*topics
4Disallow: /?*regions
5Disallow: /?*experts
6Disallow: /?*entities
7Disallow: /*?s=
8Disallow: /search/
9
10# Start disallow_search_engine_indexing_of_attachments block
11Disallow: /wp-content/uploads/2026/11/Beatty_General_Vita_2026_OCT_Brookings.pdf
12Disallow: /wp-content/uploads/2026/10/Mark-Sandys-Resume-for-Brookings.pdf
13Disallow: /wp-content/uploads/2026/10/rhinesmith_cv_2026-10-27.pdf
14Disallow: /wp-content/uploads/2026/09/GS_20260921_TechTankPodcast_AI.pdf
15Disallow: /wp-content/uploads/2026/09/20260909_TechTank_screen-time-bans_transcript.pdf
16Disallow: /wp-content/uploads/2026/09/Errata-Sept-2026.pdf
17Disallow: /wp-content/uploads/2026/09/Errata-Sept-2026.docx
18Disallow: /wp-content/uploads/2026/09/Chapter-1-Full-corrected.pdf
19Disallow: /wp-content/uploads/2026/09/Chpt-1-corrected.xlsx
20Disallow: /wp-content/uploads/2026/09/1-13-Full-corrected.pdf
21Disallow: /wp-content/uploads/2026/09/1.13-corrected.xlsx
22Disallow: /wp-content/uploads/2026/09/1-13-corrected.csv
23Disallow: /wp-content/uploads/2026/09/1-11-corrected.csv
24Disallow: /wp-content/uploads/2026/09/1.11-corrected.xlsx
25Disallow: /wp-content/uploads/2026/09/1-11-Full-corrected.pdf
26Disallow: /wp-content/uploads/2026/09/1-10-Full-corrected.pdf
27Disallow: /wp-content/uploads/2026/09/1.10-corrected.xlsx
28Disallow: /wp-content/uploads/2026/09/1-10-corrected.csv
29Disallow: /wp-content/uploads/2026/08/Brookings_ECDP_Indicators.pdf
30Disallow: /wp-content/uploads/2026/08/20260824_IH_Survey_items.pdf
31Disallow: /wp-content/uploads/2026/08/20260824_IH_Appendix.pdf
32Disallow: /wp-content/uploads/2026/08/GS_20260817_TechTankPodcast_cyber_reslience2.pdf
33Disallow: /wp-content/uploads/2026/07/GS_20260804_TechTank_AI_toddlers.pdf
34Disallow: /wp-content/uploads/2026/07/GS_20260721_TechTankPodcast_space.pdf
35Disallow: /wp-content/uploads/2026/07/Updated_Elements_ECDP-1.pdf
36Disallow: /wp-content/uploads/2026/07/Updated_Elements_ECDP.pdf
37Disallow: /wp-content/uploads/2026/07/Roundtable_3_Summary_BrookingsECDP.pdf
38Disallow: /wp-content/uploads/2026/07/20260706_Uprooted_recs.pdf
39Disallow: /wp-content/uploads/2026/07/GS_20260706_TechTankPodcast_manufacturing.pdf
40Disallow: /wp-content/uploads/2026/06/GS_20260615_TechTankPodcast_AI_democracy.docx
41Disallow: /wp-content/uploads/2026/01/A-New-Direction-for-Students-in-an-AI-World-ONE-PAGER.pdf
42Disallow: /wp-content/uploads/2026/06/20260604_chapter3_culture_wars_appendix2.pdf
43Disallow: /wp-content/uploads/2026/06/20260604_chapter2_culture_wars_appendix2.pdf
44Disallow: /wp-content/uploads/2026/06/20260604_chapter1_culture_wars_appendix2.pdf
45Disallow: /wp-content/uploads/2016/02/sarahbindercv_june_2026.pdf
46Disallow: /wp-content/uploads/2022/08/2026_resume_Courtney_Radsch.pdf
47Disallow: /wp-content/uploads/2026/06/Survey_Tables_NoWeights_FINAL.pdf
48Disallow: /wp-content/uploads/2026/06/Survey_Tables_Weights_FINAL.pdf
49Disallow: /wp-content/uploads/2026/05/Roundtable_1_Summary_ECDP.pdf
50Disallow: /wp-content/uploads/2026/05/Roundtable_2_Summary_ECDP.pdf
51Disallow: /wp-content/uploads/2026/05/Summary_Brookings_EquitableCommunityDataPathways.pdf
52Disallow: /wp-content/uploads/2026/05/Survey_Methodology_FINAL.pdf
53Disallow: /wp-content/uploads/2026/04/GS_20260504_TechTankPodcast_connectedcars.pdf
54Disallow: /wp-content/uploads/2026/04/20260430_statewide_programs_appendix-2.pdf
55Disallow: /wp-content/uploads/2026/04/GS_20260420_TechTankPodcast_chatbots.pdf
56Disallow: /wp-content/uploads/2026/04/20260416_Uprooted_recs226.pdf
57Disallow: /wp-content/uploads/2026/04/4-2.csv
58Disallow: /wp-content/uploads/2026/04/Vital_Stats_Errata_April_2026.pdf
59Disallow: /wp-content/uploads/2026/04/6-7-Full.pdf
60Disallow: /wp-content/uploads/2026/04/6-6-Full.pdf
61Disallow: /wp-content/uploads/2026/04/6-5-Full.pdf
62Disallow: /wp-content/uploads/2026/04/6-4-Full.pdf
63Disallow: /wp-content/uploads/2026/04/6-2-Full.pdf
64Disallow: /wp-content/uploads/2026/04/6-3-Full.pdf
65Disallow: /wp-content/uploads/2026/04/6-1-Full.pdf
66Disallow: /wp-content/uploads/2026/04/4-8-Full.pdf
67Disallow: /wp-content/uploads/2026/04/4-7-Full.pdf
68Disallow: /wp-content/uploads/2026/04/4-6-Full.pdf
69Disallow: /wp-content/uploads/2026/04/4-5-Full.pdf
70Disallow: /wp-content/uploads/2026/04/4-4-Full.pdf
71Disallow: /wp-content/uploads/2026/04/4-3-Full.pdf
72Disallow: /wp-content/uploads/2026/04/4-2-Full.pdf
73Disallow: /wp-content/uploads/2026/04/4-1-Full.pdf
74Disallow: /wp-content/uploads/2026/04/2-19-Full.pdf
75Disallow: /wp-content/uploads/2026/04/2-18-Full.pdf
76Disallow: /wp-content/uploads/2026/04/2-17-Full.pdf
77Disallow: /wp-content/uploads/2026/04/2-16-Full.pdf
78Disallow: /wp-content/uploads/2026/04/2-15-Full.pdf
79Disallow: /wp-content/uploads/2026/04/2-14-Full.pdf
80Disallow: /wp-content/uploads/2026/04/2-13-Full.pdf
81Disallow: /wp-content/uploads/2026/04/2-12-Full.pdf
82Disallow: /wp-content/uploads/2026/04/2-10-Full.pdf
83Disallow: /wp-content/uploads/2026/04/2-11-Full.pdf
84Disallow: /wp-content/uploads/2026/04/2-9-Full.pdf
85Disallow: /wp-content/uploads/2026/04/2-8-Full.pdf
86Disallow: /wp-content/uploads/2026/04/2-7-Full.pdf
87Disallow: /wp-content/uploads/2026/04/2-6-Full.pdf
88Disallow: /wp-content/uploads/2026/04/2-5-Full.pdf
89Disallow: /wp-content/uploads/2026/04/2-4-Full.pdf
90Disallow: /wp-content/uploads/2026/04/2-3-Full.pdf
91Disallow: /wp-content/uploads/2026/04/2-2-Full.pdf
92Disallow: /wp-content/uploads/2026/04/2-1-Full.pdf
93Disallow: /wp-content/uploads/2026/04/1-1-Full.pdf
94Disallow: /wp-content/uploads/2026/04/1-2-Full.pdf
95Disallow: /wp-content/uploads/2026/04/1-3-Full.pdf
96Disallow: /wp-content/uploads/2026/04/1-4-Full.pdf
97Disallow: /wp-content/uploads/2026/04/1-5-Full.pdf
98Disallow: /wp-content/uploads/2026/04/1-6-Full.pdf
99Disallow: /wp-content/uploads/2026/04/1-7-Full.pdf
100Disallow: /wp-content/uploads/2026/04/1-8-Full.pdf
101Disallow: /wp-content/uploads/2026/04/1-9-Full.pdf
102Disallow: /wp-content/uploads/2026/04/1-10-Full.pdf
103Disallow: /wp-content/uploads/2026/04/1-11-Full.pdf
104Disallow: /wp-content/uploads/2026/04/1-12-Full.pdf
105Disallow: /wp-content/uploads/2026/04/1-13-Full.pdf
106Disallow: /wp-content/uploads/2026/04/1-14-Full.pdf
107Disallow: /wp-content/uploads/2026/04/1-15-Full.pdf
108Disallow: /wp-content/uploads/2026/04/1-16-Full.pdf
109Disallow: /wp-content/uploads/2026/04/1-17-Full.pdf
110Disallow: /wp-content/uploads/2026/04/1-18-Full.pdf
111Disallow: /wp-content/uploads/2026/04/1-19-Full.pdf
112Disallow: /wp-content/uploads/2026/04/1-20-Full.pdf
113Disallow: /wp-content/uploads/2026/04/1-21-Full.pdf
114Disallow: /wp-content/uploads/2026/04/1-22-Full.pdf
115Disallow: /wp-content/uploads/2026/04/1-23-Full.pdf
116Disallow: /wp-content/uploads/2026/04/6-1.xlsx
117Disallow: /wp-content/uploads/2026/04/6-2.xlsx
118Disallow: /wp-content/uploads/2026/04/6-3.xlsx
119Disallow: /wp-content/uploads/2026/04/6-4.xlsx
120Disallow: /wp-content/uploads/2026/04/6-5.xlsx
121Disallow: /wp-content/uploads/2026/04/6-6.xlsx
122Disallow: /wp-content/uploads/2026/04/6-7.xlsx
123Disallow: /wp-content/uploads/2026/04/4-1.xlsx
124Disallow: /wp-content/uploads/2026/04/4-2.xlsx
125Disallow: /wp-content/uploads/2026/04/4-3.xlsx
126Disallow: /wp-content/uploads/2026/04/4-4.xlsx
127Disallow: /wp-content/uploads/2026/04/4-5.xlsx
128Disallow: /wp-content/uploads/2026/04/4-6.xlsx
129Disallow: /wp-content/uploads/2026/04/4-7.xlsx
130Disallow: /wp-content/uploads/2026/04/4-8.xlsx
131Disallow: /wp-content/uploads/2026/04/2-1.xlsx
132Disallow: /wp-content/uploads/2026/04/2-2.xlsx
133Disallow: /wp-content/uploads/2026/04/2-3.xlsx
134Disallow: /wp-content/uploads/2026/04/2-4.xlsx
135Disallow: /wp-content/uploads/2026/04/2-5.xlsx
136Disallow: /wp-content/uploads/2026/04/2-6.xlsx
137Disallow: /wp-content/uploads/2026/04/2-7.xlsx
138Disallow: /wp-content/uploads/2026/04/2-8.xlsx
139Disallow: /wp-content/uploads/2026/04/2-9.xlsx
140Disallow: /wp-content/uploads/2026/04/2-10.xlsx
141Disallow: /wp-content/uploads/2026/04/2-11.xlsx
142Disallow: /wp-content/uploads/2026/04/2-12.xlsx
143Disallow: /wp-content/uploads/2026/04/2-13.xlsx
144Disallow: /wp-content/uploads/2026/04/2-14.xlsx
145Disallow: /wp-content/uploads/2026/04/2-15.xlsx
146Disallow: /wp-content/uploads/2026/04/2-16.xlsx
147Disallow: /wp-content/uploads/2026/04/2-17.xlsx
148Disallow: /wp-content/uploads/2026/04/2-18.xlsx
149Disallow: /wp-content/uploads/2026/04/2-19.xlsx
150Disallow: /wp-content/uploads/2026/04/1.1.xlsx
151Disallow: /wp-content/uploads/2026/04/1.2.xlsx
152Disallow: /wp-content/uploads/2026/04/1.3.xlsx
153Disallow: /wp-content/uploads/2026/04/1.4.xlsx
154Disallow: /wp-content/uploads/2026/04/1.5.xlsx
155Disallow: /wp-content/uploads/2026/04/1.6.xlsx
156Disallow: /wp-content/uploads/2026/04/1.7.xlsx
157Disallow: /wp-content/uploads/2026/04/1.8.xlsx
158Disallow: /wp-content/uploads/2026/04/1.9.xlsx
159Disallow: /wp-content/uploads/2026/04/1.10.xlsx
160Disallow: /wp-content/uploads/2026/04/1.11.xlsx
161Disallow: /wp-content/uploads/2026/04/1.12.xlsx
162Disallow: /wp-content/uploads/2026/04/1.13.xlsx
163Disallow: /wp-content/uploads/2026/04/1.14.xlsx
164Disallow: /wp-content/uploads/2026/04/1.15.xlsx
165Disallow: /wp-content/uploads/2026/04/1.16.xlsx
166Disallow: /wp-content/uploads/2026/04/1.17.xlsx
167Disallow: /wp-content/uploads/2026/04/1.18.xlsx
168Disallow: /wp-content/uploads/2026/04/1.19.xlsx
169Disallow: /wp-content/uploads/2026/04/1.20.xlsx
170Disallow: /wp-content/uploads/2026/04/1.21.xlsx
171Disallow: /wp-content/uploads/2026/04/1.22.xlsx
172Disallow: /wp-content/uploads/2026/04/1.23.xlsx
173Disallow: /wp-content/uploads/2026/04/6-1.csv
174Disallow: /wp-content/uploads/2026/04/6-2.csv
175Disallow: /wp-content/uploads/2026/04/6-3.csv
176Disallow: /wp-content/uploads/2026/04/6-4.csv
177Disallow: /wp-content/uploads/2026/04/6-5.csv
178Disallow: /wp-content/uploads/2026/04/6-6.csv
179Disallow: /wp-content/uploads/2026/04/6-7.csv
180Disallow: /wp-content/uploads/2026/04/4-8.csv
181Disallow: /wp-content/uploads/2026/04/4-6.csv
182Disallow: /wp-content/uploads/2026/04/4-7.csv
183Disallow: /wp-content/uploads/2026/04/4-5.csv
184Disallow: /wp-content/uploads/2026/04/4-3.csv
185Disallow: /wp-content/uploads/2026/04/4-4.csv
186Disallow: /wp-content/uploads/2026/04/4-1-.csv
187Disallow: /wp-content/uploads/2026/04/2-18.csv
188Disallow: /wp-content/uploads/2026/04/2-19.csv
189Disallow: /wp-content/uploads/2026/04/2-17.csv
190Disallow: /wp-content/uploads/2026/04/2-16.csv
191Disallow: /wp-content/uploads/2026/04/2-14.csv
192Disallow: /wp-content/uploads/2026/04/2-15.csv
193Disallow: /wp-content/uploads/2026/04/2-13.csv
194Disallow: /wp-content/uploads/2026/04/2-11.csv
195Disallow: /wp-content/uploads/2026/04/2-12.csv
196Disallow: /wp-content/uploads/2026/04/2-10.csv
197Disallow: /wp-content/uploads/2026/04/2-9.csv
198Disallow: /wp-content/uploads/2026/04/2-7.csv
199Disallow: /wp-content/uploads/2026/04/2-8.csv
200Disallow: /wp-content/uploads/2026/04/2-6.csv
201Disallow: /wp-content/uploads/2026/04/2-5.csv
202Disallow: /wp-content/uploads/2026/04/2-3.csv
203Disallow: /wp-content/uploads/2026/04/2-4.csv
204Disallow: /wp-content/uploads/2026/04/2-2.csv
205Disallow: /wp-content/uploads/2026/04/2-1.csv
206Disallow: /wp-content/uploads/2026/04/1-1.csv
207Disallow: /wp-content/uploads/2026/04/1-2.csv
208Disallow: /wp-content/uploads/2026/04/1-3.csv
209Disallow: /wp-content/uploads/2026/04/1-4.csv
210Disallow: /wp-content/uploads/2026/04/1-5.csv
211Disallow: /wp-content/uploads/2026/04/1-6.csv
212Disallow: /wp-content/uploads/2026/04/1-7.csv
213Disallow: /wp-content/uploads/2026/04/1-8.csv
214Disallow: /wp-content/uploads/2026/04/1-9.csv
215Disallow: /wp-content/uploads/2026/04/1-10.csv
216Disallow: /wp-content/uploads/2026/04/1-12.csv
217Disallow: /wp-content/uploads/2026/04/1-11.csv
218Disallow: /wp-content/uploads/2026/04/1-13.csv
219Disallow: /wp-content/uploads/2026/04/1-14.csv
220Disallow: /wp-content/uploads/2026/04/1-15.csv
221Disallow: /wp-content/uploads/2026/04/1-16.csv
222Disallow: /wp-content/uploads/2026/04/1-17.csv
223Disallow: /wp-content/uploads/2026/04/1-18.csv
224Disallow: /wp-content/uploads/2026/04/1-19.csv
225Disallow: /wp-content/uploads/2026/04/1-20.csv
226Disallow: /wp-content/uploads/2026/04/1-21.csv
227Disallow: /wp-content/uploads/2026/04/1-23.csv
228Disallow: /wp-content/uploads/2026/04/Chapter-1-Full.pdf
229Disallow: /wp-content/uploads/2026/04/Chapter-4-Full.pdf
230Disallow: /wp-content/uploads/2026/04/Chapter-6-Full.pdf
231Disallow: /wp-content/uploads/2026/04/Chpt-2.xlsx
232Disallow: /wp-content/uploads/2026/04/Chpt-4.xlsx
233Disallow: /wp-content/uploads/2026/04/Chpt-6.xlsx
234Disallow: /wp-content/uploads/2026/04/Chpt-1.xlsx
235Disallow: /wp-content/uploads/2026/04/AndersonScottR_CV-2026.04.02.pdf
236Disallow: /wp-content/uploads/2026/02/20260215_AI_sovereignty.pdf
237Disallow: /wp-content/uploads/2026/01/Empowering-Rural-America-Through-Investment-in-Innovation-Final-Testimony_Turner-Lee_1.21.26.pdf
238Disallow: /wp-content/uploads/2026/01/GS_20260122_TechTankPodcast_data-center-water_transcript.pdf
239Disallow: /wp-content/uploads/2026/01/A-New-Direction-for-Students-in-an-AI-World-FULL-REPORT.pdf
240Disallow: /wp-content/uploads/2025/12/GS_20251208_TechTankPodcast_wired-wisdom_transcript-3.pdf
241Disallow: /wp-content/uploads/2025/12/GS_20251208_TechTankPodcast_wired-wisdom_transcript-2.pdf
242Disallow: /wp-content/uploads/2025/12/GS_20251124_TechTankPodcast_data_centers_transcript-1.docx
243Disallow: /wp-content/uploads/2025/11/GS_20251124_TechTankPodcast_data_centers_transcript.pdf
244Disallow: /wp-content/uploads/2025/11/New_MOU_July_2025.pdf
245Disallow: /wp-content/uploads/2016/09/Valant_CV_Nov2025.pdf
246Disallow: /wp-content/uploads/2016/06/Puentes-cv-November-2025.pdf
247Disallow: /wp-content/uploads/2025/10/Appendix.pdf
248Disallow: /wp-content/uploads/2025/10/GS_20251020_TechTankPodcast_AI_banking_transcript.pdf
249Disallow: /wp-content/uploads/2020/08/GS_2025_10_27_Kimberly_Jenkins_Robinson_CV.pdf
250Disallow: /wp-content/uploads/2016/06/GS_2025_10_19_Pasachoff_Brookings-CV.pdf
251Disallow: /wp-content/uploads/2020/10/GS_20251001_Edward_Vargas_CV.doc
252Disallow: /wp-content/uploads/2017/02/GS_20250929_Rachel_Augustine_Potter_CV.pdf
253Disallow: /wp-content/uploads/2025/09/GS20250929_Barker_CV.docx
254Disallow: /wp-content/uploads/2015/12/GS_20250923_John_Villasenor.pdf
255Disallow: /wp-content/uploads/2021/01/GS_09112025_David_Damore_CV.pdf
256Disallow: /wp-content/uploads/2021/05/GS_20250910_Alan_Rozenshtein_CV.pdf
257Disallow: /wp-content/uploads/2025/09/Corps-National-Brief-Public-Data-Download-FINAL.xlsx
258Disallow: /wp-content/uploads/2025/09/Corps-Careers-appendix.pdf
259Disallow: /wp-content/uploads/2021/08/GS_09022025_DanaFisherCV.pdf
260Disallow: /wp-content/uploads/2025/08/Internal-Revenue-Service-Data-8-15-2025.pdf
261Disallow: /wp-content/uploads/2016/03/GS_20250805_Molly_Reynolds_CV.pdf
262Disallow: /wp-content/uploads/2023/09/GS_07302025_Chinasa_Okolo_CV.pdf
263Disallow: /wp-content/uploads/2025/07/IRSDATA_07_23_2025-TaxGap.pdf
264Disallow: /wp-content/uploads/2025/06/Mapping-AI-readiness-v1-ES.pdf
265Disallow: /wp-content/uploads/2025/06/Mapping-AI-readiness-final.pdf
266Disallow: /wp-content/uploads/2025/06/the-invisible-infrastructure-of-inclusive-economic-growth.pdf
267Disallow: /wp-content/uploads/2025/06/Building-resilience-slide-deck.pdf
268Disallow: /wp-content/uploads/2016/08/Joseph-W-Kane-CV-Updated-June-2025.pdf
269Disallow: /wp-content/uploads/2016/06/04112025_kevin_desouzacv_2017.pdf
270Disallow: /wp-content/uploads/2023/09/Okolo-Chinasa-T.-Brookings-CV.pdf
271Disallow: /wp-content/uploads/2025/01/v4_democracy-playbook-2025.pdf
272Disallow: /wp-content/uploads/2025/01/v3_Democracy-Playbook-2025_Seven-Pillars-to-Defend-Democracy.pdf
273Disallow: /wp-content/uploads/2025/01/v2_Democracy-Playbook-2025_Seven-Pillars.pdf
274Disallow: /wp-content/uploads/2025/01/Democracy-Playbook-2025—Seven-Pillars-to-Defend-Democracy-FINAL.pdf
275Disallow: /wp-content/uploads/2024/12/GS_12112024_Barry_Rabe_CV_2024.pdf
276Disallow: /wp-content/uploads/2024/12/GS_12112024_Raj-Korpan-CV.pdf
277Disallow: /wp-content/uploads/2024/12/GS_12112024_raymondlaraja_cv.pdf
278Disallow: /wp-content/uploads/2020/11/GS_11062024_Joseph_Foti_CV.pdf
279Disallow: /wp-content/uploads/2022/08/01112024_GS_Meyer_CV.docx
280Disallow: /wp-content/uploads/2023/10/Atyia-All-Aces-Resume_CV.pdf
281Disallow: /wp-content/uploads/2023/09/GS_09252023_Andrew-Wyckoff_CV.pdf
282Disallow: /wp-content/uploads/2016/02/GS_07072023_SarahBinder-CV.pdf
283Disallow: /wp-content/uploads/2021/07/Keon_Gilbert_CV-Jan-2021.pdf
284Disallow: /wp-content/uploads/2021/01/Brookings-MacCarthy-CV-2021-Final.doc
285Disallow: /wp-content/uploads/2020/09/Kenneth-Wong_Brown-U_6_2020.pdf
286Disallow: /wp-content/uploads/2020/01/Bill-Baer-Bio_Jan-2020.pdf
287Disallow: /wp-content/uploads/2017/01/Melissa-Rogers-Resume-09-19.doc
288Disallow: /wp-content/uploads/2017/01/Melissa-Rogers-Resume-06-19-no-contact-info.pdf
289Disallow: /wp-content/uploads/2017/01/Melissa-Rogers-1-19.pdf
290Disallow: /wp-content/uploads/2016/04/reeves_cv.pdf
291Disallow: /wp-content/uploads/2016/04/20170119_ej_cv.pdf
292Disallow: /wp-content/uploads/2016/07/20160725_vita-extended-Douglas_Harris38.pdf
293Disallow: /wp-content/uploads/2016/07/Rauch_Jonathan_CV_07Jan2016.pdf
294# End disallow_search_engine_indexing_of_attachments block
295
296# START YOAST BLOCK
297# ---------------------------
298User-agent: *
299Disallow:
300
301Sitemap: https://www.brookings.edu/sitemap_index.xml
302# ---------------------------
303# END YOAST BLOCK
What do the bots actually get?
This page reads robots.txt, which states what brookings.edu asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on brookings.edu Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does brookings.edu block GPTBot?
No. brookings.edu’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does brookings.edu block ClaudeBot?
No. brookings.edu’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does brookings.edu block Google-Extended?
No. brookings.edu’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does brookings.edu block PerplexityBot?
No. brookings.edu’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find brookings.edu?
Yes. brookings.edu’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.