Who blocks AI · #379 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does psu.edu block AI crawlers?
No. psu.edu’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.psu.edu/robots.txt · 460 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | line 430 · Allow: / under User-agent: * | |
Bingbot | Microsoft | Allowed | line 430 · Allow: / under User-agent: * |
DuckDuckBot | DuckDuckGo | Allowed | line 430 · Allow: / under User-agent: * |
Applebot | Apple | Allowed | line 430 · Allow: / under User-agent: * |
PetalBot | Huawei | Allowed | line 430 · Allow: / under User-agent: * |
YandexBot | Yandex | Allowed | line 430 · Allow: / under User-agent: * |
Baiduspider | Baidu | Allowed | line 430 · Allow: / under User-agent: * |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | line 430 · Allow: / under User-agent: * |
PerplexityBot | Perplexity | Allowed | line 457 · Allow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Allowed | line 430 · Allow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Allowed | line 430 · Allow: / under User-agent: * |
Amazonbot | Amazon | Allowed | line 445 · Allow: / under User-agent: Amazonbot |
YouBot | You.com | Allowed | line 430 · Allow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | line 433 · Allow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Allowed | line 451 · Allow: / under User-agent: ClaudeBot |
Google-Extended | Allowed | line 439 · Allow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Allowed | line 442 · Allow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | line 430 · Allow: / under User-agent: * |
GoogleOther | Allowed | line 430 · Allow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Allowed | line 430 · Allow: / under User-agent: * |
Bytespider | ByteDance | Allowed | line 448 · Allow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | line 436 · Allow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Allowed | line 430 · Allow: / under User-agent: * |
Perplexity-User | Perplexity | Allowed | line 430 · Allow: / under User-agent: * |
MistralAI-User | Mistral | Allowed | line 430 · Allow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Allowed | line 430 · Allow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | line 430 · Allow: / under User-agent: * |
SemrushBot | Semrush | Allowed | line 430 · Allow: / under User-agent: * |
MJ12bot | Majestic | Allowed | line 430 · Allow: / under User-agent: * |
DotBot | Moz | Allowed | line 430 · Allow: / under User-agent: * |
View www.psu.edu/robots.txt · 460 lines
1User-agent: *
2Disallow: /search
3Disallow: /404
4Disallow: /api/
5Disallow: /admission/undergraduate/forms/bk-safd-participation
6Disallow: /admission/undergraduate/explore/campuses/2plus2-plan
7Disallow: /about/our-impact/spotlight
8Disallow: /admission/undergraduate/forms/pmm-questions
9Disallow: /admission/undergraduate/explore/visit-penn-state/sasd-university-park
10Disallow: /admission/undergraduate/forms/bus-trip-participation
11Disallow: /costs-aid/this-is-a-test
12Disallow: /admission/undergraduate/explore/visit-penn-state/2plus2-ASE
13Disallow: /admission/undergraduate/forms/letsride-phl
14Disallow: /admission/undergraduate/forms/letsride-dc
15Disallow: /admission/undergraduate/forms/letsride-ctr
16Disallow: /admission/undergraduate/forms/letsride-pgh
17Disallow: /admission/undergraduate/forms/letsride-hbrg
18Disallow: /admission/undergraduate/forms/aavp-welcome-binder
19Disallow: /admission/undergraduate/forms/vamonos-centre
20Disallow: /admission/undergraduate/forms/vamonos-hbg-rdg
21Disallow: /admission/undergraduate/forms/vamonos-md-dc
22Disallow: /admission/undergraduate/forms/vamonos-pgh
23Disallow: /admission/undergraduate/forms/vamonos-phl
24Disallow: /academics/undergraduate/majors/rail-transportation-engineering
25Disallow: /resources/admitted-students/enrolling
26Disallow: /resources/admitted-students/likely
27Disallow: /resources/admitted-students/undecided
28Disallow: /admission/undergraduate/explore/visit-penn-state/achievers
29Disallow: /legal-statements
30Disallow: /costs-aid/forms/dependency-review
31Disallow: /costs-aid/forms/prior-year-dependency-review
32Disallow: /costs-aid/forms/unaccompanied-youth
33Disallow: /costs-aid/forms/parent-date-of-birth-confirmation-fpldob
34Disallow: /costs-aid/forms/student-identity-verification-fidedn
35Disallow: /costs-aid/forms/commonwealth-appeal-form
36Disallow: /costs-aid/forms/quake-release-2027
37Disallow: /costs-aid/forms/karli-short-better-tomorrow-scholarship-application
38Disallow: /costs-aid/forms/student-marital-status-f27smr
39Disallow: /costs-aid/forms/laurel-haven-endowment-scholarship-application-form
40Disallow: /costs-aid/forms/smithfield-foods-inc-scholarship-form
41Disallow: /costs-aid/forms/student-acknowledgment-ineligibility-loans
42Disallow: /costs-aid/forms/parent-marital-status-change-form-fscefd
43Disallow: /costs-aid/forms/dependency-review-unsub-request
44Disallow: /costs-aid/forms/f26pew
45Disallow: /costs-aid/forms/parent-tax-filer-f27ptx
46Disallow: /costs-aid/forms/student-tax-filer-f27stx
47Disallow: /costs-aid/forms/parent-foreign-tax-2018
48Disallow: /costs-aid/forms/parent-foreign-tax-f27ptx
49Disallow: /costs-aid/forms/student-rollover-f26srl
50Disallow: /costs-aid/forms/student-tax-confirmation-f27sta
51Disallow: /costs-aid/forms/fhmls-2026
52Disallow: /costs-aid/forms/sabr
53Disallow: /costs-aid/forms/f26srl
54Disallow: /costs-aid/forms/f26sfi
55Disallow: /costs-aid/forms/f26ied
56Disallow: /costs-aid/forms/f26stx-tax-filer
57Disallow: /costs-aid/forms/f26sta
58Disallow: /costs-aid/forms/f26hsc
59Disallow: /costs-aid/forms/f26sha
60Disallow: /costs-aid/forms/student-income-review-f26sia
61Disallow: /costs-aid/forms/f26sia
62Disallow: /costs-aid/forms/p26pta
63Disallow: /costs-aid/forms/f26ptx-foreign-tax
64Disallow: /costs-aid/forms/f26smr
65Disallow: /costs-aid/forms/f26pmr
66Disallow: /costs-aid/forms/f26stx-non-filer
67Disallow: /costs-aid/forms/f26prl
68Disallow: /costs-aid/forms/f26pha
69Disallow: /costs-aid/forms/icoa
70Disallow: /costs-aid/forms/wdse
71Disallow: /costs-aid/forms/parent-child-support-received-2021
72Disallow: /costs-aid/forms/student-foreign-tax-2021
73Disallow: /costs-aid/forms/private-loan/self-certification
74Disallow: /costs-aid/forms/parent-household-additional-information-2024
75Disallow: /costs-aid/forms/parent-household-additional-information-2025
76Disallow: /costs-aid/forms/parent-earnings-2022
77Disallow: /costs-aid/forms/2025/parent-rollover-2022
78Disallow: /costs-aid/forms/parent-taxable-grants-scholarships-2022
79Disallow: /costs-aid/forms/student-child-support-received-2021
80Disallow: /costs-aid/forms/student-assets-2024
81Disallow: /costs-aid/forms/student-veterans-non-ed-benefits-2021
82Disallow: /costs-aid/forms/student-other-untaxed-income-2021
83Disallow: /costs-aid/forms/student-taxable-grants-scholarships-2021
84Disallow: /costs-aid/forms/student-tax-deferred-savings-2021
85Disallow: /costs-aid/forms/student-unreported-income-2021
86Disallow: /costs-aid/forms/student-federal-benefits-2021-2022
87Disallow: /costs-aid/forms/student-employment-2021
88Disallow: /costs-aid/forms/student-housing-allowance-2021
89Disallow: /costs-aid/forms/student-tax-filers-2021
90Disallow: /costs-aid/forms/student-non-filers-2021
91Disallow: /costs-aid/forms/student-coop-earnings-2021
92Disallow: /costs-aid/forms/student-combat-pay-2021
93Disallow: /costs-aid/forms/student-child-support-paid-additional-information-2021
94Disallow: /costs-aid/forms/student-child-support-2021
95Disallow: /costs-aid/forms/student-household-additional-information-2024
96Disallow: /costs-aid/forms/student-current-income-review-2021
97Disallow: /costs-aid/forms/parent-assets-2024
98Disallow: /costs-aid/forms/parent-veterans-non-educational-benefits-2021
99Disallow: /costs-aid/forms/parent-untaxed-income-2021
100Disallow: /costs-aid/forms/parent-rollover-2021
101Disallow: /costs-aid/forms/parent-tax-deferred-savings-2021
102Disallow: /costs-aid/forms/parent-employment-2021
103Disallow: /costs-aid/forms/parent-housing-allowance-2021
104Disallow: /costs-aid/forms/parent-tax-filers-2021
105Disallow: /costs-aid/forms/parent-tax-non-filers-2021
106Disallow: /costs-aid/forms/parent-earnings-2021
107Disallow: /costs-aid/forms/parent-coop-earnings-2021
108Disallow: /costs-aid/forms/parent-combat-pay-2021
109Disallow: /costs-aid/forms/parent-child-support-paid-additional-information-2021
110Disallow: /costs-aid/forms/parent-tax-additional-information-2021
111Disallow: /costs-aid/forms/student-identity-education-purpose-2024
112Disallow: /costs-aid/forms/parent-marital-status-2023
113Disallow: /costs-aid/forms/sap-appeal-third-party-documentation
114Disallow: /costs-aid/forms/student-assets-2023
115Disallow: /costs-aid/forms/student-veterans-noned-benefits-2020
116Disallow: /costs-aid/forms/student-other-untaxed-income-2020
117Disallow: /costs-aid/forms/student-taxable-grants-scholarships-2020
118Disallow: /costs-aid/forms/student-rollover-2020
119Disallow: /costs-aid/forms/student-tax-deferred-savings-2020
120Disallow: /costs-aid/forms/student-foreign-tax-2020
121Disallow: /costs-aid/forms/student-non-filers-2020
122Disallow: /costs-aid/forms/student-tax-filers2020
123Disallow: /costs-aid/forms/student-tax-additional-information-2020
124Disallow: /costs-aid/forms/student-coop-earnings-2020
125Disallow: /costs-aid/forms/student-combat-pay-2020
126Disallow: /costs-aid/forms/student-child-support-paid-additional-information-2020
127Disallow: /costs-aid/forms/student-child-support-2020
128Disallow: /costs-aid/forms/student-marital-status-2023
129Disallow: /costs-aid/forms/student-homeless-confirmation-2023
130Disallow: /costs-aid/forms/student-current-income-review
131Disallow: /costs-aid/forms/student-child-support-received-2020
132Disallow: /costs-aid/forms/parent-assets-2023
133Disallow: /costs-aid/forms/parent-tax-deferred-savings-2020
134Disallow: /costs-aid/forms/parent-employment-2019
135Disallow: /costs-aid/forms/parent-housing-allowance-2020
136Disallow: /costs-aid/forms/parent-foreign-tax-2020
137Disallow: /costs-aid/forms/student-identity-education-purpose-2023
138Disallow: /costs-aid/forms/student-spouse-assets
139Disallow: /costs-aid/forms/student-veterans-non-ed-benefits-2019
140Disallow: /costs-aid/forms/student-other-untaxed-income-2019
141Disallow: /costs-aid/forms/student-taxable-grants-scholarships-2019
142Disallow: /costs-aid/forms/student-rollover-2019
143Disallow: /costs-aid/forms/student-unreported-income-2019
144Disallow: /costs-aid/forms/student-federal-benefits-2019-2020
145Disallow: /costs-aid/forms/student-employment-2019
146Disallow: /costs-aid/forms/student-income-additional-information-2019
147Disallow: /costs-aid/forms/student-foreign-tax-2019
148Disallow: /costs-aid/forms/student-tax-filers-2019
149Disallow: /costs-aid/forms/student-tax-non-filers-2019
150Disallow: /costs-aid/forms/student-tax-additional-information-2019
151Disallow: /costs-aid/forms/student-earnings-2019
152Disallow: /costs-aid/forms/student-coop-earnings-2019
153Disallow: /costs-aid/forms/student-combat-pay-2019
154Disallow: /costs-aid/forms/student-child-support-paid-additional-information-2019
155Disallow: /costs-aid/forms/student-child-support-2019
156Disallow: /costs-aid/forms/student-marital-status-2022
157Disallow: /costs-aid/forms/student-homeless-confirmation-2022
158Disallow: /costs-aid/forms/student-child-support-received-2019
159Disallow: /costs-aid/forms/parent-marital-status-2022
160Disallow: /costs-aid/forms/parent-veterans-non-educational-benefits-2019
161Disallow: /costs-aid/forms/parent-assets-2022
162Disallow: /costs-aid/forms/parent-untaxed-income-2019
163Disallow: /costs-aid/forms/parent-taxable-grants-scholarships-2019
164Disallow: /costs-aid/forms/parent-rollover-2019
165Disallow: /costs-aid/forms/parent-tax-deferred-savings-2019
166Disallow: /costs-aid/forms/parent-employment-earnings-2019
167Disallow: /costs-aid/forms/parent-housing-allowance-2019
168Disallow: /costs-aid/forms/student-tax-additional-information-2018
169Disallow: /costs-aid/forms/student-rollover-2018
170Disallow: /costs-aid/forms/student-unreported-income-2018
171Disallow: /costs-aid/forms/student-employment-2018
172Disallow: /costs-aid/forms/student-income-additional-information-2018
173Disallow: /costs-aid/forms/student-housing-allowance-2018
174Disallow: /costs-aid/forms/student-federal-benefits-2019
175Disallow: /costs-aid/forms/student-earnings-2018
176Disallow: /costs-aid/forms/student-child-support-received-2018
177Disallow: /costs-aid/forms/student-child-support-2018
178Disallow: /costs-aid/forms/student-coop-earnings-2018
179Disallow: /costs-aid/forms/student-combat-pay-2018
180Disallow: /costs-aid/forms/student-child-support-paid-additional-information-2018
181Disallow: /costs-aid/forms/student-assets
182Disallow: /costs-aid/forms/parent-veterans-non-educational-benefits-2018
183Disallow: /costs-aid/forms/parent-tax-non-filers-2018
184Disallow: /costs-aid/forms/parent-taxable-grants-scholarships-2018
185Disallow: /costs-aid/forms/parent-tax-deferred-savings-2018
186Disallow: /costs-aid/forms/parent-tax-additional-information
187Disallow: /costs-aid/forms/parent-employment-2018
188Disallow: /costs-aid/forms/parent-housing-allowance-2018
189Disallow: /costs-aid/forms/parent-earnings-2018
190Disallow: /costs-aid/forms/parent-child-support-received-2018
191Disallow: /costs-aid/forms/parent-child-support-2018
192Disallow: /costs-aid/forms/parent-coop-earnings-2018
193Disallow: /costs-aid/forms/parent-combat-pay-2018
194Disallow: /costs-aid/forms/parent-child-support-paid-additional-information-2018
195Disallow: /costs-aid/forms/parent-assets
196Disallow: /costs-aid/forms/student-identity-education-purpose
197Disallow: /costs-aid/forms/student-identity-education-purpose-2020
198Disallow: /costs-aid/forms/parent-assets-2
199Disallow: /costs-aid/forms/parent-child-support-paid-additional-information-2017
200Disallow: /costs-aid/forms/parent-combat-pay-2017
201Disallow: /costs-aid/forms/parent-coop-earnings-2017
202Disallow: /costs-aid/forms/parent-child-support-2017
203Disallow: /costs-aid/forms/parent-taxable-grants-scholarships-2017
204Disallow: /costs-aid/forms/parent-child-support-received-2017
205Disallow: /costs-aid/forms/parent-marital-status-2020
206Disallow: /costs-aid/forms/parent-tax-deferred-savings-2017
207Disallow: /costs-aid/forms/parent-rollover-2017
208Disallow: /costs-aid/forms/parent-employment-2017
209Disallow: /costs-aid/forms/parent-housing-allowance-2017
210Disallow: /costs-aid/forms/parent-earnings-2017
211Disallow: /costs-aid/forms/parent-taxable-grants-scholarships-2021
212Disallow: /costs-aid/forms/parent-tax-filers-2017
213Disallow: /costs-aid/forms/parent-foreign-tax-2017
214Disallow: /costs-aid/forms/parent-tax-non-filers-2017
215Disallow: /costs-aid/forms/parent-untaxed-income-2017
216Disallow: /costs-aid/forms/parent-veterans-non-educational-benefits-2017
217Disallow: /costs-aid/forms/student-assets-2020
218Disallow: /costs-aid/forms/student-child-support-paid-additional-information-2017
219Disallow: /costs-aid/forms/student-combat-pay-2017
220Disallow: /costs-aid/forms/student-coop-earnings-2017
221Disallow: /costs-aid/forms/student-child-support-2017
222Disallow: /costs-aid/forms/student-child-support-received-2017
223Disallow: /costs-aid/forms/student-earnings-2017
224Disallow: /costs-aid/forms/student-federal-benefits-2017-2018
225Disallow: /costs-aid/forms/student-household-additional-information-2020
226Disallow: /costs-aid/forms/student-housing-allowance-2017
227Disallow: /costs-aid/forms/student-income-additional-information-2017
228Disallow: /costs-aid/forms/student-marital-status-2019-2020
229Disallow: /costs-aid/forms/student-employment-2017
230Disallow: /costs-aid/forms/student-rollover-2017
231Disallow: /costs-aid/forms/student-tax-additional-information-2017
232Disallow: /costs-aid/forms/student-tax-deferred-savings-2017
233Disallow: /costs-aid/forms/student-taxable-grants-scholarships-2017
234Disallow: /costs-aid/forms/student-non-filers-2017
235Disallow: /costs-aid/forms/student-foreign-tax-2017
236Disallow: /costs-aid/forms/student-other-untaxed-income-2017
237Disallow: /costs-aid/forms/student-veterans-non-ed-benefits-2017
238Disallow: /costs-aid/forms/student-homeless-confirmation-2020
239Disallow: /costs-aid/forms/student-homeless-confirmation-2021
240Disallow: /costs-aid/forms/student-selective-service
241Disallow: /costs-aid/forms/parent-tax-filers-2018
242Disallow: /costs-aid/forms/student-tax-deferred-savings-2018
243Disallow: /costs-aid/forms/student-tax-non-filers-2018
244Disallow: /costs-aid/forms/student-foreign-tax-2018
245Disallow: /costs-aid/forms/student-other-untaxed-income-2018
246Disallow: /costs-aid/forms/student-veterans-non-ed-benefits-2018
247Disallow: /costs-aid/forms/student-physician-certification
248Disallow: /costs-aid/forms/parent-citizenship-2020
249Disallow: /costs-aid/forms/parent-citizenship-2021
250Disallow: /costs-aid/forms/parent-assets-2019
251Disallow: /costs-aid/forms/withdrawal-summer-enrollment
252Disallow: /costs-aid/forms/parent-tax/non-filers-2016
253Disallow: /costs-aid/forms/parent-tax-filers-2016
254Disallow: /costs-aid/forms/student-foreign-tax-2016
255Disallow: /costs-aid/forms/parent-earnings-2016
256Disallow: /costs-aid/forms/student-tax-instructions-2016
257Disallow: /costs-aid/forms/student-tax-filers-2016
258Disallow: /costs-aid/forms/student-tax-non-filers-2016
259Disallow: /costs-aid/forms/student-foreign-tax-2016-2
260Disallow: /costs-aid/forms/student-earnings-2016
261Disallow: /costs-aid/forms/student-assets-2019
262Disallow: /costs-aid/forms/parent-tax-deferred-savings-2016
263Disallow: /costs-aid/forms/student-homeless-confirmation-2019
264Disallow: /costs-aid/forms/student-identity-education-purpose-2022
265Disallow: /costs-aid/forms/parent-tax-additional-information-2019
266Disallow: /costs-aid/forms/parent-household-additional-information-2022
267Disallow: /costs-aid/forms/parent-child-support-2019
268Disallow: /costs-aid/forms/parent-child-support-received-2019
269Disallow: /costs-aid/forms/parent-child-support-paid-additional-information-2019
270Disallow: /costs-aid/forms/parent-combat-pay-2019
271Disallow: /costs-aid/forms/parent-earnings-2019
272Disallow: /costs-aid/forms/parent-coop-earnings-2019
273Disallow: /costs-aid/forms/parent-tax-non-filers-2019
274Disallow: /costs-aid/forms/parent-tax-filers-2019
275Disallow: /costs-aid/forms/parent-foreign-tax-2019
276Disallow: /costs-aid/forms/parent-tax-additional-information-2020
277Disallow: /costs-aid/forms/parent-child-support-paid-additional-information-2020
278Disallow: /costs-aid/forms/parent-child-support-received-2020
279Disallow: /costs-aid/forms/parent-combat-pay-2020
280Disallow: /costs-aid/forms/parent-coop-earnings-2020
281Disallow: /costs-aid/forms/parent-earnings-2020
282Disallow: /costs-aid/forms/parent-tax-filers-2020
283Disallow: /costs-aid/forms/parent-tax-non-filers-2020
284Disallow: /costs-aid/forms/parent-taxable-grants-scholarships-2020
285Disallow: /costs-aid/forms/parent-rollover-2020
286Disallow: /costs-aid/forms/parent-untaxed-income-2020
287Disallow: /costs-aid/forms/parent-veterans-non-educational-benefits-2020
288Disallow: /costs-aid/forms/student-housing-allowance-2020
289Disallow: /costs-aid/forms/student-employment-2020
290Disallow: /costs-aid/forms/student-federal-benefits-2020-2021
291Disallow: /costs-aid/forms/student-unreported-income-2020
292Disallow: /costs-aid/forms/parent-marital-status-2024
293Disallow: /costs-aid/forms/parent-tax-instructions-2020
294Disallow: /costs-aid/forms/parent-household-additional-information-2021
295Disallow: /costs-aid/forms/student-household-additional-information-2021
296Disallow: /costs-aid/forms/parent-household-additional-information-2020
297Disallow: /costs-aid/forms/parent-household-additional-information-2023
298Disallow: /costs-aid/forms/student-household-additional-information-2023
299Disallow: /costs-aid/forms/student-financial-information-2021
300Disallow: /costs-aid/forms/marital-status-2024-25
301Disallow: /costs-aid/forms/marital-status/2024-25-2022-info
302Disallow: /costs-aid/forms/marital-status/2023-24-2021
303Disallow: /costs-aid/forms/student-financial-information-2022
304Disallow: /costs-aid/forms/student-financial-information-2023
305Disallow: /costs-aid/forms/costs-aid/special-circumstance-parent-documentation-2023
306Disallow: /costs-aid/forms/special-circumstance-parent-documentation-2021
307Disallow: /costs-aid/forms/parent-tax-additional-information-2017
308Disallow: /costs-aid/forms/short-term-promissory-note
309Disallow: /costs-aid/forms/student-tax-deferred-savings-2019
310Disallow: /costs-aid/forms/student-tax-additional-information-2021
311Disallow: /costs-aid/forms/student-tax/tax-filers-2017
312Disallow: /costs-aid/forms/student-taxable-grants-scholarships-2018
313Disallow: /costs-aid/forms/student-tax-filers-2018
314Disallow: /costs-aid/forms/parent-marital-status-2021
315Disallow: /costs-aid/forms/student-housing-allowance-2019
316Disallow: /costs-aid/forms/student-earnings-2020
317Disallow: /costs-aid/forms/student-earnings-2021
318Disallow: /costs-aid/forms/parent-name-ssn
319Disallow: /costs-aid/forms/parent-documentation-2022
320Disallow: /costs-aid/forms/parent-citizenship-2022
321Disallow: /costs-aid/forms/student-current-income-review-2022
322Disallow: /costs-aid/forms/student-emancipation
323Disallow: /costs-aid/forms/student-foster-orphan-ward
324Disallow: /costs-aid/forms/student-guardianship
325Disallow: /costs-aid/forms/student-name-ssn
326Disallow: /costs-aid/forms/student-tax-additional-information-2022
327Disallow: /costs-aid/forms/student-tax-filers-2022
328Disallow: /costs-aid/forms/student-foreign-income-exclusion-2022
329Disallow: /costs-aid/forms/student-rollover-2022
330Disallow: /costs-aid/forms/student-household-additional-information-2025
331Disallow: /costs-aid/forms/parent-foreign-tax-2022
332Disallow: /costs-aid/forms/student-non-filers-2022
333Disallow: /costs-aid/forms/parent-tax-non-filers-2022
334Disallow: /costs-aid/forms/parent-foreign-income-exclusion-2022
335Disallow: /costs-aid/forms/parents-tax-filers-2022
336Disallow: /costs-aid/forms/parent-citizenship
337Disallow: /costs-aid/forms/parent-foreign-tax-2021
338Disallow: /costs-aid/forms/student-foreign-tax-2022
339Disallow: /costs-aid/forms/pa-grants-9-11-form-summer
340Disallow: /costs-aid/forms/special-circumstance-marital-status
341Disallow: /costs-aid/forms/f26shh
342Disallow: /costs-aid/forms/f26phh
343Disallow: /costs-aid/forms/f26stx-foreign-tax
344Disallow: /costs-aid/forms/f26ptn
345Disallow: /costs-aid/forms/f26sas
346Disallow: /costs-aid/forms/f26ptx-tax-filer
347Disallow: /costs-aid/forms/f26ptx-non-filer
348Disallow: /costs-aid/forms/f26pfi
349Disallow: /costs-aid/forms/parent-household-2021
350Disallow: /costs-aid/forms/parent-household-size-2022
351Disallow: /costs-aid/forms/student-hs-confirmation-2024
352Disallow: /costs-aid/forms/student-household-verification-2023
353Disallow: /costs-aid/forms/student-household-2021
354Disallow: /costs-aid/forms/student-household-size-2018
355Disallow: /costs-aid/forms/verification-parent-marital-status-2025
356Disallow: /costs-aid/forms/verification-parent-tax-additional-information-2022
357Disallow: /costs-aid/forms/verification-student-marital-status-2024
358Disallow: /costs-aid/forms/verification-parent-assets-2025
359Disallow: /costs-aid/forms/student-household-2025
360Disallow: /costs-aid/forms/verification-student-marital-status-2025
361Disallow: /costs-aid/forms/parent-child-support-paid-2021
362Disallow: /costs-aid/forms/student-dob-confirmation
363Disallow: /costs-aid/forms/student-citizenship-verification
364Disallow: /costs-aid/forms/student-assets-2025
365Disallow: /costs-aid/forms/student-earnings-2025
366Disallow: /costs-aid/forms/sap-credit-eval-form.pdf
367Disallow: /costs-aid/forms/pa-grants-9-11-form-fall-spring
368Disallow: /costs-aid/forms/student-taxable-grants-scholarships-2023
369Disallow: /costs-aid/forms/student-taxable-grants-scholarships-2022
370Disallow: /costs-aid/forms/parent-federal-benefits-2023-2024
371Disallow: /costs-aid/forms/student-federal-benefits-2023-2024
372Disallow: /costs-aid/forms/reaffirmation-form
373Disallow: /costs-aid/forms/marital-status-2023-24-2022
374Disallow: /costs-aid/forms/parent-tax-instructions-2024
375Disallow: /costs-aid/forms/special-circumstance-parent-documentation
376Disallow: /costs-aid/forms/special-circumstance-student-documentation
377Disallow: /costs-aid/forms/f26pas
378Disallow: /costs-aid/forms/student-rollover-2021
379Disallow: /costs-aid/forms/student-identity-education-purpose-2025
380Disallow: /costs-aid/forms/student-homeless-confirmation-2025
381Disallow: /costs-aid/forms/student-homeless-confirmation-2024
382Disallow: /costs-aid/forms/provost-review-form
383Disallow: /costs-aid/forms/discover-award-review-form
384Disallow: /costs-aid/forms/retention-grant-review-form
385Disallow: /costs-aid/forms/land-grant-review-form
386Disallow: /costs-aid/forms/unusual-enrollment-history-ueh-appeal-form
387Disallow: /costs-aid/forms/parent-citizenship2
388Disallow: /costs-aid/forms/parent-not-on-fafsa-plus
389Disallow: /costs-aid/forms/student-homeless-confirmation-fhmls-2027
390Disallow: /costs-aid/forms/student-non-filing-letter-f27stn
391Disallow: /costs-aid/forms/sap-appeal-form.pdf
392Disallow: /costs-aid/forms/parent-taxable-grants-scholarships-2023
393Disallow: /costs-aid/forms/student-taxable-grants-scholarships-2024
394Disallow: /costs-aid/forms/student-family-size-additional-f27sha
395Disallow: /costs-aid/forms/student-federal-benefits-f27sfb
396Disallow: /costs-aid/forms/f26sew
397Disallow: /costs-aid/forms/student-income-f27sew
398Disallow: /costs-aid/forms/student-assets-f26sas
399Disallow: /costs-aid/forms/parent-non-filer-f26ptx
400Disallow: /costs-aid/forms/parent-foreign-exclusion-f27pfi
401Disallow: /costs-aid/forms/parent-taxable-grants-scholarships-f27ptg
402Disallow: /costs-aid/forms/parent-tax-filing-status-f27pta
403Disallow: /costs-aid/forms/parent-rollover-f27prl
404Disallow: /costs-aid/forms/parent-not-on-fafsa-unsub
405Disallow: /costs-aid/forms/parent-marital-status-f27pmr
406Disallow: /costs-aid/forms/parent-family-size-additional-f27pha
407Disallow: /costs-aid/forms/parent-federal-benefits-2024-2025-f27pfb
408Disallow: /costs-aid/forms/parent-income-work-f27pew
409Disallow: /costs-aid/forms/parent-assets-f27pas
410Disallow: /costs-aid/forms/student-high-school-completion-f27hsc
411Disallow: /costs-aid/forms/student-veteran-status
412Disallow: /costs-aid/forms/parent-family-size
413Disallow: /costs-aid/forms/student-family-size
414Disallow: /costs-aid/forms/non-filer-f27stx
415Disallow: /costs-aid/forms/foreign-tax-f27stx
416Disallow: /costs-aid/forms/student-foreign-income-exclusion-f27sfi
417Disallow: /costs-aid/forms/parent-irs-non-filing-f27ptn
418Disallow: /academics/campuses/york
419Disallow: /academics/campuses/wilkes-barre
420Disallow: /academics/campuses/shenango
421Disallow: /academics/campuses/new-kensington
422Disallow: /academics/campuses/mont-alto
423Disallow: /academics/campuses/fayette
424Disallow: /academics/campuses/dubois
425Disallow: /academics/campuses/campus-carlisle
426Disallow: /academics/campuses/campus-great-valley
427Disallow: /academics/campuses/campus-hershey
428Disallow: /academics/campuses/campus-williamsport
429Disallow: /academics/campuses/campus-world-campus
430Allow: /
431
432User-agent: GPTBot
433Allow: /
434
435User-agent: ChatGPT-User
436Allow: /
437
438User-agent: Google-Extended
439Allow: /
440
441User-agent: CCBot
442Allow: /
443
444User-agent: Amazonbot
445Allow: /
446
447User-agent: Bytespider
448Allow: /
449
450User-agent: ClaudeBot
451Allow: /
452
453User-agent: anthropic-ai
454Allow: /
455
456User-agent: PerplexityBot
457Allow: /
458
459Sitemap: https://www.psu.edu/sitemap.xml
460Content-Signal: ai-train=yes, search=yes, ai-input=yes
What do the bots actually get?
This page reads robots.txt, which states what psu.edu asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on psu.edu Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does psu.edu block GPTBot?
No. psu.edu’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does psu.edu block ClaudeBot?
No. psu.edu’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does psu.edu block Google-Extended?
No. psu.edu’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does psu.edu block PerplexityBot?
No. psu.edu’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find psu.edu?
Yes. psu.edu’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.