Who blocks AI · #1157 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does glassdoor.com block AI crawlers?
Yes. glassdoor.com’s robots.txt blocks 9 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 6 more. It also blocks Baiduspider among search engines.
GPTBotBlockedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotAllowedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.glassdoor.com/robots.txt · 257 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Blocked | line 211 · Disallow: / under User-agent: Baiduspider |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Blocked | line 227 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 250 · Disallow: / under User-agent: Youbot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 227 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 227 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 227 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 238 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 227 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Blocked | line 227 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Blocked | line 246 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.glassdoor.com/robots.txt · 257 lines
1# Greetings, human beings!,
2#
3# If you're sniffing around this file, and you're not a robot, we're looking to meet curious folks such as yourself.
4#
5# Think you have what it takes to join the best white-hat SEO growth hackers on the planet, and help improve the way people everywhere find jobs?
6#
7# Run - don't crawl - to apply to join Glassdoor's SEO team here http://jobs.glassdoor.com
8#
9#
10#### Rules for ANY User-Agent
11User-agent: *
12Disallow: /*?*hostSite=*
13Disallow: /1347171559/
14Disallow: /about/board/
15Disallow: /about/contact/
16Disallow: /about/faq/
17Disallow: /about/forCareerCenters/
18Disallow: /about/forLibraries/
19Disallow: /about/forStudents/
20Disallow: /about/guidelines/
21Disallow: /about/index/
22Disallow: /about/jobs/
23Disallow: /about/learn/
24Disallow: /about/overview/
25Disallow: /about/privacy/
26Disallow: /about/syndicationCenter/
27Disallow: /about/team/
28Disallow: /about/terms/
29Disallow: /about/widgetTerms/
30Disallow: /ajax/
31Disallow: /abtest
32Disallow: /browse/
33Disallow: /chat
34Disallow: /Compare/choose
35Disallow: /employerinfo/
36Disallow: /employerInfo/
37Disallow: /Explore/browse-companies
38Disallow: /home/
39Disallow: /integrations/facebook/glassdoor/eep
40Disallow: /jobview/
41Disallow: /legal/
42Disallow: /lists/
43Disallow: /more/
44Disallow: /partner/
45Disallow: /partner-center/
46Disallow: /partners/company/
47Disallow: /partners/insights/
48Disallow: /partners/jobs/
49Disallow: /partners/reports/
50Disallow: /partners/resumeView
51Disallow: /partners/settings/
52Disallow: /parts
53Disallow: /Polls
54Disallow: /profile/
55Allow: /profile/login_input.htm
56Allow: /profile/joinNow_input.htm
57Allow: /member/profile/login
58Allow: /member/profile/joinNow
59Disallow: /Resume/user-profile/
60Disallow: /rss/*
61Disallow: /search/
62Disallow: /Search/
63Disallow: /survey/
64Disallow: /surveys
65Disallow: /surveys/
66Disallow: /util/
67Disallow: /getAdSlotContentsAjax.htm
68Disallow: /developer/index.htm
69Disallow: /developer/widget/builder/
70Disallow: /event-ingestion
71
72Disallow: /employers/ec
73Disallow: /employers/enhanced/billing_*
74Disallow: /slink.htm
75Disallow: /*encryptedUserId
76Disallow: /*followId
77Disallow: /*userValidationKey
78Disallow: */trackClickAsync.htm
79
80Disallow: /mz-survey/
81Disallow: /user-activation/
82Disallow: /member/
83Disallow: /resume/build/
84Disallow: /userprofile/
85Disallow: /sourcing$
86Disallow: /knowyourworth/
87Disallow: /Reviews/index.htm?
88Disallow: *filter.searchKeyword=
89
90#logging related
91Disallow: */lib$
92Disallow: */lib/
93Disallow: */globalize/
94Disallow: */globalize$
95Disallow: */ASCIISumThreshold$
96Disallow: */LogClient$
97Disallow: */MsgBuilder$
98Disallow: */UserAgent$
99Disallow: */Constants$
100Disallow: */init/
101Disallow: */init$
102Disallow: */LogServer$
103Disallow: */GDLogger$
104Disallow: */gd-perf$
105Disallow: */gd-site-hdr-dropdown$
106Disallow: */bundles$
107Disallow: */wait$
108Disallow: */extend$
109Disallow: */strings$
110Disallow: */strings/
111Disallow: */document$
112Disallow: */*Ajax.htm
113Disallow: */json$
114Disallow: */json/
115
116# Blocking track urls (ACQ-2468)
117Disallow: /track
118
119#Blocking non standard job view and job search URLs, and paginated job SERP URLs (TRFC-2831)
120
121Disallow: /job-listing/*_IE*.htm
122Disallow: /job-listing/JV.htm?*
123Disallow: /Job/*_IP*
124
125Disallow: /Job/bwl.htm
126
127# TRFC-3125 Block 'sex jobs' jobs infosite pages from being indexed
128Disallow: /Jobs/*-sex-*Jobs-EI*
129
130# TRFC-4037 Block page from being indexed
131Disallow: /Reviews/Barbizon-scam-*
132
133# Block Pages with misaligned search intent, impacting site links SERP
134Disallow: /Jobs/Glassdoor-Jobs-E100431.htm
135Disallow: /Reviews/Glassdoor-Reviews-E100431.htm
136Disallow: /Salary/Glassdoor-Salaries-E100431.htm
137Disallow: /Reviews/EMPLOYERS-Reviews-E25702.htm
138Disallow: /Reviews/Anonymous-Reviews-E5182641.htm
139Disallow: /Salaries/search-salary-SRCH_KO0,6.htm
140
141#Blocking low-traffic employer profile pages
142Disallow: /Photos/
143Disallow: /Location/All-*-Office-Locations*
144
145# Block pages with low SEO potential to reduce crawling
146Disallow: /Compare/*-vs-*
147Disallow: /Salary/*_DEPT*
148Disallow: /Interview/*_IL*
149
150# Block specific query filter parameters
151Disallow: *filter.searchCategory=*
152Disallow: *filter.overallRating=*
153Disallow: *filter.employmentStatus=*
154Disallow: *sort=*
155Disallow: *experienceLevel=*
156Disallow: *locationId=*
157Disallow: *sort.sortType=*
158Disallow: *filter.experience=*
159Disallow: *filter.jobFunction=*
160Disallow: *filter.location=*
161
162# Blocking pagination on employer infosite TR-12
163Disallow: /Jobs/*_P*.htm*
164Disallow: /Jobs/*_IP*.htm*
165Disallow: /Reviews/*_P*.htm*
166Disallow: /Reviews/*_IP*.htm*
167Allow: /Reviews/*-reviews-SRCH_*_IP2.htm*
168Disallow: /Interview/*_P*.htm*
169Disallow: /Interview/*_IP*.htm*
170Disallow: /Benefits/*_IP*.htm*
171Disallow: /Salaries/*_IP*.htm*
172Allow: /Salaries/*_IP2.htm*
173Allow: /Salaries/*_IP3.htm*
174Allow: /Salaries/*_IP4.htm*
175Allow: /Salaries/*_IP5.htm*
176
177# Blocking bots from crawling DoubleClick for Publisher and Google Analytics related URL's (which aren't real URL's)
178Disallow: /1060761/*
179
180# Indeed callback only
181Disallow: /job-listing/indeedApplyCallback.htm
182
183# API Endpoints
184Disallow: /graph
185Disallow: /api-web/
186Disallow: /api/
187Disallow: /employers/engagement/api/
188
189#Analytics
190Disallow: /geb
191Disallow: /brand-views
192
193#Typeaheads
194Disallow: /autocomplete
195Disallow: /searchsuggest
196Disallow: /findCityAjax.htm
197
198#### END Rules for ANY User-Agent
199
200User-agent: ia_archiver
201Disallow: /
202Allow: */index.htm
203
204User-Agent: OmniExplorer_Bot
205Disallow: /
206
207User-agent: Mediapartners-Google
208Allow: /
209
210User-agent: Baiduspider
211Disallow: /
212Allow: */index.htm
213
214####Partially Block High-Quality AI/LLM Bots
215User-agent: GPTBot
216User-agent: Google-Extended
217User-agent: Amazonbot
218User-agent: anthropic-ai
219User-agent: GoogleOther
220User-agent: Claude-Web
221User-agent: ClaudeBot
222User-agent: Perplexity
223User-agent: Cohere
224User-agent: cohere-ai
225User-agent: Applebot-Extended
226User-agent: Google-CloudVertexBot
227Disallow: /
228Allow: /blog/
229Allow: /Award/
230Allow: /About/
231Allow: /employers/
232Disallow: /employers/ec
233Disallow: /employers/engagement/api/
234Disallow: /employers/enhanced/billing_*
235
236#### Block Low-Quality AI/LLM Bots
237User-agent: CCBot
238Disallow: /
239User-agent: Omgilibot
240Disallow: /
241User-agent: Omgili
242Disallow: /
243User-agent: FacebookBot
244Disallow: /
245User-agent: Bytespider
246Disallow: /
247User-agent: Diffbot
248Disallow: /
249User-agent: Youbot
250Disallow: /
251User-agent: FriendlyCrawler
252Disallow: /
253User-agent: img2dataset
254Disallow: /
255
256#### End AI/LLM Bot Blocks
257
What do the bots actually get?
This page reads robots.txt, which states what glassdoor.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on glassdoor.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does glassdoor.com block GPTBot?
Yes. glassdoor.com’s robots.txt disallows GPTBot from the whole site (line 227: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does glassdoor.com block ClaudeBot?
Yes. glassdoor.com’s robots.txt disallows ClaudeBot from the whole site (line 227: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does glassdoor.com block Google-Extended?
Yes. glassdoor.com’s robots.txt disallows Google-Extended from the whole site (line 227: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does glassdoor.com block PerplexityBot?
No. glassdoor.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find glassdoor.com?
Yes. glassdoor.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.