Who blocks AI · #968 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does upwork.com block AI crawlers?
No. upwork.com’s robots.txt lets all 19 AI crawlers we track crawl the site. It also blocks PetalBot among search engines.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.upwork.com/robots.txt · 414 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 376 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | line 388 · Allow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Allowed | line 400 · Allow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Allowed | line 394 · Allow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | line 407 · Allow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | line 410 · Allow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | line 391 · Allow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Allowed | line 397 · Allow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Allowed | line 403 · Allow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.upwork.com/robots.txt · 414 lines
1User-agent: *
2
3# Sitemaps
4Sitemap: https://www.upwork.com/sitemap.xml
5# LLMs: https://www.upwork.com/llms.txt
6
7# Directories
8Disallow: /att/
9Disallow: /att-old/
10Disallow: /freelancers/public/api/
11Disallow: /messages/
12Disallow: /*/jobs/search*
13Disallow: /search/profiles/*
14Disallow: /catalog-images/*
15Disallow: /ab/
16Disallow: /hire/de/sem/
17Disallow: /nx/
18
19# Files
20Disallow: /j/view_opening_popup.php
21Disallow: /leaving_odesk.php
22Disallow: /leaving-odesk
23Disallow: /leaving
24Disallow: /nx/top-nav-ssi/visitor-gql-token
25
26# Paths (clean URLs)
27Disallow: /rss/
28Disallow: /atom/
29Disallow: /jobs/rss
30Disallow: /jobs/atom
31Disallow: /jobs/*?source=rss
32Disallow: /upi/metrics/rum/v1
33Disallow: /upi/jslogger
34Disallow: /nx/top-nav-ssi/visitor-gql-token
35Allow: /ab/account-security/reset-password
36Allow: /ab/downloads
37Allow: /ab/g/pub/wom/prx/
38Allow: /ab/g/pub/wom/bnr/
39Allow: /nx/signup/
40
41# Exclude blog search
42Disallow: /*/wp-content/cache/
43Disallow: /*/trackback/
44Disallow: /*/feed/
45Disallow: /*/comments/
46Disallow: /blog/?s=*
47Disallow: /hiring/?s=*
48Disallow: /hiring/?seo-skill-pages=*
49Disallow: /press/?s=*
50Disallow: /blog/ajax-api/
51Disallow: /hiring/ajax-api/
52
53Disallow: /nx/signup/?accountType
54Disallow: /nx/signup/?clRedirectTo
55Disallow: /nx/signup/?dest
56Disallow: /nx/signup/?midTier
57Disallow: /nx/signup/?portal
58Disallow: /nx/signup/?redir
59Disallow: /nx/signup/?signupType
60
61Disallow: */login*?*
62Disallow: */signup*?*
63Disallow: /signup/contact-contractor/id/*
64Disallow: /signup/view-profile/id/*
65Disallow: /signup/please-verify*
66Disallow: /offers/?job*
67Disallow: /job/*/apply/$
68Disallow: /b/signup/create-account/client_contact_freelancer
69Disallow: /b/signup/create-account/freelancer_apply_job
70Disallow: /freelance-jobs/fiction-writing/
71Disallow: /ab/account-security/login?
72
73Disallow: /api*
74
75# Exclude referrals URLs
76Disallow: /referrals*
77
78# Exclude Job Search noindex URLs
79Disallow: /o/jobs/browse/*?*client_hires=*
80Disallow: /o/jobs/browse/*?*duration=*
81Disallow: /o/jobs/browse/*?*workload=*
82Disallow: /o/jobs/browse/*?*sort=*
83Disallow: /o/jobs/browse/api/sands-suggestions*
84Disallow: /o/jobs/job/*
85
86# Exclude Vega Job Search URLs for now
87Disallow: /r/
88
89# Exclude Registration Success page
90Disallow: /signup/registration-success*
91
92# Exclude temporary Vega Job Details URLs
93Disallow: /c/o/jobs/job/
94
95# Exclude Vega Profiles Search new parameters
96Disallow: /o/profiles/browse*?*exp=*
97Disallow: /o/profiles/browse*?*loc=*
98Disallow: /o/profiles/browse*?*rhrs=*
99Disallow: /o/profiles/browse*?*rate=*
100Disallow: /o/profiles/browse*?*tests=*
101Disallow: /o/profiles/browse*?*tests_10=*
102Disallow: /o/profiles/browse*?*last=*
103Disallow: /o/profiles/browse*?*pt=*
104Disallow: /o/profiles/browse*?*english=*
105Disallow: /o/profiles/browse*?*lang=*
106Disallow: /o/profiles/browse*?*local_hiring=*
107Disallow: /o/profiles/browse*?*revenue=*
108Disallow: /o/profiles/browse*?*user_pref=*
109Disallow: /o/profiles/browse/api/freelancer-search-facet/
110
111# Excluded agencies
112Disallow: /o/companies/~0180c6b44423eeda59
113
114# Nuxt testing app
115Disallow: /ab/cat/*
116
117# Block old static routes
118Disallow: /i/js/*
119Disallow: /i/css/*
120Disallow: /i/images/*
121Disallow: /i/wd/*
122Disallow: /i/wdj/*
123Disallow: /i/p/*
124Disallow: /i/trends/*
125Disallow: /i/wp-login.php
126Disallow: /i/related_users/*
127Disallow: /i/smr/*
128Disallow: /i/img/*
129Disallow: /l/cn/data-analysts-in-beijing/js/*
130Disallow: /l/cn/data-analysts-in-beijing/css/*
131
132Disallow: /jobs/
133Disallow: /track/
134
135# Block RSS/Atom feeds
136Disallow: /ab/feed/
137
138# Block UPI pages
139Disallow: /upi/jslogger
140Disallow: /upi/metrics/rum
141Disallow: /upi/psmetrics
142
143# -------------------------------
144# ---------START ME-2650---------
145# -------------------------------
146# Tracking / attribution
147Disallow: /*?utm_*
148Disallow: /*?gclid=*
149Disallow: /*?gad_source=*
150Disallow: /*?sem=*
151Disallow: /*?trk=*
152Disallow: /*?promobnr=*
153Disallow: /*?promo=*
154Disallow: /*?channel=*
155Disallow: /*?mp_source=*
156Disallow: /*?redir=*
157Disallow: /*?ref=*
158Disallow: /*?referrer_url_path=*
159Disallow: /*?share=*
160Disallow: /*?a=*
161Disallow: /*?p=*
162Disallow: /*?name=*
163Disallow: /*?image=*
164Disallow: /*?track=*
165
166# Navigation / pagination / session noise
167Disallow: /*?page=*
168Disallow: /*?items_per_page=*
169Disallow: /*?order=*
170Disallow: /*?sort=*
171Disallow: /*?nav_dir=*
172Disallow: /*?viewmode=*
173Disallow: /*?device-view=*
174Disallow: /*?dfh_uid=*
175Disallow: /*?dm_to_uid=*
176Disallow: /*?s=*
177
178# Signup / verification and related params
179Disallow: /*?accountType=*
180Disallow: /*?portal=*
181Disallow: /*?dest=*
182Disallow: /*?reg-success=*
183Disallow: /*?frkscc=*
184Disallow: /nx/signup/verify-email/*
185Disallow: /nx/signup/verify-phone*
186
187# Jobs / job board
188Disallow: /*?gh_jid=*
189Disallow: /*?loc=*
190
191# Attachment / bot noise
192Disallow: /*?attachment-id=*
193Disallow: /*?_pxhc=*
194Disallow: /*?ff_UPL635ChatPro=*
195
196# Facets / filters (scoped to services to avoid over-blocking)
197Disallow: /services/*?all=*
198Disallow: /services/*?start=*
199Disallow: /services/*?cat=*
200Disallow: /services/*?price_range=*
201Disallow: /services/*?talent_quality=*
202Disallow: /services/*?spec_choice=*
203# -------------------------------
204# ----------END ME-2650----------
205# -------------------------------
206
207# -------------------------------
208# ---------START ME-2888---------
209# -------------------------------
210# Ad platform click IDs
211Disallow: /*?fbclid=*
212Disallow: /*?msclkid=*
213Disallow: /*?ttclid=*
214Disallow: /*?qclid=*
215Disallow: /*?yclid=*
216
217# Campaign / partner / promo tracking
218Disallow: /*?vt_cmp=*
219Disallow: /*?mf_ct_campaign=*
220Disallow: /*?partner_code=*
221Disallow: /*?promo_code=*
222
223# Pagination / view variants
224Disallow: /*?*_page=*
225Disallow: /*?offset=*
226Disallow: /*?view=*
227Disallow: /*?page_variant=*
228Disallow: /*?toc-variant-*
229Disallow: /*?navPos=*
230
231# Session / request IDs
232Disallow: /*?sid=*
233Disallow: /*?rid=*
234Disallow: /*?kuid=*
235Disallow: /*?ssrid=*
236
237# Shortlinks and malformed query strings
238Disallow: /*?shortlink*
239Disallow: /*?!*
240Disallow: /*?&*
241
242# Internal framework / redirect params
243Disallow: /*?_rsc=*
244Disallow: /*?_redirected=*
245
246# Google Analytics cross-domain linker params
247Disallow: /*?_ga=*
248Disallow: /*?_gl=*
249# -------------------------------
250# ----------END ME-2888----------
251# -------------------------------
252
253# -------------------------------
254# ---------START ME-3153---------
255# -------------------------------
256# Product / profile / org context (avoid duplicate indexable URLs)
257Disallow: /*?nbs=
258Disallow: /*?viewMode=
259Disallow: /*?companyReference=
260Disallow: /*?companyreference=
261Disallow: /*?client_id=
262Disallow: /*?wom=
263Disallow: /*?pin=
264Disallow: /*?english_level=
265Disallow: /*?landing_url=
266Disallow: /*?via=
267Disallow: /*?org=
268Disallow: /*?category=
269Disallow: /*?tot=
270Disallow: /*?tab=
271Disallow: /*?t=
272Disallow: /*?preventDismiss=
273Disallow: /*?wtime=
274Disallow: /*?referral=
275Disallow: /*?wsrc=
276Disallow: /*?step=
277Disallow: /*?id=
278Disallow: /*?title=
279Disallow: /*?qpn-profile-completeness=
280Disallow: /*?create-service-profile=
281Disallow: /*?deleted-project-uid=
282Disallow: /*?weekdayJdUid=
283Disallow: /*?location=
284Disallow: /*?pageTitle=
285Disallow: /*?mbp-open=
286Disallow: /*?forceEditor=
287Disallow: /*?lor=
288Disallow: /*?regv1=
289Disallow: /*?openModal=
290Disallow: /*?modal=
291Disallow: /*?flow=
292Disallow: /*?mpSource=
293Disallow: /*?type=
294Disallow: /*?section=
295Disallow: /*?ee=
296Disallow: /*?web=
297Disallow: /*?uni=
298Disallow: /*?euidtkncn=
299Disallow: /*?os=
300
301# Search / navigation / generic query noise
302Disallow: /*?query=
303Disallow: /*?link_search=
304Disallow: /*?q=
305Disallow: /*?source=
306
307# Affiliate / paid / partner attribution
308Disallow: /*?irclickid=
309Disallow: /*?srsltid=
310Disallow: /*?gh_src=
311Disallow: /*?fpr=
312Disallow: /*?partnerId=
313Disallow: /*?campaignid=
314Disallow: /*?campaign=
315Disallow: /*?ppc=
316Disallow: /*?afsrc=
317Disallow: /*?irgwc=
318Disallow: /*?gc_id=
319Disallow: /*?bp-pevc-promocode=
320Disallow: /*?_kx=
321
322# Third-party widgets / reviews / share / app deep links
323Disallow: /*?speechify_reviews=
324Disallow: /*?product_studio_contact=
325Disallow: /*?open-share=
326Disallow: /*?mobile-app=
327
328# Bot / challenge / CDN (Cloudflare)
329Disallow: /*?__cf_chl_tk=
330
331# Marketing automation / email / analytics IDs
332Disallow: /*?__hstc=
333Disallow: /*?_hsenc=
334Disallow: /*?hsFormKey=
335Disallow: /*?hsPreviewerApp=
336Disallow: /*?hsSkipCache=
337Disallow: /*?mc_cid=
338Disallow: /*?itcat=
339Disallow: /*?roistat_visit=
340Disallow: /*?atr=
341Disallow: /*?ajs_aid=
342Disallow: /*?psafe_param=
343Disallow: /*?requestId=
344
345# CMS preview / versioning / WordPress ecosystem noise
346Disallow: /*?ver=
347Disallow: /*?preview_id=
348Disallow: /*?elementor-preview=
349
350# E-commerce / checkout / promos (third-party or legacy flows)
351Disallow: /*?xcod=
352Disallow: /*?pStoreID=
353Disallow: /*?brand_id=
354Disallow: /*?cdConvenio=
355Disallow: /*?cartflows_flow=
356Disallow: /*?Access_Code=
357Disallow: /*?kclt=
358Disallow: /*?ciphertext=
359Disallow: /*?lessDeposite=
360
361# Geo / locale variants
362Disallow: /*?IDDrzava=
363# -------------------------------
364# ----------END ME-3153----------
365# -------------------------------
366
367# Block Wayback Machine
368User-agent: ia_archiver
369Allow: /legal/
370Disallow: /
371
372User-agent: Linespider
373Disallow: /
374
375User-agent: PetalBot
376Disallow: /
377
378User-agent: proximic
379Disallow: /
380
381# -------------------------------
382# ---------START ME-3861---------
383# -------------------------------
384# AI search/retrieval and user-fetch crawlers: Allow / = full site access.
385# These groups do not inherit User-agent: * rules.
386
387User-agent: OAI-SearchBot
388Allow: /
389
390User-agent: ChatGPT-User
391Allow: /
392
393User-agent: Claude-SearchBot
394Allow: /
395
396User-agent: Claude-User
397Allow: /
398
399User-agent: PerplexityBot
400Allow: /
401
402User-agent: Perplexity-User
403Allow: /
404
405# Deliberate training opt-ins for Google Gemini and Apple Intelligence.
406User-agent: Google-Extended
407Allow: /
408
409User-agent: Applebot-Extended
410Allow: /
411# -------------------------------
412# ----------END ME-3861----------
413# -------------------------------
414
What do the bots actually get?
This page reads robots.txt, which states what upwork.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on upwork.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does upwork.com block GPTBot?
No. upwork.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does upwork.com block ClaudeBot?
No. upwork.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does upwork.com block Google-Extended?
No. upwork.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does upwork.com block PerplexityBot?
No. upwork.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find upwork.com?
Yes. upwork.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.