Who blocks AI · #331 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does xing.com block AI crawlers?
No. xing.com’s robots.txt lets all 19 AI crawlers we track crawl the site, though it keeps PerplexityBot, Claude-SearchBot and 4 more out of some paths. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotPartly blockedOAI-SearchBotAllowedClaudeBotPartly blockedPerplexityBotPartly blockedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.xing.com/robots.txt · 254 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Partly blocked | line 241 · Disallow: /img/users line 242 · Disallow: /start/signup?* and 9 more |
Claude-SearchBot | Anthropic | Partly blocked | line 241 · Disallow: /img/users line 242 · Disallow: /start/signup?* and 9 more |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Partly blocked | line 241 · Disallow: /img/users line 242 · Disallow: /start/signup?* and 9 more |
ClaudeBot | Anthropic | Partly blocked | line 241 · Disallow: /img/users line 242 · Disallow: /start/signup?* and 9 more |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Partly blocked | line 241 · Disallow: /img/users line 242 · Disallow: /start/signup?* and 9 more |
Perplexity-User | Perplexity | Partly blocked | line 241 · Disallow: /img/users line 242 · Disallow: /start/signup?* and 9 more |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.xing.com/robots.txt · 254 lines
1User-agent: CrazyWebCrawler-Spider
2Disallow: /
3
4User-agent: *
5Disallow: /img/users
6Disallow: /app/search
7Disallow: /cgi-bin/search.fpl
8Disallow: /patterns
9Disallow: /jobs/posting_clicks
10Disallow: /user-ass/jobs/posting_pdf
11Disallow: /publicsearch/
12Disallow: /search/
13Disallow: /people/search/
14Disallow: /r/
15Disallow: /xbp/search
16Disallow: /go/login
17Disallow: /ads/
18Disallow: /start/
19Disallow: /jobs/search/
20Disallow: /jobs/search?*
21Allow: /start/signup
22Disallow: /start/signup?*
23Disallow: /signup?*
24Disallow: /xing-one/api
25Disallow: /graphql/api
26Disallow: /social/share/
27Disallow: /analytics/
28Disallow: /spi/
29Disallow: /social_plugins/
30Disallow: /xas/api/
31Disallow: /graphql/
32Disallow: /cross-domain-bridge.html
33Disallow: */external-application
34Disallow: /pages/*:CSV2024*
35Disallow: /pages/买*
36Disallow: /pages/购*
37Disallow: /pages/加*
38Disallow: /pages/葳*
39Disallow: /pages/药*
40Disallow: /pages/迷*
41Disallow: /pages/哪*
42Disallow: /pages/网*
43Disallow: /pages/里*
44Disallow: /pages/水*
45Disallow: /pages/情*
46Disallow: /pages/卖*
47Disallow: /pages/有*
48Disallow: /pages/的*
49Disallow: /pages/么*
50Disallow: /pages/催*
51Disallow: /pages/售*
52Disallow: /pages/到*
53Disallow: /pages/出*
54Disallow: /pages/春*
55Disallow: /pages/强*
56Disallow: /pages/乖*
57Disallow: /pages/多*
58Disallow: /pages/上*
59Disallow: /pages/昏*
60Disallow: /pages/用*
61Disallow: /pages/在*
62Disallow: /pages/怎*
63Disallow: /pages/口*
64Disallow: /pages/少*
65Disallow: /pages/可*
66Disallow: /pages/香*
67Disallow: /pages/钱*
68Disallow: /pages/店*
69Disallow: /pages/幻*
70Disallow: /pages/以*
71Disallow: /pages/效*
72Disallow: /pages/订*
73Disallow: /pages/粉*
74Disallow: /pages/作*
75Disallow: /pages/听*
76Disallow: /pages/剂*
77Disallow: /pages/人*
78Disallow: /pages/蝇*
79Disallow: /pages/苍*
80Disallow: /pages/眠*
81Disallow: /pages/线*
82Disallow: /pages/女*
83Disallow: /pages/话*
84Disallow: /pages/正*
85Disallow: /pages/喷*
86Disallow: /pages/什*
87Disallow: /pages/烟*
88Disallow: /pages/睡*
89Disallow: /pages/服*
90Disallow: /pages/奸*
91Disallow: /pages/制*
92Disallow: /pages/鱼*
93Disallow: /pages/宝*
94Disallow: /pages/失*
95Disallow: /pages/魂*
96Disallow: /pages/如*
97Disallow: /pages/何*
98Disallow: /pages/安*
99Disallow: /pages/站*
100Disallow: /pages/品*
101Disallow: /pages/格*
102Disallow: /pages/能*
103Disallow: /pages/淘*
104Disallow: /pages/液*
105Disallow: /pages/媚*
106Disallow: /pages/发*
107Disallow: /pages/价*
108Disallow: /pages/付*
109Disallow: /pages/专*
110Disallow: /pages/联*
111Disallow: /pages/系*
112Disallow: /pages/城*
113Disallow: /pages/商*
114Disallow: /pages/晕*
115Disallow: /pages/三*
116Disallow: /pages/滴*
117Disallow: /pages/伟*
118Disallow: /pages/让*
119Disallow: /pages/方*
120Disallow: /pages/性*
121Disallow: /pages/力*
122Disallow: /pages/仑*
123Disallow: /pages/闲*
124Disallow: /pages/忆*
125Disallow: /pages/片*
126Disallow: /pages/果*
127Disallow: /pages/进*
128Disallow: /pages/卡*
129Disallow: /pages/金*
130Disallow: /pages/肩*
131Disallow: /pages/最*
132Disallow: /pages/拍*
133Disallow: /pages/吃*
134Disallow: /pages/叫*
135Disallow: /pages/倒*
136Disallow: /pages/蓝*
137Disallow: /pages/吗*
138Disallow: /pages/去*
139Disallow: /pages/货*
140Disallow: /pages/款*
141Disallow: /pages/唑*
142Disallow: /pages/哥*
143Disallow: /pages/致*
144Disallow: /pages/暴*
145Disallow: /pages/之*
146Disallow: /pages/醒*
147Disallow: /pages/邮*
148Disallow: /pages/箱*
149Disallow: /pages/岛*
150Disallow: /pages/不*
151Disallow: /pages/神*
152Disallow: /pages/夜*
153Disallow: /pages/千*
154Disallow: /pages/会*
155Disallow: /pages/魔*
156Disallow: /pages/雾*
157Disallow: /pages/物*
158Disallow: /pages/外*
159Disallow: /pages/咸*
160Disallow: /pages/艳*
161Disallow: /pages/美*
162Disallow: /pages/糖*
163Disallow: /pages/猎*
164Disallow: /pages/式*
165Disallow: /pages/姦*
166Disallow: /pages/使*
167Disallow: /pages/了*
168Disallow: /pages/东*
169Disallow: /pages/销*
170Disallow: /pages/道*
171Disallow: /pages/身*
172Disallow: /pages/蒙*
173Disallow: /pages/精*
174Disallow: /pages/灵*
175Disallow: /pages/汗*
176Disallow: /pages/微*
177Disallow: /pages/宗*
178Disallow: /pages/国*
179Disallow: /pages/副*
180Disallow: /pages/麻*
181Disallow: /pages/静*
182Disallow: /pages/镇*
183Disallow: /pages/真*
184Disallow: /pages/渠*
185Disallow: /pages/本*
186Disallow: /pages/日*
187Disallow: /pages/平*
188Disallow: /pages/因*
189Disallow: /pages/台*
190Disallow: /pages/信*
191Disallow: /pages/麦*
192Disallow: /pages/特*
193Disallow: /pages/漫*
194Disallow: /pages/比*
195Disallow: /pages/时*
196Disallow: /pages/拼*
197Disallow: /pages/恶*
198Disallow: /pages/恋*
199Disallow: /pages/弥*
200Disallow: /pages/延*
201Disallow: /pages/宴*
202Disallow: /pages/奈*
203Disallow: /pages/天*
204Disallow: /pages/丘*
205Disallow: /pages/骚*
206Disallow: /pages/觉*
207Disallow: /pages/热*
208Disallow: /pages/子*
209Disallow: /pages/助*
210Disallow: /pages/京*
211Disallow: /pages/马*
212Disallow: /pages/隐*
213Disallow: /pages/请*
214Disallow: /pages/红*
215Disallow: /pages/玩*
216Disallow: /pages/爽*
217Disallow: /pages/活*
218Disallow: /pages/是*
219Disallow: /pages/快*
220Disallow: /pages/型*
221Disallow: /pages/西*
222Disallow: /pages/蜘*
223Disallow: /pages/蛛*
224Disallow: /pages/生*
225Disallow: /pages/想*
226Disallow: /pages/做*
227Disallow: /api/unified-underdog
228
229User-agent: pimeyes.com crawler
230Disallow: /
231
232User-agent: GPTBot
233User-agent: GPTUser
234User-agent: ClaudeBot
235User-agent: Claude-User
236User-agent: Claude-SearchBot
237User-agent: PerplexityBot
238User-agent: Perplexity-User
239Allow: /jobs/search/
240Allow: /jobs/search?*
241Disallow: /img/users
242Disallow: /start/signup?*
243Disallow: /signup?*
244Disallow: /xing-one/api
245Disallow: /graphql/api
246Disallow: /social/share/
247Disallow: /analytics/
248Disallow: /spi/
249Disallow: /social_plugins/
250Disallow: /xas/api/
251Disallow: /graphql/
252
253Sitemap: https://www.xing.com/sitemap.xml.gz
254
What do the bots actually get?
This page reads robots.txt, which states what xing.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on xing.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does xing.com block GPTBot?
Partly. xing.com’s robots.txt keeps GPTBot out of some paths, such as /img/users, and lets it crawl the rest. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does xing.com block ClaudeBot?
Partly. xing.com’s robots.txt keeps ClaudeBot out of some paths, such as /img/users, and lets it crawl the rest. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does xing.com block Google-Extended?
No. xing.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does xing.com block PerplexityBot?
Partly. xing.com’s robots.txt keeps PerplexityBot out of some paths, such as /img/users, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find xing.com?
Yes. xing.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.