Who blocks AI · #2 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does facebook.com block AI crawlers?
Yes. facebook.com’s robots.txt blocks all 19 AI crawlers we track, from GPTBot and ClaudeBot to PerplexityBot. It also blocks PetalBot, YandexBot and Baiduspider among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- facebook.com/robots.txt · 1,050 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 338 · Disallow: /*/plugins/* line 339 · Disallow: /?*next= and 39 more | |
Bingbot | Microsoft | Partly blocked | line 101 · Disallow: /*/plugins/* line 102 · Disallow: /?*next= and 39 more |
DuckDuckBot | DuckDuckGo | Partly blocked | line 244 · Disallow: /*/plugins/* line 245 · Disallow: /?*next= and 39 more |
Applebot | Apple | Partly blocked | line 54 · Disallow: /*/plugins/* line 55 · Disallow: /?*next= and 39 more |
PetalBot | Huawei | Blocked | line 33 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Blocked | line 1047 · Disallow: / under User-agent: * |
Baiduspider | Baidu | Blocked | line 1047 · Disallow: / under User-agent: * |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 1047 · Disallow: / under User-agent: * |
PerplexityBot | Perplexity | Blocked | line 30 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 1047 · Disallow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Blocked | line 1047 · Disallow: / under User-agent: * |
Amazonbot | Amazon | Blocked | line 12 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 1047 · Disallow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 27 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 21 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 24 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 1047 · Disallow: / under User-agent: * |
Applebot-Extended | Apple | Blocked | line 15 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Blocked | line 1047 · Disallow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Blocked | line 1047 · Disallow: / under User-agent: * |
Bytespider | ByteDance | Blocked | line 1047 · Disallow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 1047 · Disallow: / under User-agent: * |
Claude-User | Anthropic | Blocked | line 1047 · Disallow: / under User-agent: * |
Perplexity-User | Perplexity | Blocked | line 1047 · Disallow: / under User-agent: * |
MistralAI-User | Mistral | Blocked | line 1047 · Disallow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Blocked | line 1047 · Disallow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 1047 · Disallow: / under User-agent: * |
SemrushBot | Semrush | Blocked | line 1047 · Disallow: / under User-agent: * |
MJ12bot | Majestic | Blocked | line 1047 · Disallow: / under User-agent: * |
DotBot | Moz | Blocked | line 1047 · Disallow: / under User-agent: * |
View facebook.com/robots.txt · 1,050 lines
1# Notice: Collection of data on Facebook through automated means is
2# prohibited unless you have express written permission from Facebook
3# and may only be conducted for the limited purpose contained in said
4# permission.
5# All authorized user-agents listed on this page must comply with Meta’s
6# Automated Data Collection Terms available at:
7# https://www.facebook.com/legal/automated_data_collection_terms
8User-agent: __usage_ai-training
9Disallow: /
10
11User-agent: Amazonbot
12Disallow: /
13
14User-agent: Applebot-Extended
15Disallow: /
16
17User-agent: Brightbot
18Disallow: /
19
20User-agent: ClaudeBot
21Disallow: /
22
23User-agent: Google-Extended
24Disallow: /
25
26User-agent: GPTBot
27Disallow: /
28
29User-agent: PerplexityBot
30Disallow: /
31
32User-agent: PetalBot
33Disallow: /
34
35User-agent: Scrapy
36Disallow: /
37
38User-agent: uptimerobot
39Disallow: /
40
41User-agent: viberbot
42Disallow: /
43
44User-agent: YaK
45Disallow: /
46
47User-agent: Yandex
48Disallow: /
49
50User-agent: Yeti
51Disallow: /
52
53User-agent: Applebot
54Disallow: /*/plugins/*
55Disallow: /?*next=
56Disallow: /a/bz?
57Disallow: /ajax/
58Disallow: /album.php
59Disallow: /business/*&categories
60Disallow: /business/*?categories
61Disallow: /business/help/search*&query=
62Disallow: /business/help/search*?query=
63Disallow: /business/news*&tags
64Disallow: /business/news*?tags
65Disallow: /business/search*&q=
66Disallow: /business/search*?q=
67Disallow: /checkpoint/
68Disallow: /contact_importer/
69Disallow: /dialog/
70Disallow: /fbml/ajax/dialog/
71Disallow: /feeds/
72Disallow: /file_download.php
73Disallow: /job_application/
74Disallow: /l.php
75Disallow: /login.php*&next=
76Disallow: /login.php/?next=
77Disallow: /login.php?next=
78Disallow: /login/*&next=
79Disallow: /login/?next=
80Disallow: /login/device-based/regular/login/*&next=
81Disallow: /login/device-based/regular/login/?next=
82Disallow: /meta-ai-answers/
83Disallow: /moments_app/
84Disallow: /p.php
85Disallow: /photos.php
86Disallow: /plugins/
87Disallow: /share.php
88Disallow: /share/
89Disallow: /sharer.php
90Disallow: /sharer/
91Disallow: /tr/
92Disallow: /tr?
93Disallow: /ufi/reaction/profile/browser/
94Disallow: /x/oauth/
95Allow: /ajax/bootloader-endpoint/
96Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
97Allow: /careers/
98Allow: /safetycheck/
99
100User-agent: Bingbot
101Disallow: /*/plugins/*
102Disallow: /?*next=
103Disallow: /a/bz?
104Disallow: /ajax/
105Disallow: /album.php
106Disallow: /business/*&categories
107Disallow: /business/*?categories
108Disallow: /business/help/search*&query=
109Disallow: /business/help/search*?query=
110Disallow: /business/news*&tags
111Disallow: /business/news*?tags
112Disallow: /business/search*&q=
113Disallow: /business/search*?q=
114Disallow: /checkpoint/
115Disallow: /contact_importer/
116Disallow: /dialog/
117Disallow: /fbml/ajax/dialog/
118Disallow: /feeds/
119Disallow: /file_download.php
120Disallow: /job_application/
121Disallow: /l.php
122Disallow: /login.php*&next=
123Disallow: /login.php/?next=
124Disallow: /login.php?next=
125Disallow: /login/*&next=
126Disallow: /login/?next=
127Disallow: /login/device-based/regular/login/*&next=
128Disallow: /login/device-based/regular/login/?next=
129Disallow: /meta-ai-answers/
130Disallow: /moments_app/
131Disallow: /p.php
132Disallow: /photos.php
133Disallow: /plugins/
134Disallow: /share.php
135Disallow: /share/
136Disallow: /sharer.php
137Disallow: /sharer/
138Disallow: /tr/
139Disallow: /tr?
140Disallow: /ufi/reaction/profile/browser/
141Disallow: /x/oauth/
142Allow: /ajax/bootloader-endpoint/
143Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
144Allow: /careers/
145Allow: /safetycheck/
146
147User-agent: deepcrawl
148Disallow: /
149Disallow: /*/plugins/*
150Disallow: /?*next=
151Disallow: /a/bz?
152Disallow: /ajax/
153Disallow: /album.php
154Disallow: /business/*&categories
155Disallow: /business/*?categories
156Disallow: /business/help/search*&query=
157Disallow: /business/help/search*?query=
158Disallow: /business/news*&tags
159Disallow: /business/news*?tags
160Disallow: /business/search*&q=
161Disallow: /business/search*?q=
162Disallow: /checkpoint/
163Disallow: /contact_importer/
164Disallow: /dialog/
165Disallow: /fbml/ajax/dialog/
166Disallow: /feeds/
167Disallow: /file_download.php
168Disallow: /job_application/
169Disallow: /l.php
170Disallow: /login.php*&next=
171Disallow: /login.php/?next=
172Disallow: /login.php?next=
173Disallow: /login/*&next=
174Disallow: /login/?next=
175Disallow: /login/device-based/regular/login/*&next=
176Disallow: /login/device-based/regular/login/?next=
177Disallow: /meta-ai-answers/
178Disallow: /moments_app/
179Disallow: /p.php
180Disallow: /photos.php
181Disallow: /plugins/
182Disallow: /share.php
183Disallow: /share/
184Disallow: /sharer.php
185Disallow: /sharer/
186Disallow: /tr/
187Disallow: /tr?
188Disallow: /ufi/reaction/profile/browser/
189Disallow: /x/oauth/
190Allow: /ajax/bootloader-endpoint/
191Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
192Allow: /business/
193Allow: /careers/
194Allow: /safetycheck/
195
196User-agent: Discordbot
197Disallow: /*/plugins/*
198Disallow: /?*next=
199Disallow: /a/bz?
200Disallow: /ajax/
201Disallow: /album.php
202Disallow: /business/*&categories
203Disallow: /business/*?categories
204Disallow: /business/help/search*&query=
205Disallow: /business/help/search*?query=
206Disallow: /business/news*&tags
207Disallow: /business/news*?tags
208Disallow: /business/search*&q=
209Disallow: /business/search*?q=
210Disallow: /checkpoint/
211Disallow: /contact_importer/
212Disallow: /dialog/
213Disallow: /fbml/ajax/dialog/
214Disallow: /feeds/
215Disallow: /file_download.php
216Disallow: /job_application/
217Disallow: /l.php
218Disallow: /login.php*&next=
219Disallow: /login.php/?next=
220Disallow: /login.php?next=
221Disallow: /login/*&next=
222Disallow: /login/?next=
223Disallow: /login/device-based/regular/login/*&next=
224Disallow: /login/device-based/regular/login/?next=
225Disallow: /meta-ai-answers/
226Disallow: /moments_app/
227Disallow: /p.php
228Disallow: /photos.php
229Disallow: /plugins/
230Disallow: /share.php
231Disallow: /share/
232Disallow: /sharer.php
233Disallow: /sharer/
234Disallow: /tr/
235Disallow: /tr?
236Disallow: /ufi/reaction/profile/browser/
237Disallow: /x/oauth/
238Allow: /ajax/bootloader-endpoint/
239Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
240Allow: /careers/
241Allow: /safetycheck/
242
243User-agent: DuckDuckBot
244Disallow: /*/plugins/*
245Disallow: /?*next=
246Disallow: /a/bz?
247Disallow: /ajax/
248Disallow: /album.php
249Disallow: /business/*&categories
250Disallow: /business/*?categories
251Disallow: /business/help/search*&query=
252Disallow: /business/help/search*?query=
253Disallow: /business/news*&tags
254Disallow: /business/news*?tags
255Disallow: /business/search*&q=
256Disallow: /business/search*?q=
257Disallow: /checkpoint/
258Disallow: /contact_importer/
259Disallow: /dialog/
260Disallow: /fbml/ajax/dialog/
261Disallow: /feeds/
262Disallow: /file_download.php
263Disallow: /job_application/
264Disallow: /l.php
265Disallow: /login.php*&next=
266Disallow: /login.php/?next=
267Disallow: /login.php?next=
268Disallow: /login/*&next=
269Disallow: /login/?next=
270Disallow: /login/device-based/regular/login/*&next=
271Disallow: /login/device-based/regular/login/?next=
272Disallow: /meta-ai-answers/
273Disallow: /moments_app/
274Disallow: /p.php
275Disallow: /photos.php
276Disallow: /plugins/
277Disallow: /share.php
278Disallow: /share/
279Disallow: /sharer.php
280Disallow: /sharer/
281Disallow: /tr/
282Disallow: /tr?
283Disallow: /ufi/reaction/profile/browser/
284Disallow: /x/oauth/
285Allow: /ajax/bootloader-endpoint/
286Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
287Allow: /careers/
288Allow: /safetycheck/
289
290User-agent: facebookexternalhit
291Disallow: /*/plugins/*
292Disallow: /?*next=
293Disallow: /a/bz?
294Disallow: /ajax/
295Disallow: /album.php
296Disallow: /business/*&categories
297Disallow: /business/*?categories
298Disallow: /business/help/search*&query=
299Disallow: /business/help/search*?query=
300Disallow: /business/news*&tags
301Disallow: /business/news*?tags
302Disallow: /business/search*&q=
303Disallow: /business/search*?q=
304Disallow: /checkpoint/
305Disallow: /contact_importer/
306Disallow: /dialog/
307Disallow: /fbml/ajax/dialog/
308Disallow: /feeds/
309Disallow: /file_download.php
310Disallow: /job_application/
311Disallow: /l.php
312Disallow: /login.php*&next=
313Disallow: /login.php/?next=
314Disallow: /login.php?next=
315Disallow: /login/*&next=
316Disallow: /login/?next=
317Disallow: /login/device-based/regular/login/*&next=
318Disallow: /login/device-based/regular/login/?next=
319Disallow: /meta-ai-answers/
320Disallow: /moments_app/
321Disallow: /p.php
322Disallow: /photos.php
323Disallow: /plugins/
324Disallow: /share.php
325Disallow: /share/
326Disallow: /sharer.php
327Disallow: /sharer/
328Disallow: /tr/
329Disallow: /tr?
330Disallow: /ufi/reaction/profile/browser/
331Disallow: /x/oauth/
332Allow: /ajax/bootloader-endpoint/
333Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
334Allow: /careers/
335Allow: /safetycheck/
336
337User-agent: Googlebot
338Disallow: /*/plugins/*
339Disallow: /?*next=
340Disallow: /a/bz?
341Disallow: /ajax/
342Disallow: /album.php
343Disallow: /business/*&categories
344Disallow: /business/*?categories
345Disallow: /business/help/search*&query=
346Disallow: /business/help/search*?query=
347Disallow: /business/news*&tags
348Disallow: /business/news*?tags
349Disallow: /business/search*&q=
350Disallow: /business/search*?q=
351Disallow: /checkpoint/
352Disallow: /contact_importer/
353Disallow: /dialog/
354Disallow: /fbml/ajax/dialog/
355Disallow: /feeds/
356Disallow: /file_download.php
357Disallow: /job_application/
358Disallow: /l.php
359Disallow: /login.php*&next=
360Disallow: /login.php/?next=
361Disallow: /login.php?next=
362Disallow: /login/*&next=
363Disallow: /login/?next=
364Disallow: /login/device-based/regular/login/*&next=
365Disallow: /login/device-based/regular/login/?next=
366Disallow: /meta-ai-answers/
367Disallow: /moments_app/
368Disallow: /p.php
369Disallow: /photos.php
370Disallow: /plugins/
371Disallow: /share.php
372Disallow: /share/
373Disallow: /sharer.php
374Disallow: /sharer/
375Disallow: /tr/
376Disallow: /tr?
377Disallow: /ufi/reaction/profile/browser/
378Disallow: /x/oauth/
379Allow: /ajax/bootloader-endpoint/
380Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
381Allow: /careers/
382Allow: /safetycheck/
383
384User-agent: Googlebot-Image
385Disallow: /*/plugins/*
386Disallow: /?*next=
387Disallow: /a/bz?
388Disallow: /ajax/
389Disallow: /album.php
390Disallow: /business/*&categories
391Disallow: /business/*?categories
392Disallow: /business/help/search*&query=
393Disallow: /business/help/search*?query=
394Disallow: /business/news*&tags
395Disallow: /business/news*?tags
396Disallow: /business/search*&q=
397Disallow: /business/search*?q=
398Disallow: /checkpoint/
399Disallow: /contact_importer/
400Disallow: /dialog/
401Disallow: /fbml/ajax/dialog/
402Disallow: /feeds/
403Disallow: /file_download.php
404Disallow: /job_application/
405Disallow: /l.php
406Disallow: /login.php*&next=
407Disallow: /login.php/?next=
408Disallow: /login.php?next=
409Disallow: /login/*&next=
410Disallow: /login/?next=
411Disallow: /login/device-based/regular/login/*&next=
412Disallow: /login/device-based/regular/login/?next=
413Disallow: /map_tile.php
414Disallow: /meta-ai-answers/
415Disallow: /moments_app/
416Disallow: /p.php
417Disallow: /photos.php
418Disallow: /plugins/
419Disallow: /share.php
420Disallow: /share/
421Disallow: /sharer.php
422Disallow: /sharer/
423Disallow: /static_map.php
424Disallow: /tr/
425Disallow: /tr?
426Disallow: /ufi/reaction/profile/browser/
427Disallow: /x/oauth/
428Allow: /ajax/bootloader-endpoint/
429Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
430Allow: /careers/
431Allow: /safetycheck/
432
433User-agent: Googlebot-Video
434Disallow: /*/plugins/*
435Disallow: /?*next=
436Disallow: /a/bz?
437Disallow: /ajax/
438Disallow: /album.php
439Disallow: /business/*&categories
440Disallow: /business/*?categories
441Disallow: /business/help/search*&query=
442Disallow: /business/help/search*?query=
443Disallow: /business/news*&tags
444Disallow: /business/news*?tags
445Disallow: /business/search*&q=
446Disallow: /business/search*?q=
447Disallow: /checkpoint/
448Disallow: /contact_importer/
449Disallow: /dialog/
450Disallow: /fbml/ajax/dialog/
451Disallow: /feeds/
452Disallow: /file_download.php
453Disallow: /job_application/
454Disallow: /l.php
455Disallow: /login.php*&next=
456Disallow: /login.php/?next=
457Disallow: /login.php?next=
458Disallow: /login/*&next=
459Disallow: /login/?next=
460Disallow: /login/device-based/regular/login/*&next=
461Disallow: /login/device-based/regular/login/?next=
462Disallow: /meta-ai-answers/
463Disallow: /moments_app/
464Disallow: /p.php
465Disallow: /photos.php
466Disallow: /plugins/
467Disallow: /share.php
468Disallow: /share/
469Disallow: /sharer.php
470Disallow: /sharer/
471Disallow: /tr/
472Disallow: /tr?
473Disallow: /ufi/reaction/profile/browser/
474Disallow: /x/oauth/
475Allow: /ajax/bootloader-endpoint/
476Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
477Allow: /careers/
478Allow: /safetycheck/
479
480User-agent: Googlebot-News
481Disallow: /*/plugins/*
482Disallow: /?*next=
483Disallow: /a/bz?
484Disallow: /ajax/
485Disallow: /album.php
486Disallow: /business/*&categories
487Disallow: /business/*?categories
488Disallow: /business/help/search*&query=
489Disallow: /business/help/search*?query=
490Disallow: /business/news*&tags
491Disallow: /business/news*?tags
492Disallow: /business/search*&q=
493Disallow: /business/search*?q=
494Disallow: /checkpoint/
495Disallow: /contact_importer/
496Disallow: /dialog/
497Disallow: /fbml/ajax/dialog/
498Disallow: /feeds/
499Disallow: /file_download.php
500Disallow: /job_application/
501Disallow: /l.php
502Disallow: /login.php*&next=
503Disallow: /login.php/?next=
504Disallow: /login.php?next=
505Disallow: /login/*&next=
506Disallow: /login/?next=
507Disallow: /login/device-based/regular/login/*&next=
508Disallow: /login/device-based/regular/login/?next=
509Disallow: /meta-ai-answers/
510Disallow: /moments_app/
511Disallow: /p.php
512Disallow: /photos.php
513Disallow: /plugins/
514Disallow: /share.php
515Disallow: /share/
516Disallow: /sharer.php
517Disallow: /sharer/
518Disallow: /tr/
519Disallow: /tr?
520Disallow: /ufi/reaction/profile/browser/
521Disallow: /x/oauth/
522Allow: /ajax/bootloader-endpoint/
523Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
524Allow: /careers/
525Allow: /safetycheck/
526
527User-agent: Google-InspectionTool
528Disallow: /*/plugins/*
529Disallow: /?*next=
530Disallow: /a/bz?
531Disallow: /ajax/
532Disallow: /album.php
533Disallow: /business/*&categories
534Disallow: /business/*?categories
535Disallow: /business/help/search*&query=
536Disallow: /business/help/search*?query=
537Disallow: /business/news*&tags
538Disallow: /business/news*?tags
539Disallow: /business/search*&q=
540Disallow: /business/search*?q=
541Disallow: /checkpoint/
542Disallow: /contact_importer/
543Disallow: /dialog/
544Disallow: /fbml/ajax/dialog/
545Disallow: /feeds/
546Disallow: /file_download.php
547Disallow: /job_application/
548Disallow: /l.php
549Disallow: /login.php*&next=
550Disallow: /login.php/?next=
551Disallow: /login.php?next=
552Disallow: /login/*&next=
553Disallow: /login/?next=
554Disallow: /login/device-based/regular/login/*&next=
555Disallow: /login/device-based/regular/login/?next=
556Disallow: /meta-ai-answers/
557Disallow: /moments_app/
558Disallow: /p.php
559Disallow: /photos.php
560Disallow: /plugins/
561Disallow: /share.php
562Disallow: /share/
563Disallow: /sharer.php
564Disallow: /sharer/
565Disallow: /tr/
566Disallow: /tr?
567Disallow: /ufi/reaction/profile/browser/
568Disallow: /x/oauth/
569Allow: /ajax/bootloader-endpoint/
570Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
571Allow: /careers/
572Allow: /safetycheck/
573
574User-agent: lumar
575Disallow: /
576Disallow: /*/plugins/*
577Disallow: /?*next=
578Disallow: /a/bz?
579Disallow: /ajax/
580Disallow: /album.php
581Disallow: /business/*&categories
582Disallow: /business/*?categories
583Disallow: /business/help/search*&query=
584Disallow: /business/help/search*?query=
585Disallow: /business/news*&tags
586Disallow: /business/news*?tags
587Disallow: /business/search*&q=
588Disallow: /business/search*?q=
589Disallow: /checkpoint/
590Disallow: /contact_importer/
591Disallow: /dialog/
592Disallow: /fbml/ajax/dialog/
593Disallow: /feeds/
594Disallow: /file_download.php
595Disallow: /job_application/
596Disallow: /l.php
597Disallow: /login.php*&next=
598Disallow: /login.php/?next=
599Disallow: /login.php?next=
600Disallow: /login/*&next=
601Disallow: /login/?next=
602Disallow: /login/device-based/regular/login/*&next=
603Disallow: /login/device-based/regular/login/?next=
604Disallow: /meta-ai-answers/
605Disallow: /moments_app/
606Disallow: /p.php
607Disallow: /photos.php
608Disallow: /plugins/
609Disallow: /share.php
610Disallow: /share/
611Disallow: /sharer.php
612Disallow: /sharer/
613Disallow: /tr/
614Disallow: /tr?
615Disallow: /ufi/reaction/profile/browser/
616Disallow: /x/oauth/
617Allow: /ajax/bootloader-endpoint/
618Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
619Allow: /business/
620Allow: /careers/
621Allow: /safetycheck/
622
623User-agent: LinkedInBot
624Disallow: /*/plugins/*
625Disallow: /?*next=
626Disallow: /a/bz?
627Disallow: /ajax/
628Disallow: /album.php
629Disallow: /business/*&categories
630Disallow: /business/*?categories
631Disallow: /business/help/search*&query=
632Disallow: /business/help/search*?query=
633Disallow: /business/news*&tags
634Disallow: /business/news*?tags
635Disallow: /business/search*&q=
636Disallow: /business/search*?q=
637Disallow: /checkpoint/
638Disallow: /contact_importer/
639Disallow: /dialog/
640Disallow: /fbml/ajax/dialog/
641Disallow: /feeds/
642Disallow: /file_download.php
643Disallow: /job_application/
644Disallow: /l.php
645Disallow: /login.php*&next=
646Disallow: /login.php/?next=
647Disallow: /login.php?next=
648Disallow: /login/*&next=
649Disallow: /login/?next=
650Disallow: /login/device-based/regular/login/*&next=
651Disallow: /login/device-based/regular/login/?next=
652Disallow: /meta-ai-answers/
653Disallow: /moments_app/
654Disallow: /p.php
655Disallow: /photos.php
656Disallow: /plugins/
657Disallow: /share.php
658Disallow: /share/
659Disallow: /sharer.php
660Disallow: /sharer/
661Disallow: /tr/
662Disallow: /tr?
663Disallow: /ufi/reaction/profile/browser/
664Disallow: /x/oauth/
665Allow: /ajax/bootloader-endpoint/
666Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
667Allow: /careers/
668Allow: /safetycheck/
669
670User-agent: msnbot
671Disallow: /*/plugins/*
672Disallow: /?*next=
673Disallow: /a/bz?
674Disallow: /ajax/
675Disallow: /album.php
676Disallow: /business/*&categories
677Disallow: /business/*?categories
678Disallow: /business/help/search*&query=
679Disallow: /business/help/search*?query=
680Disallow: /business/news*&tags
681Disallow: /business/news*?tags
682Disallow: /business/search*&q=
683Disallow: /business/search*?q=
684Disallow: /checkpoint/
685Disallow: /contact_importer/
686Disallow: /dialog/
687Disallow: /fbml/ajax/dialog/
688Disallow: /feeds/
689Disallow: /file_download.php
690Disallow: /job_application/
691Disallow: /l.php
692Disallow: /login.php*&next=
693Disallow: /login.php/?next=
694Disallow: /login.php?next=
695Disallow: /login/*&next=
696Disallow: /login/?next=
697Disallow: /login/device-based/regular/login/*&next=
698Disallow: /login/device-based/regular/login/?next=
699Disallow: /meta-ai-answers/
700Disallow: /moments_app/
701Disallow: /p.php
702Disallow: /photos.php
703Disallow: /plugins/
704Disallow: /share.php
705Disallow: /share/
706Disallow: /sharer.php
707Disallow: /sharer/
708Disallow: /tr/
709Disallow: /tr?
710Disallow: /ufi/reaction/profile/browser/
711Disallow: /x/oauth/
712Allow: /ajax/bootloader-endpoint/
713Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
714Allow: /careers/
715Allow: /safetycheck/
716
717User-agent: Pinterestbot
718Disallow: /*/plugins/*
719Disallow: /?*next=
720Disallow: /a/bz?
721Disallow: /ajax/
722Disallow: /album.php
723Disallow: /business/*&categories
724Disallow: /business/*?categories
725Disallow: /business/help/search*&query=
726Disallow: /business/help/search*?query=
727Disallow: /business/news*&tags
728Disallow: /business/news*?tags
729Disallow: /business/search*&q=
730Disallow: /business/search*?q=
731Disallow: /checkpoint/
732Disallow: /contact_importer/
733Disallow: /dialog/
734Disallow: /fbml/ajax/dialog/
735Disallow: /feeds/
736Disallow: /file_download.php
737Disallow: /job_application/
738Disallow: /l.php
739Disallow: /login.php*&next=
740Disallow: /login.php/?next=
741Disallow: /login.php?next=
742Disallow: /login/*&next=
743Disallow: /login/?next=
744Disallow: /login/device-based/regular/login/*&next=
745Disallow: /login/device-based/regular/login/?next=
746Disallow: /meta-ai-answers/
747Disallow: /moments_app/
748Disallow: /p.php
749Disallow: /photos.php
750Disallow: /plugins/
751Disallow: /share.php
752Disallow: /share/
753Disallow: /sharer.php
754Disallow: /sharer/
755Disallow: /tr/
756Disallow: /tr?
757Disallow: /ufi/reaction/profile/browser/
758Disallow: /x/oauth/
759Allow: /ajax/bootloader-endpoint/
760Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
761Allow: /careers/
762Allow: /safetycheck/
763
764User-agent: Screaming Frog SEO Spider
765Disallow: /*/plugins/*
766Disallow: /?*next=
767Disallow: /a/bz?
768Disallow: /ajax/
769Disallow: /album.php
770Disallow: /business/*&categories
771Disallow: /business/*?categories
772Disallow: /business/help/search*&query=
773Disallow: /business/help/search*?query=
774Disallow: /business/news*&tags
775Disallow: /business/news*?tags
776Disallow: /business/search*&q=
777Disallow: /business/search*?q=
778Disallow: /checkpoint/
779Disallow: /contact_importer/
780Disallow: /dialog/
781Disallow: /fbml/ajax/dialog/
782Disallow: /feeds/
783Disallow: /file_download.php
784Disallow: /job_application/
785Disallow: /l.php
786Disallow: /login.php*&next=
787Disallow: /login.php/?next=
788Disallow: /login.php?next=
789Disallow: /login/*&next=
790Disallow: /login/?next=
791Disallow: /login/device-based/regular/login/*&next=
792Disallow: /login/device-based/regular/login/?next=
793Disallow: /meta-ai-answers/
794Disallow: /moments_app/
795Disallow: /p.php
796Disallow: /photos.php
797Disallow: /plugins/
798Disallow: /share.php
799Disallow: /share/
800Disallow: /sharer.php
801Disallow: /sharer/
802Disallow: /tr/
803Disallow: /tr?
804Disallow: /ufi/reaction/profile/browser/
805Disallow: /x/oauth/
806Allow: /ajax/bootloader-endpoint/
807Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
808Allow: /careers/
809Allow: /safetycheck/
810
811User-agent: seznambot
812Disallow: /*/plugins/*
813Disallow: /?*next=
814Disallow: /a/bz?
815Disallow: /ajax/
816Disallow: /album.php
817Disallow: /business/*&categories
818Disallow: /business/*?categories
819Disallow: /business/help/search*&query=
820Disallow: /business/help/search*?query=
821Disallow: /business/news*&tags
822Disallow: /business/news*?tags
823Disallow: /business/search*&q=
824Disallow: /business/search*?q=
825Disallow: /checkpoint/
826Disallow: /contact_importer/
827Disallow: /dialog/
828Disallow: /fbml/ajax/dialog/
829Disallow: /feeds/
830Disallow: /file_download.php
831Disallow: /job_application/
832Disallow: /l.php
833Disallow: /login.php*&next=
834Disallow: /login.php/?next=
835Disallow: /login.php?next=
836Disallow: /login/*&next=
837Disallow: /login/?next=
838Disallow: /login/device-based/regular/login/*&next=
839Disallow: /login/device-based/regular/login/?next=
840Disallow: /meta-ai-answers/
841Disallow: /moments_app/
842Disallow: /p.php
843Disallow: /photos.php
844Disallow: /plugins/
845Disallow: /share.php
846Disallow: /share/
847Disallow: /sharer.php
848Disallow: /sharer/
849Disallow: /tr/
850Disallow: /tr?
851Disallow: /ufi/reaction/profile/browser/
852Disallow: /x/oauth/
853Allow: /ajax/bootloader-endpoint/
854Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
855Allow: /careers/
856Allow: /safetycheck/
857
858User-agent: Slurp
859Disallow: /*/plugins/*
860Disallow: /?*next=
861Disallow: /a/bz?
862Disallow: /ajax/
863Disallow: /album.php
864Disallow: /business/*&categories
865Disallow: /business/*?categories
866Disallow: /business/help/search*&query=
867Disallow: /business/help/search*?query=
868Disallow: /business/news*&tags
869Disallow: /business/news*?tags
870Disallow: /business/search*&q=
871Disallow: /business/search*?q=
872Disallow: /checkpoint/
873Disallow: /contact_importer/
874Disallow: /dialog/
875Disallow: /fbml/ajax/dialog/
876Disallow: /feeds/
877Disallow: /file_download.php
878Disallow: /job_application/
879Disallow: /l.php
880Disallow: /login.php*&next=
881Disallow: /login.php/?next=
882Disallow: /login.php?next=
883Disallow: /login/*&next=
884Disallow: /login/?next=
885Disallow: /login/device-based/regular/login/*&next=
886Disallow: /login/device-based/regular/login/?next=
887Disallow: /meta-ai-answers/
888Disallow: /moments_app/
889Disallow: /p.php
890Disallow: /photos.php
891Disallow: /plugins/
892Disallow: /share.php
893Disallow: /share/
894Disallow: /sharer.php
895Disallow: /sharer/
896Disallow: /tr/
897Disallow: /tr?
898Disallow: /ufi/reaction/profile/browser/
899Disallow: /x/oauth/
900Allow: /ajax/bootloader-endpoint/
901Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
902Allow: /careers/
903Allow: /safetycheck/
904
905User-agent: teoma
906Disallow: /*/plugins/*
907Disallow: /?*next=
908Disallow: /a/bz?
909Disallow: /ajax/
910Disallow: /album.php
911Disallow: /business/*&categories
912Disallow: /business/*?categories
913Disallow: /business/help/search*&query=
914Disallow: /business/help/search*?query=
915Disallow: /business/news*&tags
916Disallow: /business/news*?tags
917Disallow: /business/search*&q=
918Disallow: /business/search*?q=
919Disallow: /checkpoint/
920Disallow: /contact_importer/
921Disallow: /dialog/
922Disallow: /fbml/ajax/dialog/
923Disallow: /feeds/
924Disallow: /file_download.php
925Disallow: /job_application/
926Disallow: /l.php
927Disallow: /login.php*&next=
928Disallow: /login.php/?next=
929Disallow: /login.php?next=
930Disallow: /login/*&next=
931Disallow: /login/?next=
932Disallow: /login/device-based/regular/login/*&next=
933Disallow: /login/device-based/regular/login/?next=
934Disallow: /meta-ai-answers/
935Disallow: /moments_app/
936Disallow: /p.php
937Disallow: /photos.php
938Disallow: /plugins/
939Disallow: /share.php
940Disallow: /share/
941Disallow: /sharer.php
942Disallow: /sharer/
943Disallow: /tr/
944Disallow: /tr?
945Disallow: /ufi/reaction/profile/browser/
946Disallow: /x/oauth/
947Allow: /ajax/bootloader-endpoint/
948Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
949Allow: /careers/
950Allow: /safetycheck/
951
952User-agent: TelegramBot
953Disallow: /*/plugins/*
954Disallow: /?*next=
955Disallow: /a/bz?
956Disallow: /ajax/
957Disallow: /album.php
958Disallow: /business/*&categories
959Disallow: /business/*?categories
960Disallow: /business/help/search*&query=
961Disallow: /business/help/search*?query=
962Disallow: /business/news*&tags
963Disallow: /business/news*?tags
964Disallow: /business/search*&q=
965Disallow: /business/search*?q=
966Disallow: /checkpoint/
967Disallow: /contact_importer/
968Disallow: /dialog/
969Disallow: /fbml/ajax/dialog/
970Disallow: /feeds/
971Disallow: /file_download.php
972Disallow: /job_application/
973Disallow: /l.php
974Disallow: /login.php*&next=
975Disallow: /login.php/?next=
976Disallow: /login.php?next=
977Disallow: /login/*&next=
978Disallow: /login/?next=
979Disallow: /login/device-based/regular/login/*&next=
980Disallow: /login/device-based/regular/login/?next=
981Disallow: /meta-ai-answers/
982Disallow: /moments_app/
983Disallow: /p.php
984Disallow: /photos.php
985Disallow: /plugins/
986Disallow: /share.php
987Disallow: /share/
988Disallow: /sharer.php
989Disallow: /sharer/
990Disallow: /tr/
991Disallow: /tr?
992Disallow: /ufi/reaction/profile/browser/
993Disallow: /x/oauth/
994Allow: /ajax/bootloader-endpoint/
995Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
996Allow: /careers/
997Allow: /safetycheck/
998
999User-agent: Twitterbot
1000Disallow: /*/plugins/*
1001Disallow: /?*next=
1002Disallow: /a/bz?
1003Disallow: /ajax/
1004Disallow: /album.php
1005Disallow: /business/*&categories
1006Disallow: /business/*?categories
1007Disallow: /business/help/search*&query=
1008Disallow: /business/help/search*?query=
1009Disallow: /business/news*&tags
1010Disallow: /business/news*?tags
1011Disallow: /business/search*&q=
1012Disallow: /business/search*?q=
1013Disallow: /checkpoint/
1014Disallow: /contact_importer/
1015Disallow: /dialog/
1016Disallow: /fbml/ajax/dialog/
1017Disallow: /feeds/
1018Disallow: /file_download.php
1019Disallow: /job_application/
1020Disallow: /l.php
1021Disallow: /login.php*&next=
1022Disallow: /login.php/?next=
1023Disallow: /login.php?next=
1024Disallow: /login/*&next=
1025Disallow: /login/?next=
1026Disallow: /login/device-based/regular/login/*&next=
1027Disallow: /login/device-based/regular/login/?next=
1028Disallow: /meta-ai-answers/
1029Disallow: /moments_app/
1030Disallow: /p.php
1031Disallow: /photos.php
1032Disallow: /plugins/
1033Disallow: /share.php
1034Disallow: /share/
1035Disallow: /sharer.php
1036Disallow: /sharer/
1037Disallow: /tr/
1038Disallow: /tr?
1039Disallow: /ufi/reaction/profile/browser/
1040Disallow: /x/oauth/
1041Allow: /ajax/bootloader-endpoint/
1042Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
1043Allow: /careers/
1044Allow: /safetycheck/
1045
1046User-agent: *
1047Disallow: /
1048
1049
1050
What do the bots actually get?
This page reads robots.txt, which states what facebook.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on facebook.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does facebook.com block GPTBot?
Yes. facebook.com’s robots.txt disallows GPTBot from the whole site (line 27: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does facebook.com block ClaudeBot?
Yes. facebook.com’s robots.txt disallows ClaudeBot from the whole site (line 21: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does facebook.com block Google-Extended?
Yes. facebook.com’s robots.txt disallows Google-Extended from the whole site (line 24: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does facebook.com block PerplexityBot?
Yes. facebook.com’s robots.txt disallows PerplexityBot from the whole site (line 30: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find facebook.com?
Not through its own crawler. facebook.com’s robots.txt disallows OAI-SearchBot from the whole site (line 1047: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.