Who blocks AI · #547 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does buzzfeed.com block AI crawlers?
Yes. buzzfeed.com’s robots.txt blocks 12 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 9 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.buzzfeed.com/robots.txt · 241 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 89 · Disallow: /comments-api/ line 90 · Disallow: /content-reactions-api/ | |
Bingbot | Microsoft | Partly blocked | line 3 · Disallow: /*.xml$ line 4 · Disallow: /buzz/*.xml$ and 15 more |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Partly blocked | line 89 · Disallow: /comments-api/ line 90 · Disallow: /content-reactions-api/ |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Blocked | line 96 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Blocked | line 114 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 50 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 93 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 62 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 83 · Disallow: / under User-agent: google-extended | |
CCBot | Common Crawl | Blocked | line 59 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 56 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 105 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 111 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Blocked | line 117 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Blocked | line 108 · Disallow: / under User-agent: Meta-ExternalFetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Blocked | line 77 · Disallow: / under User-agent: dotbot |
View www.buzzfeed.com/robots.txt · 241 lines
1User-agent: msnbot
2Crawl-delay: 120
3Disallow: /*.xml$
4Disallow: /buzz/*.xml$
5Disallow: /category/*.xml$
6Disallow: /mobile/
7Disallow: *?s=mobile
8Disallow: *?s=bpage-next
9Disallow: *?s=lightbox
10Disallow: /fabordrab/
11Disallow: /bfmp/
12Disallow: /buzzfeed/
13Disallow: /contest
14Disallow: /contests
15Disallow: /plugin/
16Disallow: /embed/
17Disallow: /_comments/
18Disallow: /bookmarks/
19Disallow: /drafts/
20
21User-agent: *
22Disallow: /buzz/*.xml$
23Disallow: /category/*.xml$
24Disallow: /mobile/
25Disallow: *?s=bpage-next
26Disallow: *?s=lightbox
27Disallow: *?s=feedpager
28Disallow: /fabordrab/
29Disallow: /bfmp/
30Disallow: /buzzfeed/
31Disallow: /contest
32Disallow: /contests
33Disallow: /_ga/
34Disallow: /static/
35Disallow: /dashboard/
36Disallow: /plugin/
37Disallow: /api/
38Disallow: /buzzfeed/api/
39Disallow: /embed/
40Disallow: /_comments/
41Disallow: /bookmarks/
42Disallow: /drafts/
43Disallow: /bpage-preview/
44Disallow: /cms_preview_ui/
45Disallow: /cms_preview_ui_static/
46Disallow: /search/
47
48User-agent: Amazonbot
49Allow: /rss-feeds/feeds/bf-amazon-ai-grounding.extended.xml
50Disallow: /
51
52User-agent: anthropic-ai
53Disallow: /
54
55User-agent: Applebot-Extended
56Disallow: /
57
58User-agent: CCBot
59Disallow: /
60
61User-agent: ClaudeBot
62Disallow: /
63
64User-agent: Claude-Web
65Disallow: /
66
67User-agent: cohere-ai
68Disallow: /
69
70User-agent: Diffbot
71Disallow: /
72
73User-agent: discobot
74Disallow: /
75
76User-agent: dotbot
77Disallow: /
78
79User-agent: FacebookBot
80Disallow: /
81
82User-agent: google-extended
83Disallow: /
84
85User-agent: Google-Extended
86Disallow: /
87
88User-agent: Googlebot
89Disallow: /comments-api/
90Disallow: /content-reactions-api/
91
92User-agent: GPTBot
93Disallow: /
94
95User-agent: PerplexityBot
96Disallow: /
97
98User-agent: TurnitinBot
99Disallow: /
100
101User-agent: magpie-crawler
102Disallow: /
103
104User-agent: meta-externalagent
105Disallow: /
106
107User-agent: Meta-ExternalFetcher
108Disallow: /
109
110User-agent: Bytespider
111Disallow: /
112
113User-agent: DuckAssistBot
114Disallow: /
115
116User-agent: Perplexity-User
117Disallow: /
118
119User-agent: Timpibot
120Disallow: /
121
122User-agent: Ai2Bot-Dolma
123Disallow: /
124
125User-agent: omgilibot
126Disallow: /
127
128User-agent: TtoYouBot
129Disallow: /
130
131User-agent: WebGPTBot
132Disallow: /
133
134User-agent: Slurp
135Crawl-delay: 4
136
137User-agent: Verity/1.1
138Crawl-delay: 20
139Allow: /$
140Allow: /*
141Disallow: /*.xml$
142Disallow: /buzz/*.xml$
143Disallow: /category/*.xml$
144Disallow: /mobile/
145Disallow: *?s=mobile
146Disallow: *?s=bpage-next
147Disallow: *?s=lightbox
148Disallow: *?s=feedpager
149Disallow: /fabordrab/
150Disallow: /bfmp/
151Disallow: /buzzfeed/
152Disallow: /contest
153Disallow: /contests
154Disallow: /_ga/
155Disallow: /static/
156Disallow: /dashboard/
157Disallow: /plugin/
158Disallow: /api/
159Disallow: /buzzfeed/api/
160Disallow: /embed/
161Disallow: /_comments/
162Disallow: /bookmarks/
163Disallow: /drafts/
164Disallow: /bpage-preview/
165Disallow: /cms_preview_ui/
166Disallow: /cms_preview_ui_static/
167Disallow: /search/
168
169User-agent: Scope3/2.0 (scope3.com)
170Allow: /$
171Allow: /*
172Disallow: /*.xml$
173Disallow: /buzz/*.xml$
174Disallow: /category/*.xml$
175Disallow: /mobile/
176Disallow: *?s=mobile
177Disallow: *?s=bpage-next
178Disallow: *?s=lightbox
179Disallow: *?s=feedpager
180Disallow: /fabordrab/
181Disallow: /bfmp/
182Disallow: /buzzfeed/
183Disallow: /contest
184Disallow: /contests
185Disallow: /_ga/
186Disallow: /static/
187Disallow: /dashboard/
188Disallow: /plugin/
189Disallow: /api/
190Disallow: /buzzfeed/api/
191Disallow: /embed/
192Disallow: /_comments/
193Disallow: /bookmarks/
194Disallow: /drafts/
195Disallow: /bpage-preview/
196Disallow: /cms_preview_ui/
197Disallow: /cms_preview_ui_static/
198Disallow: /search/
199
200User-agent: AmazonAdBot
201Allow: /$
202Allow: /*
203Disallow: /*.xml$
204Disallow: /buzz/*.xml$
205Disallow: /category/*.xml$
206Disallow: /mobile/
207Disallow: *?s=bpage-next
208Disallow: *?s=lightbox
209Disallow: *?s=feedpager
210Disallow: /fabordrab/
211Disallow: /bfmp/
212Disallow: /buzzfeed/
213Disallow: /contest
214Disallow: /contests
215Disallow: /_ga/
216Disallow: /static/
217Disallow: /dashboard/
218Disallow: /plugin/
219Disallow: /api/
220Disallow: /buzzfeed/api/
221Disallow: /embed/
222Disallow: /_comments/
223Disallow: /bookmarks/
224Disallow: /drafts/
225Disallow: /bpage-preview/
226Disallow: /cms_preview_ui/
227Disallow: /cms_preview_ui_static/
228Disallow: /search/
229
230Sitemap: https://www.buzzfeed.com/sitemap/buzzfeed.xml
231Sitemap: https://www.buzzfeed.com/sitemap/tasty.xml
232Sitemap: https://www.buzzfeed.com/sitemap/video.xml
233Sitemap: https://www.buzzfeed.com/sitemap/shopping.xml
234Sitemap: https://www.buzzfeed.com/sitemap/news.xml
235Sitemap: https://www.buzzfeed.com/sitemap/buzzfeed-community.xml
236Sitemap: https://www.buzzfeed.com/sitemap/editions.xml
237Sitemap: https://www.buzzfeed.com/sitemap/news-archive.xml
238
239User-agent: yacybot
240Disallow: /
241
What do the bots actually get?
This page reads robots.txt, which states what buzzfeed.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on buzzfeed.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does buzzfeed.com block GPTBot?
Yes. buzzfeed.com’s robots.txt disallows GPTBot from the whole site (line 93: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does buzzfeed.com block ClaudeBot?
Yes. buzzfeed.com’s robots.txt disallows ClaudeBot from the whole site (line 62: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does buzzfeed.com block Google-Extended?
Yes. buzzfeed.com’s robots.txt disallows Google-Extended from the whole site (line 83: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does buzzfeed.com block PerplexityBot?
Yes. buzzfeed.com’s robots.txt disallows PerplexityBot from the whole site (line 96: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find buzzfeed.com?
Yes. buzzfeed.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.