Who blocks AI · #532 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does snapchat.com block AI crawlers?
Yes. snapchat.com’s robots.txt blocks 11 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 8 more. It also blocks PetalBot among search engines.
GPTBotBlockedOAI-SearchBotPartly blockedClaudeBotBlockedPerplexityBotPartly blockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.snapchat.com/robots.txt · 240 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 128 · Disallow: /getreplies line 129 · Disallow: /explore/ | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Partly blocked | line 128 · Disallow: /getreplies line 129 · Disallow: /explore/ |
PetalBot | Huawei | Blocked | line 189 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Partly blocked | line 23 · Disallow: /getreplies line 24 · Disallow: /explore/ |
PerplexityBot | Perplexity | Partly blocked | line 35 · Disallow: /getreplies line 36 · Disallow: /explore/ |
Claude-SearchBot | Anthropic | Partly blocked | line 47 · Disallow: /getreplies line 48 · Disallow: /explore/ |
DuckAssistBot | DuckDuckGo | Partly blocked | line 59 · Disallow: /getreplies line 60 · Disallow: /explore/ |
Amazonbot | Amazon | Blocked | line 72 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 221 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 149 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 100 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 133 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 92 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 80 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Blocked | line 137 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Blocked | line 14 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 88 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Partly blocked | line 29 · Disallow: /getreplies line 30 · Disallow: /explore/ |
Claude-User | Anthropic | Partly blocked | line 53 · Disallow: /getreplies line 54 · Disallow: /explore/ |
Perplexity-User | Perplexity | Partly blocked | line 41 · Disallow: /getreplies line 42 · Disallow: /explore/ |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Blocked | line 18 · Disallow: / under User-agent: Meta-ExternalFetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.snapchat.com/robots.txt · 240 lines
1# *
2User-agent: *
3Disallow: /verificationofemployment
4Disallow: /static_files/deletion_request.pdf
5Disallow: /static/gla/
6Disallow: /invite/
7
8# Meta-WebIndexer
9User-agent: Meta-WebIndexer
10Allow: /
11
12# Meta-ExternalAgent
13User-agent: Meta-ExternalAgent
14Disallow: /
15
16# Meta-ExternalFetcher
17User-agent: Meta-ExternalFetcher
18Disallow: /
19
20# OAI-SearchBot
21User-agent: OAI-SearchBot
22Allow: /
23Disallow: /getreplies
24Disallow: /explore/
25
26# ChatGPT-User
27User-agent: ChatGPT-User
28Allow: /
29Disallow: /getreplies
30Disallow: /explore/
31
32# PerplexityBot
33User-agent: PerplexityBot
34Allow: /
35Disallow: /getreplies
36Disallow: /explore/
37
38# Perplexity-User
39User-agent: Perplexity-User
40Allow: /
41Disallow: /getreplies
42Disallow: /explore/
43
44# Claude-SearchBot
45User-agent: Claude-SearchBot
46Allow: /
47Disallow: /getreplies
48Disallow: /explore/
49
50# Claude-User
51User-agent: Claude-User
52Allow: /
53Disallow: /getreplies
54Disallow: /explore/
55
56# DuckAssistBot
57User-agent: DuckAssistBot
58Allow: /
59Disallow: /getreplies
60Disallow: /explore/
61
62# AI2Bot
63User-agent: AI2Bot
64Disallow: /
65
66# Ai2Bot-Dolma
67User-agent: Ai2Bot-Dolma
68Disallow: /
69
70# Amazonbot
71User-agent: Amazonbot
72Disallow: /
73
74# anthropic-ai
75User-agent: anthropic-ai
76Disallow: /
77
78# Applebot-Extended
79User-agent: Applebot-Extended
80Disallow: /
81
82# Brightbot 1.0
83User-agent: Brightbot 1.0
84Disallow: /
85
86# Bytespider
87User-agent: Bytespider
88Disallow: /
89
90# CCBot
91User-agent: CCBot
92Disallow: /
93
94# Claude-Web
95User-agent: Claude-Web
96Disallow: /
97
98# ClaudeBot
99User-agent: ClaudeBot
100Disallow: /
101
102# cohere-ai
103User-agent: cohere-ai
104Disallow: /
105
106# cohere-training-data-crawler
107User-agent: cohere-training-data-crawler
108Disallow: /
109
110# Crawlspace
111User-agent: Crawlspace
112Disallow: /
113
114# Diffbot
115User-agent: Diffbot
116Disallow: /
117
118# FacebookBot
119User-agent: FacebookBot
120Disallow: /
121
122# FriendlyCrawler
123User-agent: FriendlyCrawler
124Disallow: /
125
126# Googlebot
127User-agent: Googlebot
128Disallow: /getreplies
129Disallow: /explore/
130
131# Google-Extended
132User-agent: Google-Extended
133Disallow: /
134
135# GoogleOther
136User-agent: GoogleOther
137Disallow: /
138
139# GoogleOther-Image
140User-agent: GoogleOther-Image
141Disallow: /
142
143# GoogleOther-Video
144User-agent: GoogleOther-Video
145Disallow: /
146
147# GPTBot
148User-agent: GPTBot
149Disallow: /
150
151# iaskspider/2.0
152User-agent: iaskspider/2.0
153Disallow: /
154
155# ICC-Crawler
156User-agent: ICC-Crawler
157Disallow: /
158
159# ImagesiftBot
160User-agent: ImagesiftBot
161Disallow: /
162
163# img2dataset
164User-agent: img2dataset
165Disallow: /
166
167# ISSCyberRiskCrawler
168User-agent: ISSCyberRiskCrawler
169Disallow: /
170
171# Kangaroo Bot
172User-agent: Kangaroo Bot
173Disallow: /
174
175# omgili
176User-agent: omgili
177Disallow: /
178
179# omgilibot
180User-agent: omgilibot
181Disallow: /
182
183# PanguBot
184User-agent: PanguBot
185Disallow: /
186
187# PetalBot
188User-agent: PetalBot
189Disallow: /
190
191# Scrapy
192User-agent: Scrapy
193Disallow: /
194
195# SemrushBot-OCOB
196User-agent: SemrushBot-OCOB
197Disallow: /
198
199# SemrushBot-SWA
200User-agent: SemrushBot-SWA
201Disallow: /
202
203# Sidetrade indexer bot
204User-agent: Sidetrade indexer bot
205Disallow: /
206
207# Timpibot
208User-agent: Timpibot
209Disallow: /
210
211# VelenPublicWebCrawler
212User-agent: VelenPublicWebCrawler
213Disallow: /
214
215# Webzio-Extended
216User-agent: Webzio-Extended
217Disallow: /
218
219# YouBot
220User-agent: YouBot
221Disallow: /
222
223# Host
224Host: https://www.snapchat.com
225
226# Sitemaps
227Sitemap: https://www.snapchat.com/sitemap.xml
228Sitemap: https://www.snapchat.com/lens/sitemap.xml
229Sitemap: https://www.snapchat.com/sitemaps/place/sitemap-index.xml
230Sitemap: https://www.snapchat.com/sitemaps/profile/sitemap-index.xml
231Sitemap: https://www.snapchat.com/sitemaps/p/sitemap-index.xml
232Sitemap: https://www.snapchat.com/sitemaps/lens_creators/sitemap-index.xml
233Sitemap: https://www.snapchat.com/sitemaps/tag/sitemap-index.xml
234Sitemap: https://www.snapchat.com/sitemaps/topic/sitemap-index.xml
235Sitemap: https://www.snapchat.com/sitemaps/entertainment_funny/sitemap-index.xml
236Sitemap: https://www.snapchat.com/sitemaps/top_spotlights/sitemap-index.xml
237Sitemap: https://www.snapchat.com/sitemaps/local_city_leaf/sitemap-index.xml
238Sitemap: https://www.snapchat.com/sitemaps/local_nbhd_leaf/sitemap-index.xml
239Sitemap: https://www.snapchat.com/sitemaps/local_city_hub/sitemap-index.xml
240
What do the bots actually get?
This page reads robots.txt, which states what snapchat.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on snapchat.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does snapchat.com block GPTBot?
Yes. snapchat.com’s robots.txt disallows GPTBot from the whole site (line 149: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does snapchat.com block ClaudeBot?
Yes. snapchat.com’s robots.txt disallows ClaudeBot from the whole site (line 100: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does snapchat.com block Google-Extended?
Yes. snapchat.com’s robots.txt disallows Google-Extended from the whole site (line 133: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does snapchat.com block PerplexityBot?
Partly. snapchat.com’s robots.txt keeps PerplexityBot out of some paths, such as /getreplies, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find snapchat.com?
Partly. snapchat.com’s robots.txt keeps OAI-SearchBot out of some paths, such as /getreplies, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.