Who blocks AI · #875 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does imageshack.us block AI crawlers?
Yes. imageshack.us’s robots.txt blocks 10 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 7 more. It also blocks PetalBot among search engines.
GPTBotBlockedOAI-SearchBotPartly blockedClaudeBotBlockedPerplexityBotPartly blockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- imageshack.us/robots.txt · 166 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Partly blocked | line 161 · Disallow: /download/ line 163 · Disallow: /auth/ |
PetalBot | Huawei | Blocked | line 123 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Partly blocked | line 161 · Disallow: /download/ line 163 · Disallow: /auth/ |
PerplexityBot | Perplexity | Partly blocked | line 161 · Disallow: /download/ line 163 · Disallow: /auth/ |
Claude-SearchBot | Anthropic | Partly blocked | line 161 · Disallow: /download/ line 163 · Disallow: /auth/ |
DuckAssistBot | DuckDuckGo | Partly blocked | line 161 · Disallow: /download/ line 163 · Disallow: /auth/ |
Amazonbot | Amazon | Blocked | line 62 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 123 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 62 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 62 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 62 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 62 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 62 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Blocked | line 62 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Blocked | line 62 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 62 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Partly blocked | line 161 · Disallow: /download/ line 163 · Disallow: /auth/ |
Claude-User | Anthropic | Partly blocked | line 161 · Disallow: /download/ line 163 · Disallow: /auth/ |
Perplexity-User | Perplexity | Partly blocked | line 161 · Disallow: /download/ line 163 · Disallow: /auth/ |
MistralAI-User | Mistral | Partly blocked | line 161 · Disallow: /download/ line 163 · Disallow: /auth/ |
Meta-ExternalFetcher | Meta | Partly blocked | line 161 · Disallow: /download/ line 163 · Disallow: /auth/ |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View imageshack.us/robots.txt · 166 lines
1# robots.txt for imageshack.com (IS-193)
2#
3# Policy:
4# - Crawlers that collect content to train AI models or build datasets are
5# disallowed everywhere.
6# - Search-engine crawlers, link-preview fetchers and user-initiated
7# fetchers (an AI assistant opening a page because a person asked it to,
8# with a citation back to us) are allowed, because they send visitors.
9#
10# Grouping (RFC 9309): a crawler obeys only the most specific group that
11# names it, and ignores "User-agent: *" when its own name is listed. That is
12# why the allowed crawlers are listed in the SAME group as "*": they get the
13# same rules, and nothing has to be kept in sync. Rules are path prefixes, so
14# "Disallow: /download/" blocks /download/x but not /downloads.
15#
16# Images are served from imagizer.imageshack.com, which has its own robots.txt.
17#
18# /my/ (account pages), /track, /email/ and /search/ are NOT blocked on
19# purpose: they are noindex or redirect anonymous visitors away (/my/* to
20# /login, /search 301s to /my/images), and a crawler that may not fetch a
21# URL never sees its noindex or redirect, so a linked URL could still be
22# listed as "blocked by robots.txt".
23
24# AI training and dataset crawlers.
25User-agent: AI2Bot
26User-agent: Ai2Bot-Dolma
27User-agent: Amazonbot
28User-agent: anthropic-ai
29User-agent: Applebot-Extended
30User-agent: bedrockbot
31User-agent: Bytespider
32User-agent: CCBot
33User-agent: Claude-Web
34User-agent: ClaudeBot
35User-agent: CloudVertexBot
36User-agent: cohere-ai
37User-agent: cohere-training-data-crawler
38User-agent: Cotoyogi
39User-agent: Diffbot
40User-agent: FacebookBot
41User-agent: FriendlyCrawler
42User-agent: Google-CloudVertexBot
43User-agent: Google-Extended
44User-agent: GoogleOther
45User-agent: GoogleOther-Image
46User-agent: GoogleOther-Video
47User-agent: GPTBot
48User-agent: ICC-Crawler
49User-agent: ImagesiftBot
50User-agent: img2dataset
51User-agent: Kangaroo Bot
52User-agent: meta-externalagent
53User-agent: Meta-ExternalAgent
54User-agent: MistralAI-Training
55User-agent: PanguBot
56User-agent: SBIntuitionsBot
57User-agent: Timpibot
58User-agent: VelenPublicWebCrawler
59User-agent: Webzio-Extended
60User-agent: YandexAdditional
61User-agent: YandexAdditionalBot
62Disallow: /
63
64# Training opt-out for every crawler that honors it, whatever its name
65# (draft-kelley-aipref-training-preference-signal, IS-358). Not a real crawler,
66# so search is unaffected. Anthropic honors it; OpenAI, Microsoft and Google
67# have said they will. Keep the per-crawler groups above until each does.
68User-Agent: __usage_ai-training
69Disallow: /
70
71# Autonomous browsing agents and deep-research bots. They act for a user but
72# browse on their own and rarely send a visitor back.
73User-agent: ChatGPT Agent
74User-agent: Devin
75User-agent: Gemini-Deep-Research
76User-agent: GoogleAgent-Mariner
77User-agent: NovaAct
78User-agent: Operator
79Disallow: /
80
81# Scrapers, data brokers, brand-monitoring and SEO tools, and small AI search
82# engines that send little or no traffic.
83User-agent: AddSearchBot
84User-agent: aiHitBot
85User-agent: Andibot
86User-agent: Awario
87User-agent: bigsur.ai
88User-agent: Brightbot 1.0
89User-agent: Crawlspace
90User-agent: Datenbank Crawler
91User-agent: Echobot Bot
92User-agent: EchoboxBot
93User-agent: Factset_spyderbot
94User-agent: FirecrawlAgent
95User-agent: Google-Firebase
96User-agent: iaskspider/2.0
97User-agent: ISSCyberRiskCrawler
98User-agent: LinerBot
99User-agent: MyCentralAIScraperBot
100User-agent: netEstate Imprint Crawler
101User-agent: omgili
102User-agent: omgilibot
103User-agent: Panscient
104User-agent: panscient.com
105User-agent: PetalBot
106User-agent: PhindBot
107User-agent: Poseidon Research Crawler
108User-agent: QualifiedBot
109User-agent: QuillBot
110User-agent: quillbot.com
111User-agent: Scrapy
112User-agent: SemrushBot-OCOB
113User-agent: SemrushBot-SWA
114User-agent: ShapBot
115User-agent: Sidetrade indexer bot
116User-agent: TerraCotta
117User-agent: Thinkbot
118User-agent: TikTokSpider
119User-agent: WARDBot
120User-agent: wpbot
121User-agent: YaK
122User-agent: YouBot
123Disallow: /
124
125# Everyone else, including these named crawlers, which are listed so that a
126# crawler matching names loosely (e.g. "Applebot" vs "Applebot-Extended")
127# still lands here:
128# search indexes: Applebot, OAI-SearchBot, PerplexityBot,
129# Claude-SearchBot, meta-webindexer
130# user-initiated: ChatGPT-User, Perplexity-User, Claude-User,
131# DuckAssistBot, MistralAI-User, Meta-ExternalFetcher
132# link previews: facebookexternalhit, Twitterbot, LinkedInBot,
133# Slackbot, Discordbot, TelegramBot, WhatsApp,
134# Pinterestbot
135User-agent: Applebot
136User-agent: OAI-SearchBot
137User-agent: PerplexityBot
138User-agent: Claude-SearchBot
139User-agent: meta-webindexer
140User-agent: ChatGPT-User
141User-agent: Perplexity-User
142User-agent: Claude-User
143User-agent: DuckAssistBot
144User-agent: MistralAI-User
145User-agent: MistralAI-User/1.0
146User-agent: meta-externalfetcher
147User-agent: Meta-ExternalFetcher
148User-agent: facebookexternalhit
149User-agent: Twitterbot
150User-agent: LinkedInBot
151User-agent: Slackbot
152User-agent: Discordbot
153User-agent: TelegramBot
154User-agent: WhatsApp
155User-agent: Pinterestbot
156User-agent: *
157# How content may be used once fetched (Cloudflare Content Signals Policy,
158# IS-358): search and live AI answers yes, AI training no.
159Content-Signal: search=yes, ai-input=yes, ai-train=no
160# Original-file downloads.
161Disallow: /download/
162# Login/logout/token endpoints.
163Disallow: /auth/
164
165Sitemap: https://imageshack.com/sitemap.xml
166
What do the bots actually get?
This page reads robots.txt, which states what imageshack.us asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on imageshack.us Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
- 26 October 2026: now blocks part of the site to Applebot; now blocks part of the site to OAI-SearchBot; now blocks part of the site to PerplexityBot; now blocks part of the site to Claude-SearchBot; now blocks part of the site to DuckAssistBot; now blocks part of the site to ChatGPT-User; now blocks part of the site to Claude-User; now blocks part of the site to Perplexity-User; now blocks part of the site to MistralAI-User; now blocks part of the site to Meta-ExternalFetcher.
Questions
Does imageshack.us block GPTBot?
Yes. imageshack.us’s robots.txt disallows GPTBot from the whole site (line 62: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does imageshack.us block ClaudeBot?
Yes. imageshack.us’s robots.txt disallows ClaudeBot from the whole site (line 62: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does imageshack.us block Google-Extended?
Yes. imageshack.us’s robots.txt disallows Google-Extended from the whole site (line 62: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does imageshack.us block PerplexityBot?
Partly. imageshack.us’s robots.txt keeps PerplexityBot out of some paths, such as /download/, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find imageshack.us?
Partly. imageshack.us’s robots.txt keeps OAI-SearchBot out of some paths, such as /download/, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.