Who blocks AI · #599 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does fliphtml5.com block AI crawlers?
Yes. fliphtml5.com’s robots.txt blocks 5 of the 19 AI crawlers we track, including CCBot, Meta-ExternalAgent, Bytespider and 2 more. It also blocks PetalBot and Baiduspider among search engines.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotBlocked- robots.txt
- fliphtml5.com/robots.txt · 399 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 95 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Blocked | line 21 · Disallow: / under User-agent: Baiduspider |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Blocked | line 91 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 123 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Blocked | line 101 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 85 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 93 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 205 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Blocked | line 189 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Blocked | line 209 · Disallow: / under User-agent: DotBot |
View fliphtml5.com/robots.txt · 399 lines
1# ███████╗██╗ ██╗██████╗ ██╗ ██╗████████╗███╗ ███╗██╗ ███████╗
2# ██╔════╝██║ ██║██╔══██╗██║ ██║╚══██╔══╝████╗ ████║██║ ██╔════╝
3# █████╗ ██║ ██║██████╔╝███████║ ██║ ██╔████╔██║██║ ███████╗
4# ██╔══╝ ██║ ██║██╔═══╝ ██╔══██║ ██║ ██║╚██╔╝██║██║ ╚════██║
5# ██║ ███████╗██║██║ ██║ ██║ ██║ ██║ ╚═╝ ██║███████╗ ███████║
6# ╚═╝ ╚══════╝╚═╝╚═╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝╚══════╝ ╚══════╝
7
8# ============================================================
9# Updated-Date: 2026-06-29
10# Author: Leary Liang
11# Allowed AI bots: OpenAI (GPTBot, ChatGPT-User, OAI-SearchBot)
12# Anthropic (ClaudeBot, anthropic-ai, Claude-Web, Claude-SearchBot)
13# Apple (Applebot-Extended), Perplexity
14# Blocked: Meta, Amazon, ByteDance, Common Crawl, all CN LLMs
15# ============================================================
16
17# ============================================================
18# SECTION 1: Chinese Search Engines - FULL BLOCK
19# ============================================================
20User-agent: Baiduspider
21Disallow: /
22User-agent: baiduspider
23Disallow: /
24User-agent: Baiduspider-image
25Disallow: /
26User-agent: Baiduspider-video
27Disallow: /
28User-agent: Baiduspider-news
29Disallow: /
30User-agent: Baiduspider-favo
31Disallow: /
32User-agent: Baiduspider-cpro
33Disallow: /
34User-agent: Baiduspider-ads
35Disallow: /
36User-agent: Sosospider
37Disallow: /
38User-agent: sogou spider
39Disallow: /
40User-agent: Sogou News Spider
41Disallow: /
42User-agent: Sogou inst spider
43Disallow: /
44User-agent: Sogou spider2
45Disallow: /
46User-agent: Sogou blog
47Disallow: /
48User-agent: Sogou Orion spider
49Disallow: /
50User-agent: Sogou web spider
51Disallow: /
52User-agent: YodaoBot
53Disallow: /
54User-agent: Jike Spider
55Disallow: /
56User-agent: 360Spider
57Disallow: /
58User-agent: 360spider-image
59Disallow: /
60User-agent: 360spider-video
61Disallow: /
62User-agent: HaosouSpider
63Disallow: /
64User-agent: YisouSpider
65Disallow: /
66User-agent: EasouSpider
67Disallow: /
68User-agent: ToutiaoSpider
69Disallow: /
70User-agent: Yahoo! Slurp China
71Disallow: /
72User-agent: EtaoSpider
73Disallow: /
74User-agent: yaanibot
75Disallow: /
76
77# ============================================================
78# SECTION 2: AI Training Bots - BLOCK (selected)
79# NOT blocked: GPTBot, ChatGPT-User, OAI-SearchBot,
80# ClaudeBot, anthropic-ai, Claude-Web, Claude-SearchBot,
81# Applebot-Extended, PerplexityBot
82# ============================================================
83
84User-agent: meta-externalagent
85Disallow: /
86User-agent: Meta-ExternalAgent
87Disallow: /
88User-agent: FacebookBot
89Disallow: /
90User-agent: Amazonbot
91Disallow: /
92User-agent: Bytespider
93Disallow: /
94User-agent: PetalBot
95Disallow: /
96User-agent: PanguBot
97Disallow: /
98User-agent: ChatGLM-Spider
99Disallow: /
100User-agent: CCBot
101Disallow: /
102User-agent: cohere-ai
103Disallow: /
104User-agent: cohere-training-data-crawler
105Disallow: /
106User-agent: Diffbot
107Disallow: /
108User-agent: AI2Bot
109Disallow: /
110User-agent: AI2Bot-Dolma
111Disallow: /
112User-agent: ImagesiftBot
113Disallow: /
114User-agent: img2dataset
115Disallow: /
116User-agent: omgili
117Disallow: /
118User-agent: Omgilibot
119Disallow: /
120User-agent: Webzio-Extended
121Disallow: /
122User-agent: YouBot
123Disallow: /
124User-agent: Timpibot
125Disallow: /
126User-agent: Kangaroo Bot
127Disallow: /
128User-agent: Sidetrade indexer bot
129Disallow: /
130User-agent: VelenPublicWebCrawler
131Disallow: /
132
133# ============================================================
134# SECTION 2B: Chinese LLM Crawlers - FULL BLOCK
135# ============================================================
136User-agent: iaskspider
137Disallow: /
138User-agent: iaskspider/2.0
139Disallow: /
140User-agent: 360AISpider
141Disallow: /
142User-agent: TouTiaoSpider
143Disallow: /
144User-agent: MoonshotAI
145Disallow: /
146User-agent: Kimi-Spider
147Disallow: /
148User-agent: DeepSeekBot
149Disallow: /
150User-agent: DeepSeek-Crawler
151Disallow: /
152User-agent: TencentBot
153Disallow: /
154User-agent: BaichuanBot
155Disallow: /
156User-agent: BaichuanAI
157Disallow: /
158User-agent: SparkBot
159Disallow: /
160User-agent: xfspider
161Disallow: /
162User-agent: SkyworkBot
163Disallow: /
164User-agent: MiniMax-Spider
165Disallow: /
166User-agent: QwenSpider
167Disallow: /
168User-agent: DashScope
169Disallow: /
170User-agent: WenxinSpider
171Disallow: /
172User-agent: ERNIE-Bot
173Disallow: /
174User-agent: SenseBot
175Disallow: /
176User-agent: SenseChat
177Disallow: /
178User-agent: YuewenAI
179Disallow: /
180User-agent: YiBot
181Disallow: /
182User-agent: 01ai
183Disallow: /
184
185# ============================================================
186# SECTION 3: SEO Analysis Crawlers - FULL BLOCK
187# ============================================================
188User-agent: SemrushBot
189Disallow: /
190User-agent: SemrushBot-BA
191Disallow: /
192User-agent: SemrushBot-SI
193Disallow: /
194User-agent: SemrushBot-SWA
195Disallow: /
196User-agent: SemrushBot-CT
197Disallow: /
198User-agent: SemrushBot-OCOB
199Disallow: /
200User-agent: SemrushBot-FT
201Disallow: /
202User-agent: SplitSignalBot
203Disallow: /
204User-agent: AhrefsBot
205Disallow: /
206User-agent: MJ12bot
207Disallow: /
208User-agent: DotBot
209Disallow: /
210User-agent: Rogerbot
211Disallow: /
212User-agent: BLEXBot
213Disallow: /
214User-agent: DataForSeoBot
215Disallow: /
216User-agent: Screaming Frog SEO Spider
217Disallow: /
218User-agent: SearchmetricsBot
219Disallow: /
220User-agent: SeobilityBot
221Disallow: /
222User-agent: SeekportBot
223Disallow: /
224User-agent: SeekrBot
225Disallow: /
226User-agent: SEOkicks-Robot
227Disallow: /
228User-agent: SEOkicks
229Disallow: /
230User-agent: SEOstats
231Disallow: /
232User-agent: SeoSiteCheckup
233Disallow: /
234User-agent: SEOdiver Bot
235Disallow: /
236User-agent: SEO-Checker-bot
237Disallow: /
238User-agent: SiteAuditBot
239Disallow: /
240User-agent: cocolyzebot
241Disallow: /
242User-agent: Linguee Bot
243Disallow: /
244User-agent: Linguee
245Disallow: /
246User-agent: serpstatbot
247Disallow: /
248User-agent: Mediatoolkitbot
249Disallow: /
250User-agent: ZoominfoBot
251Disallow: /
252User-agent: SEOlyticsCrawler
253Disallow: /
254User-agent: Sistrix
255Disallow: /
256User-agent: SISTRIX Crawler
257Disallow: /
258User-agent: Sitebulb
259Disallow: /
260User-agent: SafeDNSBot
261Disallow: /
262
263# ============================================================
264# SECTION 4: Marketing / Listening / Link Monitoring - BLOCK
265# HubSpot: precise block (SEO only, keep Email/Content crawlers)
266# ============================================================
267User-agent: HubSpot Crawler
268Disallow: /
269User-agent: HubSpot Webcrawler
270Disallow: /
271User-agent: AwarioBot
272Disallow: /
273User-agent: AwarioRssBot
274Disallow: /
275User-agent: AwarioSmartBot
276Disallow: /
277User-agent: BomboraBot
278Disallow: /
279User-agent: Clickagy Intelligence Bot
280Disallow: /
281User-agent: trendictionbot
282Disallow: /
283User-agent: Morningscore Bot
284Disallow: /
285User-agent: BrandVerity
286Disallow: /
287User-agent: PiplBot
288Disallow: /
289User-agent: DomainStatsBot
290Disallow: /
291User-agent: LinkWalker
292Disallow: /
293User-agent: FindLinks
294Disallow: /
295User-agent: linkCheck
296Disallow: /
297User-agent: CheckLinks
298Disallow: /
299User-agent: www.deadlinkchecker.com
300Disallow: /
301User-agent: getlinkinfo.com
302Disallow: /
303User-agent: harsilbot
304Disallow: /
305User-agent: MBCrawler
306Disallow: /
307User-agent: AddSearchBot
308Disallow: /
309User-agent: RankFlex
310Disallow: /
311User-agent: nooze-bot
312Disallow: /
313User-agent: OpenindexSpider
314Disallow: /
315User-agent: OpenWebSpider
316Disallow: /
317User-agent: YaK
318Disallow: /
319User-agent: webgains-bot
320Disallow: /
321User-agent: Qwantify
322Disallow: /
323User-agent: Qwantify-prod
324Disallow: /
325User-agent: BacklinksExtendedBot
326Disallow: /
327User-agent: BuckBot
328Disallow: /
329User-agent: Pcore-HTTP
330Disallow: /
331
332# ============================================================
333# SECTION 5: Default rules for all other bots
334# ============================================================
335User-agent: *
336Disallow: */help/
337Disallow: */edit-book/*
338Disallow: */404.php
339Disallow: */access-statistics.php
340Disallow: */accountinfo.php
341Disallow: */bookinfo.php
342Disallow: */buyvip.php
343Disallow: */forget-password.php
344Disallow: */invoice.php
345Disallow: */left.php
346Disallow: */paymentinfo.php
347Disallow: */payvipinfo.php
348Disallow: */register-successfully.php
349Disallow: */reset-password.php
350Disallow: /pass/
351Disallow: /bookvisitinfo.html
352Disallow: /statistic-server/
353Disallow: /api/
354Disallow: /check/
355Disallow: /learning-center/*/tags/
356Disallow: /*/learning-center/tags/
357Disallow: /ebooks/
358Disallow: */homepage/
359Disallow: /*/*/templates/
360Disallow: *?search=
361Disallow: /blog/page/
362Disallow: /blog/*/page/
363Disallow: /blog/*/*/page/
364Disallow: /blog/*/*/*/page/
365Disallow: /blog/*/*/*/*/page/
366Disallow: /*/javascript/config.js
367Disallow: /*/slide_javascript/slideJS.js
368Disallow: /*/files/pageEditor.js
369Disallow: /*/javascript/plugin.json
370Disallow: /*/getui.json
371Disallow: /*/user/*.json
372Disallow: /*config.js*
373Disallow: /*slideJS.js*
374Disallow: /*pageEditor.js*
375Disallow: /*plugin.json*
376Disallow: /*/power.json
377Disallow: /online-tools/*/task-tool-*
378Disallow: /online-tools/task-tool-*
379Disallow: /ht001/
380Disallow: /new_ht001/
381
382# ============================================================
383# SITEMAPS
384# ============================================================
385Sitemap: https://fliphtml5.com/sitemap/books.xml
386Sitemap: https://fliphtml5.com/sitemap/books-image-index.xml
387Sitemap: https://fliphtml5.com/sitemap/flipbooks-index.xml
388Sitemap: https://fliphtml5.com/sitemap/templates/index.xml
389Sitemap: https://fliphtml5.com/sitemap/bookcases.xml
390Sitemap: https://fliphtml5.com/sitemap/latest-books.xml
391Sitemap: https://fliphtml5.com/sitemap/pages.xml
392Sitemap: https://fliphtml5.com/blog/sitemap_index.xml
393Sitemap: https://online.fliphtml5.com/sitemap/books-index.xml
394Sitemap: https://online.fliphtml5.com/sitemap/books-index-new.xml
395Sitemap: https://online.fliphtml5.com/sitemap/latest-books-index.xml
396Sitemap: https://online.fliphtml5.com/sitemap/latest-books-index-new.xml
397Sitemap: https://help.fliphtml5.com/wp-sitemap.xml
398Sitemap: https://exh.fliphtml5.com/sitemap/sitemap.xml
399Sitemap: https://e.fliphtml5.com/_sitemap/sitemap.xml
What do the bots actually get?
This page reads robots.txt, which states what fliphtml5.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on fliphtml5.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does fliphtml5.com block GPTBot?
No. fliphtml5.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does fliphtml5.com block ClaudeBot?
No. fliphtml5.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does fliphtml5.com block Google-Extended?
No. fliphtml5.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does fliphtml5.com block PerplexityBot?
No. fliphtml5.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find fliphtml5.com?
Yes. fliphtml5.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.