Bot directory · AI training

AI training · ByteDance

Bytespider

Bytespider is the crawler associated with ByteDance, the company behind TikTok. ByteDance publishes no documentation for it: no stated purpose, no robots.txt policy and no IP list. That makes a firewall rule on its user agent the dependable way to block it.

Checked against ByteDance’s documentation on 19 October 2026

Operator
ByteDance
Purpose
Model training. Crawlers that collect pages to train models. Blocking them does not affect search or AI answers.
robots.txt token
Bytespider
We test with
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)
the string commonly seen in server logs. ByteDance doesn’t document one.
Documentation
None published by the operator.

What Bytespider does

ByteDance doesn’t document Bytespider, so everything said about it comes from observation, not from ByteDance. It is widely reported to collect data for ByteDance’s AI models and to crawl heavily. We found no official page describing it on ByteDance’s sites.

Cloudflare includes Bytespider in the AI crawlers its managed robots.txt blocks, alongside GPTBot, ClaudeBot and CCBot.

The Bytespider user agent

RequestUser agent string
Pages, as seen in logsMozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)

Not documented by ByteDance. Match on Bytespider, case-insensitively.

How Bytespider reads robots.txt

User-agent: Bytespider
Disallow: /

Add it, but don’t rely on it alone. With no published policy, nothing commits Bytespider to obeying robots.txt, and site owners have reported it doesn’t.

Bytespider IP addresses

ByteDance publishes no IP list and no reverse DNS names for Bytespider, so there is no way to verify one. A request with its user agent may be the real crawler or an impostor; either way, a user-agent rule refuses it.

Block Bytespider at your firewall or CDN

robots.txt asks; a firewall enforces. With no published robots.txt policy, this is the dependable way to keep Bytespider out. Match the name case-insensitively and anywhere in the string, never a full string or a version.

Cloudflare

In Security → WAF → Custom rules, create a rule with this expression and the action Block. Cloudflare’s contains is case-sensitive, hence lower().

(lower(http.user_agent) contains "bytespider")

Cloudflare also has one-click controls: AI bot policies, which block AI crawlers by what they do (search, agent or training), and AI Crawl Control, which lets you allow or block crawlers one by one and shows which of them fetch your pages.

nginx

# in the http block
map $http_user_agent $blocked_bot {
    default 0;
    ~*bytespider 1;
}

# in the server block
if ($blocked_bot) { return 403; }

Block Bytespider

Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt. ByteDance publishes no robots.txt policy for Bytespider, so pair this with a firewall rule on its user agent.

User-agent: Bytespider
Disallow: /

Allow Bytespider when everything else is blocked

A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.

User-agent: Bytespider
Allow: /

User-agent: *
Disallow: /

Does Bytespider reach your site?

The checker applies your robots.txt the way Bytespider does, then requests your page with the user agent above, alongside 29 other bots, and opens the report on Bytespider.

Questions about Bytespider

What is Bytespider?

A web crawler associated with ByteDance, TikTok’s parent company. ByteDance publishes no documentation for it; it is widely reported to collect data for ByteDance’s AI models.

Does Bytespider respect robots.txt?

ByteDance hasn’t published a policy, and site owners have reported that it ignores robots.txt. Disallow it in robots.txt and also block its user agent at your firewall.

How do I block Bytespider?

Add a Bytespider group with Disallow: / to robots.txt, and refuse requests whose user agent contains “bytespider” at your server or CDN, for example with the Cloudflare or nginx rule on this page.

What IP addresses does Bytespider use?

ByteDance doesn’t publish them, so requests claiming to be Bytespider can’t be verified.

Other ai training: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, GoogleOther, Meta-ExternalAgent.

Build a complete file with the robots.txt generator.