Bytespider: blocking ByteDance's crawler properly

Operator

ByteDance

Type

Training crawler

robots.txt token

Bytespider

Follows robots.txt

Widely reported as inconsistent

Used for

Collecting web content for ByteDance, including training its AI models

User agent

Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])

How to verify

Match the user agent; blocking reliably usually needs a firewall rule rather than robots.txt alone

What is Bytespider?

Bytespider appears in many server logs at high volume. Unlike most AI crawlers from large Western operators, it offers little public documentation, and its value to site owners is hard to see: it does not power a search product that sends meaningful referral traffic in most markets.

That combination, high load and little return, is why Bytespider is one of the most commonly blocked agents on the web.

What happens if you block Bytespider?

Add a Bytespider group with Disallow: / to express your preference, then enforce it. On Cloudflare and most CDNs you can create a rule that blocks requests whose user agent contains Bytespider. Blocking it has no known effect on your visibility in the major AI assistants or search engines.

Should you allow Bytespider?

For most sites, block it. It consumes bandwidth and server capacity without an obvious visibility benefit. The exception is businesses that specifically want presence in ByteDance products, and even then you may want rate limiting rather than open access.

Check whether Bytespider can reach your site

Enter your domain or a specific page. The checker reads your live robots.txt, picks the group Bytespider would use, and applies the longest matching rule, the same way major crawlers resolve conflicts.

robots.txt rules for Bytespider

Put each group in the robots.txt file at the root of your domain. User agent names are matched without regard to case, and the most specific matching group wins.

User-agent: Bytespider
Allow: /

User-agent: Bytespider
Disallow: /

User-agent: Bytespider
Disallow: /account/
Disallow: /checkout/
Allow: /

The rule most people get wrong: once Bytespider has its own group, it ignores everything under User-agent: *. If your wildcard group disallows paths such as /admin/, repeat those lines inside the Bytespider group too.

Because Bytespider is not reliably governed by robots.txt, a robots.txt rule states your preference but does not enforce it. To enforce a block, add a firewall rule. On Cloudflare, a custom WAF rule with this expression and the Block action does it:

(http.user_agent contains "Bytespider")

How to verify real Bytespider traffic

Match on the Bytespider token in the user agent. Because it is so often blocked, some scrapers imitate other agents instead of imitating Bytespider, so pair the rule with general rate limiting on unusual request volumes.

To see how often it visits, count requests by user agent in your access log. On a typical Nginx server:

grep -i "Bytespider" /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

Bytespider compared with similar agents

See all 20 agents side by side in the AI crawler directory.

Bytespider FAQ

Abd Shanti

Co-founder and strategy lead at Outline Technologies, the team behind FreeGPTSEO and AI Citation Monitor. Abd works on how brands get found and cited by search engines and AI assistants.