Bytespider: should you allow or block it?

The scary crawler of 2024 is a shrinking one in 2026. The advice to block it with robots.txt was always wrong.

Bytespider earned its reputation. In 2024 it was the single most active AI crawler on the web, hammering servers and reaching pages owners had marked off-limits. The panic that followed produced a wave of guides telling everyone to block it, most of them with a robots.txt rule that Bytespider simply ignores.

This guide covers what Bytespider is, whether it still respects robots.txt, whether it is even a threat worth worrying about anymore, and the block that actually works. The short version: block it, but do it at your firewall, not in a text file it does not obey.

What is Bytespider?

Bytespider is the web crawler operated by ByteDance, the company behind TikTok, Douyin, CapCut, and Toutiao. It scrapes public pages to build training data for ByteDance's AI systems, chiefly Doubao, its ChatGPT competitor, and to feed recommendation and AI features across its apps. It reads your content, but it does not send visitors or cite you back.

Doubao is the reason it crawls so hard. Doubao is one of China's most-used AI chatbots, and training a model at that scale takes enormous volumes of text, which is what Bytespider is out collecting. Your blog post is raw material for a model you will probably never use.

The scope is broad by design. Bytespider gathers content to power recommendation algorithms, AI features, and training sets across ByteDance's whole product suite, not one app. That is why it crawls so widely: it is feeding a company, not a single model.

Its arc follows ByteDance's AI push. Bytespider ramped hard through 2023 and 2024 as ByteDance raced to build Doubao, which is how it briefly became the busiest AI crawler on the web. As that first training rush passed and more sites blocked it, its volume fell off sharply.

One distinction matters before you touch any config: Bytespider is not the only ByteDance bot. TikTokSpider and Doubaobot are separate user agents tied to TikTok and Doubao surfaces, so blocking Bytespider does not cut you off from those. This guide is about Bytespider, the training and scraping crawler.

Does Bytespider respect robots.txt?

No, not reliably. ByteDance says Bytespider follows the robots exclusion protocol, but the evidence says otherwise, and this is the single most important fact about it. Server logs repeatedly show Bytespider fetching your robots.txt and then crawling the very pages you marked Disallow.

This is not a one-off complaint. A 2026 analysis of AI crawler behavior found Bytespider among the agents that reached disallowed pages on roughly half the European sites that named them in robots.txt. Compliant crawlers like GPTBot and ClaudeBot honor the file, while Bytespider treats it as a suggestion.

The pattern in the logs is consistent enough to plan around. Bytespider requests your robots.txt, so it clearly reads the file, then issues follow-up requests to paths under Disallow anyway. Whether that is deliberate policy or a broken implementation does not matter to you, because the effect is identical.

That single fact rewrites the whole how-to. Every guide that says "just add Disallow for Bytespider" is giving you a rule the crawler will read and then step around. You need enforcement, not a polite request.

Is Bytespider still a threat in 2026?

Far less than it was, and the numbers are stark. Bytespider was the most active AI crawler in 2024, but its volume has collapsed, so the 2024-era panic guides are describing a problem that has mostly receded. It is now a minor crawler that happens to behave badly, not the web-wide bandwidth threat it once was.

Cloudflare's network data tells the story. Bytespider fell from 42% of AI crawler traffic in May 2024 to about 7% a year later, dropping from the number one spot to number five, as its raw request volume declined roughly 85%. Across all crawlers it slid from second place to eighth.

When it does show up, though, it is still rude. Bytespider is documented crawling at a higher request rate than GPTBot and continuing under server load, with reports of hundreds of gigabytes pulled from a single large site in a month. Lower total volume does not mean gentle; it means fewer sites get hit hard.

So calibrate your reaction. Bytespider is worth blocking, but it is not an emergency for most sites anymore, and if you already blocked it in 2024 you can leave that in place and move on. The engines eating real crawl budget today are GPTBot, ClaudeBot, and Meta-ExternalAgent.

Should you allow or block Bytespider?

Block it, unless you specifically want your content in ByteDance's AI. Bytespider sends no referral traffic, cites nobody, and feeds training rather than any surface that links back to you, so blocking it protects your content and cuts server load at zero SEO cost. Blocking Bytespider has no effect on your Google or Bing rankings.

The one real reason to allow it is intent. If you publish content you want inside Doubao or aimed at ByteDance's Chinese-language audience, letting Bytespider train on you is the price of entry, the same way you let GPTBot in if you want to be in ChatGPT's knowledge. For a Western B2B site with no China play, that upside is close to zero.

For a concrete default, block it. Unless ByteDance is a market you actively sell into, add the firewall rule, keep the robots.txt line for honesty, and treat allowing it as the exception that needs a reason. Most sites will never find that reason.

How do you actually block Bytespider?

You block it at the server, because it ignores robots.txt. Add the robots.txt rule for good hygiene and honest signaling, then enforce the block with a firewall rule that matches its user agent. The robots.txt line is the request; the firewall rule is what makes it stick.

The robots.txt line itself is two rows: a line reading User-agent colon Bytespider, then Disallow colon slash. Keep it, because it states your intent clearly and blocks the crawlers that do behave. Just do not rely on it to stop this one.

In Cloudflare, go to Security, then WAF, and create a custom rule: if the user agent contains Bytespider, block it or serve a managed challenge. The same idea works in NGINX or an .htaccess rule that returns a 403 on that user agent. This targets the one thing the crawler cannot easily change per request, its identifying string.

Then verify the rule took. For a day after it goes live, watch your logs: requests carrying the Bytespider user agent should come back as 403 or a challenge, not 200. If they are still getting through, the rule is usually matching the wrong field or sitting behind a cache that answers the bot before your firewall ever sees it.

One thing blocking will not do is undo the past. Anything Bytespider already scraped is already in ByteDance's hands, and a block only stops future crawls. That is a reason to block sooner rather than later, but not a reason to expect your old content to disappear from a model that has already trained on it.

Do not try to block it by IP. Bytespider rotates through thousands of addresses, and some sit on shared cloud ranges you would break legitimate services by banning. The user-agent rule is both simpler and more reliable, and it is the approach the crawler-control tools converge on.

robots.txt is a sign on the door. Bytespider reads the sign and walks in anyway. If you want it out, you need a lock, and the lock is a firewall rule.The one-line rule

How is Bytespider different from other AI crawlers?

Bytespider stands out for taking without giving and for ignoring the rules other crawlers follow. GPTBot, ClaudeBot, and Meta-ExternalAgent all honor robots.txt and feed products that can cite or surface you. Bytespider honors nothing and feeds an ecosystem that, for most Western sites, sends nothing back.

The takeaway is that not all crawlers deserve the same policy. Blocking GPTBot or Google-Extended is a real tradeoff, because you may want the visibility their products offer, whereas blocking Bytespider is close to free. We cover the friendlier bots in our GPTBot allow-or-block guide and Google-Extended guide.

How do you tell if Bytespider is hitting your site?

Check your server logs for the user agent, which contains the string Bytespider. Filter your access logs or CDN analytics for that token and you will see its request volume, the paths it hits, and whether it is touching pages you disallowed. That last check tells you fast whether robots.txt is being ignored on your own site.

Most CDNs and analytics tools now break out AI bots by name, so you can watch Bytespider alongside GPTBot and the rest. If its volume is trivial, which for many sites it now is, you can block it once and stop thinking about it. If it is heavy, the firewall rule pays for itself in saved bandwidth immediately.

Set a rough threshold so you are not watching forever. If Bytespider is a fraction of a percent of your requests, a one-time block is plenty. If it spikes into real bandwidth, put an alert on that user agent so you catch the next surge before it shows up on your hosting bill.

Crawling is only half the picture, though. Knowing which bots read you is a control question; knowing which AI answers actually name you is a visibility question, and the two are separate. That second one is what our AI visibility guide is about.

Frequently asked questions

Block it at the firewall, then move on

Do this next: add a robots.txt Disallow for Bytespider for the record, then create a WAF or server rule that blocks any request whose user agent contains Bytespider. That is the version that actually holds.

Then redirect the worry. Bytespider is a shrinking, low-payoff crawler, so once it is blocked, spend your attention on the engines that send real traffic and citations, and measure where you show up with your AI visibility fundamentals in hand.