Nearly 90% of our AI crawler traffic is from ByteDance

Report by HAProxy shows nearly 90% of AI crawler traffic comes from TikTok's parent company Bytedance's Bytespider. HAProxy Edge analytics reveal significant bot traffic, highlighting opportunities and risks for content sites. Depending on business objectives, strategies can include allowing bot discovery or protecting content value by blocking AI crawlers with technical solutions like HAProxy Enterprise.

• Bytedance's Bytespider accounts for roughly 90% of HAProxy's bot traffic.
• AI crawlers pose new challenges for content-heavy sites.
• HAProxy offers tools to mitigate bot risks.
• AI bots often ignore 'robots.txt' directives.
• HAProxy Edge provides global traffic insights.