🧀 BigCheese.ai

Social

AI crawlers need to be more respectful

🧀

Read the Docs reports an increase in abusive site crawling by AI products, causing issues such as excessive bandwidth costs. The company details how one AI crawler's bug resulted in 73 TB of downloads, amounting to over $5,000 in fees, and another instance where Facebook's content downloader used 10 TB of data. Read the Docs is taking measures to address abuse, including blocking identified AI Crawler traffic, enhancing rate limiting, and CDN caching improvements.

  • 73 TB of data downloaded by one crawler in May 2024.
  • One day saw almost 10 TB of data downloaded by this crawler.
  • Facebook's downloader used up 10 TB in June 2024.
  • Blocking AI crawlers cut bandwidth by 75%.
  • Abusive crawlers causing financial strain on Read the Docs.