Should you use Rust in LLM based tools for performance?

This article explores the performance differences between Rust and Python in the context of text (code) processing tools integrated with large language models (LLMs). Rust-based tools like Swiftide are compared to Python's Langchain, with a focus on benchmarking involving data processing, generating embeddings, and vector database insertion. Interesting findings about pre-processing steps and overall efficiency are discussed.

• Rust offers zero-cost abstractions and concurrency.
• 90% of time in Rust is in the ONNX runtime.
• Langchain's Markdown step caused a CPU bottleneck.
• Benchmarks were done on a NVIDIA A6000 GPU.
• The Rust Book embeddings take 3 seconds on GPU.