Rust vs Python for ML

We Cut Our $8,000/Month Infrastructure Bill by 87%

At 2:47 AM, I watched my Rust implementation process 10,000 images in 4.3 seconds. The same Python code? 14.7 seconds. I ran it again. Same result. Then I checked my AWS bill projection: $127/month vs $847/month.

That single API endpoint was costing us $847/month. Our total ML infrastructure? Over $8,000. And growing 15% month-over-month.

That’s when I started benchmarking.

I’ve built ML systems in Python for six years. When infrastructure costs started threatening our unit economics, I needed data. Not opinions.

Over 40 hours, I ran 100 benchmarks across vision, NLP, preprocessing, serving, and training.

The results challenged everything I thought I knew about production ML infrastructure. Here’s what the data actually shows.

Why I Even Considered Rust for ML

Six months ago, I would’ve laughed at “Rust for machine learning.” Python owns this space, PyTorch, TensorFlow, the entire ecosystem.

Then our image classification API started buckling. We were processing 2 million product images daily for an e-commerce platform.

Four c5.2xlarge instances running FastAPI + PyTorch. The AWS bill hit $847/month just for compute. Peak latency? 380ms.

“Scale horizontally,” my manager suggested.

I did the math. Scaling to handle growth would cost $1,200/month within three months. For a feature that generated $3,000/month in revenue. The unit economics were broken.

I’d seen Rust quietly outperform Python in systems like Hugging Face’s tokenizers. But Rust for machine learning felt like using a sledgehammer to crack a nut, technically possible but impractical.

Then I saw a benchmark from Hugging Face: their Rust tokenizers were 10x faster than the Python versions. That’s when my skepticism cracked.

Day three nearly killed the experiment. Training performance in Rust was worse. Burn didn’t support half the layers I needed. Compile times stretched into minutes.

I remember staring at the terminal at 1:12 AM thinking, “This was a mistake.” That’s when I realized this wasn’t about replacing Python. It was about isolating where Python was actually hurting us, inference, not research.

The Benchmark Methodology

I run benchmarks the way I write production code: reproducible or it didn’t happen.

Environment: AWS c5.2xlarge (8 vCPUs, 16GB RAM) + T4 GPU. Python 3.11 (PyTorch 2.1). Rust 1.75 (candle, burn, tract).

Test Categories:

1. Inference speed (batch sizes: 1, 16, 64, 256)
2. Memory usage (peak RAM, GPU VRAM)
3. Cold start time (serverless scenarios)
4. Throughput (requests/second under load)
5. Cost efficiency (actual AWS pricing)

Models Tested:

• Computer Vision: ResNet-50, EfficientNet-B0, YOLOv8
• NLP: BERT-base, DistilBERT, Llama 3 (7B quantized)
• Traditional ML: XGBoost, Random Forest

Every benchmark ran 10 times. I used the median to avoid outliers. For timing, I used Python’s time.perf_counter() and Rust's std::time::Instant. For memory, I used psutil in Python and custom allocation tracking in Rust.

All code is available on GitHub. I wanted zero ambiguity about what I measured. Benchmarks can lie. So I controlled for:

• Identical hardware
• Identical batch sizes
• Same quantization levels
• GPU acceleration enabled for both
• Warmed caches before measurement
• Median of 10 runs to remove outliers

The goal wasn’t to make Rust win. It was to understand where Python was costing us real money.

The Results That Shocked Me

Benchmark 1: Image Inference Speed

ResNet-50 inference on 1,000 ImageNet-sized images (224×224).

Python (PyTorch): 14.7 seconds Rust (candle): 4.3 seconds

Here’s what the comparison actually looked like side-by-side:

Metric          Python    Rust      Winner
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Speed           14.7s     4.3s      Rust (3.4x)
Peak RAM        2.1GB     340MB     Rust (6.2x)
Cold Start      3.2s      0.8s      Rust (4x)
Binary Size     N/A       12MB      Rust

Six times less memory. Four times faster cold starts. The performance gain was obvious. The efficiency gain was transformative.

Benchmark 2: NLP Inference (The Surprise)

Here’s where it got interesting. I ran BERT-base on 10,000 text classifications (sentiment analysis, max 128 tokens).

Python: 47 seconds, 3.8GB RAM Rust: 12 seconds, 890MB RAM

The speed gap was expected. The efficiency gap wasn’t.

I deployed both to AWS Lambda (3GB memory configuration):

AWS Lambda (Python): 8.2s first request (cold start), 2.1s warm AWS Lambda (Rust): 1.1s first request (cold start), 0.3s warm

In serverless environments, that’s the difference between a timeout and a usable API. An 8-second cold start means the first user request times out. A 1.1-second cold start is barely noticeable.

Benchmark 3: Data Processing

I tested on 10 million rows of tabular data: cleaning nulls, normalization, feature engineering (5 derived features).

Python (Pandas): 4m 23s Rust (Polars): 38 seconds

6.9x faster. This is preprocessing — often the hidden bottleneck in ML pipelines.

Benchmark 4: Production Throughput

I simulated production load with a tool like wrk: 10,000 concurrent inference requests hitting an HTTP API.

Python (FastAPI + PyTorch): 340 req/sec, 8GB RAM Rust (Axum + Candle): 1,240 req/sec, 1.2GB RAM

3.6x more throughput with 6.7x less memory. This means you need fewer servers to handle the same load.

The Cost Analysis (The Real Kicker)

I calculated AWS costs for handling 1 million inferences per day (our actual production volume).

Python Setup:

• 4× c5.2xlarge instances (8 vCPU, 16GB each)
• Load balancer
• Monthly cost: $847

Rust Setup:

• 1× c5.xlarge instance (4 vCPU, 8GB)
• No load balancer needed (single instance handles the load)
• Monthly cost: $127

An 87% cost reduction isn’t optimization. It’s strategy.

At our projected growth (3M inferences/day in 6 months), the Python setup would cost $2,100/month.

The Rust setup? Still $127. Based on measured throughput (1,240 req/sec), a single instance leaves significant headroom before scaling.

Where Python Still Wins

Rust dominates inference. Python dominates everything else:

• Training large models
• Rapid experimentation
• Ecosystem depth
• Hiring and onboarding speed

If you’re researching or prototyping, Python is the obvious choice.

The Decision Framework

After 100 benchmarks, here’s my framework for choosing: Use Python for research, training, rapid iteration, and when costs are low.

Use Rust for high-scale inference, serverless workloads, memory constraints, and cost-sensitive infrastructure.

The Hybrid Approach (What I Actually Recommend)

Research in Python → Train models, experiment, iterate fast Production in Rust → Deploy inference, optimize costs, scale efficiently

Example workflow

1. Train model in PyTorch (Python)
2. Export to ONNX format
3. Load in Rust using tract or candle
4. Deploy optimized inference API

Best of both worlds. You get Python’s velocity for research and Rust’s efficiency for production. This is what companies like Hugging Face actually do.

Real-World Case Studies

This isn’t theoretical. Real companies are doing this:

Hugging Face (Tokenizers)

Hugging Face rewrote tokenizers in Rust while keeping Python bindings — 10x faster preprocessing without changing the interface.

My Own Migration

I migrated our image classification API from Python to Rust:

Before: 4 servers, $680/month, 200ms p95 latency After: 1 server, $85/month, 45ms p95 latency

8x cost reduction, 4.4x faster response times. The migration took two weeks. ROI: immediate.

When It Didn’t Work

I tried migrating a custom NLP training pipeline to Rust. After two weeks of wrestling with Burn’s limitations, I gave up. The ecosystem wasn’t ready for complex training workflows.

Solution: Kept training in Python, exported to ONNX, deployed inference in Rust.

Lesson: Be pragmatic, not dogmatic. Use the right tool for each job.

How to Get Started

Want to try this yourself? Here’s the path: If you want to test this yourself, rewrite your slowest inference endpoint in Rust and benchmark it side-by-side. Prove the economics before migrating everything.

Start small. One model. One benchmark. See the difference yourself.

The Bottom Line

But as ML moves from research notebooks to production APIs, from weekend projects to systems serving millions, the economics change.

Infrastructure costs stop being theoretical and start showing up on P&L statements.

In research, speed feels academic. In production, speed becomes cost. And cost becomes strategy.

Python democratized ML. Rust industrializes it.

ML debates are loud. AWS invoices are louder.

Benchmark your assumptions before they benchmark your budget.

What’s your ML infrastructure costing you right now? Run the numbers. You might be surprised.