Oxidizing your Python with some Rust

5 min readNov 4, 2020

Python and its ecosystem are filled with a plethora of libraries and packages that make up some of Stack Overflow’s most popular tools & technologies [1]. The ecosystem is thoroughly embedded and the ease of use and learning curve is beginner-friendly. This is Python’s main strength: someone who is curious about the language can pick it up with little to no headache, especially with the vast number of resources to help kick-start their learning. However, with most things in life, the ease of use has its drawbacks.

Figure 1: Energy, Time, and Memory Consumption across programming languages

Making Sense of Energy, Time, and Memory Performance

The figure above displays the energy, time, and memory performance across several programming languages. Python lags in energy, and time performance, but ranks slightly higher in the memory performance category[2]. Although Python is easy to use, its performance is not as strong as many other languages.

These runtime attributes are critical to modern engineering and are almost positively associated with improved software. When software can run faster, with less memory, and use less power (good for the environment when thinking of computing at-scale), it’s almost always better.

On the opposite end of the spectrum, Rust ranks second in energy, and time performance — only trailing behind C — and reasonably well in memory consumption [2]. Rust’s values, from the rust-lang website, give us a sense of what their design goals are: [3]

Performance
Reliability
Productivity

Rust competes with C and C++ with respect to performance. Rust has some very key guarantees that make it reliable. An excerpt from the Rust website states, [3]

“Rust’s rich type system and ownership model guarantee memory-safety and thread-safety — enabling you to eliminate many classes of bugs at compile-time.”

An analysis of chromium, the core of Google Chrome & Chrome OS, found that 70% of their bugs are caused by memory safety issues [4]. Rust eliminates these classes of bugs at compile-time with the help of the Borrow Checker removing this class of bugs at compile time [5]. You can have confidence that a successful compilation will remain reliable and perform well.

Figure 2: Chromium High Severity Bug Classification

Getting Started — Building Extensions for Python

The pyo3 Rust crate (package) lets us build Python extensions that can allow us to build highly efficient, highly performant, and safe code in Rust. At the same time, Python users can utilize these tools without learning a whole new language.

In this example, I will go over a simple example of building a Python extension in Rust. We will be using the maturin build system by pyo3. This build system works seamlessly for building python packages from Rust. To start this tutorial I will be creating the package oxidizer that contains a function that calculates the sum of primes to a given value n. Wherever you see oxidizer, please note you should replace this with your own project / package / module name.

Please install Rust, Python, and maturin before continuing.

New Package

First, we need to create a new Rust library using

cargo new oxidizer --lib

This will generate our Rust library with a ./src directory and some Cargo files. Next, let’s add the pyo3 dependency to our ./Cargo.toml file and specify the crate-type to be a cdylib.

Next, create a pyproject.toml file to specify the build system for our python project.

Next, we’ll need to specify our setup.py file with the following:

Implementation

Here is a trivial implementation of calculating the sum of primes from 0 to n. The key parts are the #[pyfunction] and #[pymodule] macros. These macros are what will be publicly visible to the python package. I will make use of using the primal crate to calculate the primality of a number. Both Rust and Python will be making use of the Miller-Rabin primality test [6].

Now, all we need to do is create the remaining folders and files for maturin. I’ll create the ./oxidizer folder and create the __init__.py file inside with the following content:

from .oxidizer import sum_primes

Now we build our Python package with maturin

maturin build --release

This builds the python wheels into ./target/wheels/ . We can now install this package with:

cd target/wheels && pip install oxidizer-{...}

Alternatively,maturin can install locally for you with the develop command

maturin develop

I highly recommend going through the maturin documentation for an in depth explanation.

Benchmarks

Below I’ve implemented a pytest micro-benchmark that compares the two implementations.

Here are the results.

Figure 3: Visualization of Benchmark Results

The benchmark speaks for itself. Rust has significantly better performance and runs faster in this benchmark. Roughly speaking, the benchmark displays a ~20x performance increase by using Rust as opposed to Python in this scenario in each benchmark attribute.

By writing Python extensions in Rust, there are many performance benefits in terms of time. We also get memory safety with compile-time guarantees that no data races will occur (assuming there’s no use of the unsafe keyword). End-users don’t need to know the implementation details of our Rust package, but they can reap the benefits without sacrificing the comfort and ergonomics of using Python. Python may not ever be replaced by another language, especially in the data space. However, there is always room for improvement. This implementation style is already used in projects like TensorFlow , and PyTorch where the core of the library is built in C++, and the bindings are made available to Python. By implementing functionality to lower-level languages like Rust, and having the ability to use the library in higher level languages like Python, it will provide us memory safety guarantees and significantly improve the performance of the package without the need to change the end user’s expectations and experience of working in Python.

Appendix

Here are some Rust resources if this blog post hasn’t convinced you to get take a sip of the sweet, sweet Rust kool-aid yet.