Cython — use C/C++ functions in Python

I was working on a project where I had a C++ file and I needed to use some functions from that file. Now the main language which I was using for my project was Python, and converting the entire C++ source code to Python was a very tedious job…

This gave me a thought, what if there was a way to compile the C++ source code and use the functions from it directly into python? This way, I wouldn’t have to translate the entire C++ code into python.

I googled about this and came across a python module, Cython.

This is Cython’s official github repo. According to the readme:

Cython is a Python compiler that makes writing C extensions for Python as easy as Python itself. Cython is based on Pyrex, but supports more cutting edge functionality and optimizations.

Cython translates Python code to C/C++ code, but additionally supports calling C functions and declaring C types on variables and class attributes. This allows the compiler to generate very efficient C code from Cython code.

Here is a small guide on how to use Cython to call C/C++ functions from python. I have tried to cover as many cases (like C++ structures, array of strings, functions with structure pointers as arguments, etc.) as possible.

First, install Cython using:

pip install cython

This is the sample C++ code which I will be using:

helper.h

#ifndef HELPER_H
#define HELPER_H

#include <string>
#include <vector>
#include <stdint.h>
using namespace std;

#define SIZE 5

typedef struct {
    int length;
    uint64_t numbers[SIZE];
} nums;

string concatenate_all(vector<string> &strings, int &ret_code);
void square_numbers(nums *obj, int &ret_code);

#endif

helper.cpp

#include "helper.h"

string concatenate_all(vector<string> &strings, int &ret_code){
    int len = strings.size();
    if(len == 0){
        ret_code = -1;
        return "";
    }
    string temp = "";
    for(int i=0; i<len; i++){
        temp += strings[i];
    }
    ret_code = 1;
    return temp;
}

void square_numbers(nums *obj, int &ret_code){
    int len = obj->length;
    for(int i=0; i<len; i++){
        if(obj->numbers[i] > 100){
            ret_code = -1;
            return;
        }
    }
    for(int i=0; i<len; i++){
        obj->numbers[i] = (obj->numbers[i])*(obj->numbers[i]);
    }
    ret_code = 1;
}

The code basically consists of two functions:

• concatenate_all() — Takes in a vector of strings, and concatenates them. There is another parameter called ret_code which stores -1 if the vector of strings is empty else 1.
• square_numbers() — Takes in a structure nums (which basically consists of a length variable and an array of numbers) and squares each number. There is another parameter called ret_code which stores -1 if any of the number is greater than 100, else 1.

Now, our objective is to call these functions using Python. For this, we will create 3 files:

• wrapper.pyx — This will contain the Python wrappers for the C++ functions (i.e., the functions which will be called by our Python file).
• setup.py — This file contains the build instructions for our wrapper and the C++ file.
• caller.py — This is the Python script in which we need to call the C++ functions.

Let’s start with wrapper.pyx

# cython: c_string_type=unicode, c_string_encoding=utf8

from libc.stdint cimport uint64_t
from libcpp.string cimport string
from libcpp.vector cimport vector

cdef extern from "helper.h":
    ctypedef struct nums:
        int length
        uint64_t numbers[5]

    string concatenate_all(vector[string] &strings, int &ret_code)
    void square_numbers(nums *obj, int &ret_code)

def w_concatenate_all(vector[string] &strings, int &ret_code):
    return concatenate_all(strings, ret_code), ret_code

def w_square_numbers(nums obj, int &ret_code):
    cdef nums *nums_ptr = &obj
    square_numbers(nums_ptr, ret_code)
    return obj, ret_code

• The first line tells Cython how to encode C-type strings (if not mentioned we will have to encode the strings to bytes in Python before calling the function).
• In the cdef extern from “helper.h” section, we declare our structure which should have the same parameters as the one defined in our helper.h file, and we also write the declarations of the C++ functions we want to use.

NOTE: The cdef statements are used to declare C variables, functions, etc.

• Then we define the Python analogous of the C++ functions. Note that we cannot directly pass nums obj in our Python function because nums* is now a Python variable and Python does not support pointers. Instead, we used a little workaround by this line:

cdef nums *nums_ptr = &obj

• Also note that the changes made in obj and ret_code in the C++ function will be reflected here in the wrapper, but these changes do not carry forward to our caller python file, which is why I had the function return the values of those variables (If someone knows or figures out if there is a way to modify the variables passed as reference in the Python file itself directly, do share).

Let’s move on to setup.py

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

extensions = [Extension('wrapper',
                        ['wrapper.pyx', 'helper.cpp'],
                        language="c++",
                        extra_link_args=["-lz"]
                    )]
setup(
    name='wrapper', 
    ext_modules=cythonize(extensions)
)

This is the standard syntax for building the wrapper with our C++ file.

Now, on to caller.py

from wrapper import *

print("------------------------- FOR NUMBERS -------------------------\n\n")

print("CASE 1: A NUMBER GREATER THAN 100")
list_of_numbers = {
    "length": 5,
    "numbers": [1, 20, 104, 3, 22]
}
ret_code = 0
print("* Initial List:", list_of_numbers)
list_of_numbers, ret_code = w_square_numbers(list_of_numbers, ret_code)
print("* Square:", list_of_numbers)
print("* Return Code:", ret_code)

print("\nCASE 2: ALL NUMBERS LESS THAN OR EQUAL TO 100")
list_of_numbers = {
    "length": 5,
    "numbers": [1, 2, 3, 4, 100]
}
ret_code = 0
print("* Initial List:", list_of_numbers)
list_of_numbers, ret_code = w_square_numbers(list_of_numbers, ret_code)
print("* Square:", list_of_numbers)
print("* Return Code:", ret_code)

print("\n\n------------------------- FOR STRINGS -------------------------\n\n")

print("CASE 1: ARRAY LENGTH 0")
list_of_strings = []
ret_code = 0
print("* Initial List:", list_of_strings)
concatenated_string, ret_code = w_concatenate_all(list_of_strings, ret_code)
print("* Concatenated:", concatenated_string)
print("* Return Code:", ret_code)

print("\nCASE 2: ARRAY LENGTH 5")
list_of_strings = ["I ", "am ", "a ", "human ", "being"]
ret_code = 0
print("* Initial List:", list_of_strings)
concatenated_string, ret_code = w_concatenate_all(list_of_strings, ret_code)
print("* Concatenated:", concatenated_string)
print("* Return Code:", ret_code)

I wrote almost all the possible cases for our functions here.

• We first import our wrapper.
• For the square function, we have our first case in we have one number greater than 100 in our array. This should have a return code of -1.
• In the second case of our square function, we should get the squares of all the numbers with a return code of 1.
• For the concatenate function, we have our first case in which we have a 0 length array. This should give a return code of -1.
• In the second case of our concatenate function, we should get the concatenated string with a return code of 1.

This finishes the code. Now, we need the compile our Cython script and then execute it using our caller.py.

This is the command used for compiling:

python setup.py build_ext --inplace

And to execute our script we can simply run it, i.e.:

python caller.py

I wrote a small Bash script for this purpose:

#!/bin/bash

function clean(){
    rm -rf build *.so wrapper.cpp
}

function compile(){
    python setup.py build_ext --inplace
}

function run(){
    python caller.py
}

if [ "$1" == "clean" ]; then
    clean
elif [ "$1" == "compile" ]; then
    clean
    compile
elif [ "$1" == "crun" ]; then
    clean
    compile
    run
elif [ "$1" == "run" ]; then
    run
else
    echo "Invalid argument(s)"
fi

We can simply compile + run our code using:

./util.sh crun

This is the output of the script:

------------------------- FOR NUMBERS -------------------------

CASE 1: A NUMBER GREATER THAN 100
* Initial List: {'length': 5, 'numbers': [1, 20, 104, 3, 22]}
* Square: {'length': 5, 'numbers': [1, 20, 104, 3, 22]}
* Return Code: -1

CASE 2: ALL NUMBERS LESS THAN OR EQUAL TO 100
* Initial List: {'length': 5, 'numbers': [1, 2, 3, 4, 100]}
* Square: {'length': 5, 'numbers': [1, 4, 9, 16, 10000]}
* Return Code: 1

------------------------- FOR STRINGS -------------------------

CASE 1: ARRAY LENGTH 0
* Initial List: []
* Concatenated: 
* Return Code: -1

CASE 2: ARRAY LENGTH 5
* Initial List: ['I ', 'am ', 'a ', 'human ', 'being']
* Concatenated: I am a human being
* Return Code: 1

We can see that this indeed worked as expected.

This was how we can use Python to call C++ functions using Cython.

Thanks for reading!