Numba vs NumPy vs cuPy in Python: A Comparison

CuPy vs. Numpy vs. Numba

TDS Editors Rinu Gour Amit Shekhar

Hello Folks, Today we will learn Numba vs NumPy vs cuPy in Python: A Comparison.

Python is a versatile language, but when it comes to performance, especially in numerical computations, it can sometimes fall short. This is where libraries like Numba, Numpy, and Cython come into play. Each of these libraries offers unique advantages and optimizations. In this blog post, we’ll compare Numba, Numpy, and Cython, focusing on their syntax and time complexity.

The Eternal Debate!

Numba, Cupy, and Numpy are three popular libraries used for numerical computation in Python. While they share some similarities, each has its strengths and weaknesses, Here’s a brief comparison:

1. NumPy (Numerical Python)

NumPy is the fundamental package for scientific computing in Python. It provides support for arrays, matrics, and many mathematical functions.

Example 1:

import numpy as np

# create a 2D array
arr = np.array([[1,2,3],[4,5,6]])

# perform matrix multiplication
result = np.matmul(arr,arr.T)

print(result)

Here, we see a simple eg:

import numpy as np

# creating an array
arr = np.array([1,2,3,4,5])

# perform an operations
res = np.sum(arr)
print(res) 

#output=>15

• Maturity: Most mature and widely used library(created in 2005).
• CPU-only: Runs on CPU (Central Processing Unit).
• Dynamic Tying: Works with dynamic typing which can lead to slower performance.
• Broadcasting: Supports broadcasting, which allows operations on an array with different shapes.

import numpy as np

# Matrix of shape (3, 3)
matrix = np.array([[1, 2, 3], 
                   [4, 5, 6], 
                   [7, 8, 9]])

# Vector of shape (3,)
vector = np.array([1, 0, 1])

# Broadcasting will add the vector to each row of the matrix
result = matrix + vector

print(result)

# [[ 2  2  4]
# [ 5  5  7]
# [ 8  8 10]]

• Linear algebra: Includes a wide range of linear algebra functions (eg numpy.linalg )

import numpy as np

# Square matrix
matrix = np.array([[1, 2],[3, 4]])

# Calculate the inverse
inverse = np.linalg.inv(matrix)

print(inverse)

# [[-2.   1. ]
# [ 1.5 -0.5]]

2. Numba

A portion of Python and NumPy code can be converted into quick machine code using Numba, a Just-In-Time (JIT) compiler. This is especially helpful for speeding up functions with intricate mathematical calculations or loops.

Example 1:

import numba as nb 
import numpy as np

# create a 2D array
arr = np.array([[1,2,3],[4,5,6]])

# Define a function with numba's JIT complier
@nb.jit(nopython=True)
def matmul(arr)
result = zp.zeros((arr.shape[0], arr.shape[1]))
for i in range(arr.shape[0]):
    for j in range(arr.shape[1]):
        for k in range(arr.shape[1]):
            result[i,j]+=arr[i,k]*arr[k,j]
return result

# perform matrix maultiplication
result = matmul(arr)
print(result)

• Just-In-Time(JIT) compilation: Compiles Python code into efficient machine code at runtime.
• CPU and GPU support: Can run on both CPU and GPU (with CUDA or ROCm support).
• Static Typing: Requires static typing, Which can lead to faster performance.
• Limited broadcasting: Broadcasting is not supported compared to Numpy.

import numpy as np
from numba import njit

# A function to add a scalar to an array using Numba
@njit
def add_scalar(arr, scalar):
    result = np.empty_like(arr)
    for i in range(arr.shape[0]):
        result[i] = arr[i] + scalar
    return result

# Example array and scalar
arr = np.array([1, 2, 3, 4, 5])
scalar = 10

# Apply the Numba-optimized function
result = add_scalar(arr, scalar)
print(result)

# output [11 12 13 14 15]

• Linear algebra: Limited linear algebra support as compared to numpy.

import numpy as np
from numba import njit

@njit
def gaussian_elimination(A, B):
    """
    Solves the system of linear equations Ax = B using Gaussian elimination.

    A: Coefficient matrix (n x n)
    B: Right-hand side vector (n,)
    """
    n = len(B)
    # Forward elimination
    for i in range(n):
        # Make the diagonal element 1, and reduce the row
        factor = A[i, i]
        for j in range(i, n):
            A[i, j] /= factor
        B[i] /= factor

        # Eliminate the current column in the rows below
        for k in range(i + 1, n):
            factor = A[k, i]
            for j in range(i, n):
                A[k, j] -= factor * A[i, j]
            B[k] -= factor * B[i]

    # Backward substitution
    x = np.zeros_like(B)
    for i in range(n - 1, -1, -1):
        x[i] = B[i]
        for j in range(i + 1, n):
            x[i] -= A[i, j] * x[j]

    return x

# Coefficient matrix A and right-hand side vector B
A = np.array([[2.0, 1.0, -1.0], 
              [-3.0, -1.0, 2.0], 
              [-2.0, 1.0, 2.0]])

B = np.array([8.0, -11.0, -3.0])

# Solving Ax = B using Gaussian elimination
solution = gaussian_elimination(A.copy(), B.copy())  # Pass copies of A and B since they will be modified
print("Solution:", solution)

#output
Solution: [ 2.  3. -1.]

3. cuPy

cuPy is an open-source package called cuPy makes it possible for NVIDIA GPUs to support NumPy-like syntax. It is perfect for large-scale data and scientific computing because it is made to speed up computations by shifting array operations to the GPU.

Example 1:

import cupy as cp

# create a 2D array on GPU
arr = cp.array([[1,2,3],[4,5,6]])

# perform matrix multiplication on the GPU only
result = cp.matmul(arr,arr.T)

print(result)

• GPU-only: Designed especially for NVIDIA GPUs (with CUDA support)
• Drop-in replacement for Numpy: Almost identical API to Numpy, making it easy to transition.
• GPU acceleration: Leverages GPU acceleration for significant performance boosts.
• Limited CPU support: No optimized for CPU-only execution.
• Linear algebra: Includes a wide range of linear algebra functionalities, Similar to Numpy.

import cupy as cp

# Coefficient matrix A and right-hand side vector B
A = cp.array([[2.0, 1.0, -1.0], 
              [-3.0, -1.0, 2.0], 
              [-2.0, 1.0, 2.0]])

B = cp.array([8.0, -11.0, -3.0])

# Solve Ax = B using CuPy's solve function
x = cp.linalg.solve(A, B)

print("Solution:", x)

# output
# Solution: [2. 3. -1.]

Now, Let’s compare the time complexity of each library for simple matrix multiplication operations.

Note: Here we are considering only Example 1 :

We’ll use the following setup:

• Matrix Size: 1000x1000
• Number of iterations: 100

Here are the results:

Library                     Time(sec)                          Time Complexity

Numpy                        0.23                                O(n^3)
Numba                        0.013                               O(n^3)
cuPy                         0.0013                              O(n^3)

As expected, cuPy is the fastest due to its GPU acceleration. Numba is also significantly faster than Numpy due to its JIT compilation. However, all three libraries have the same time complexity O(n^3) for matrix multiplication.

All three matrix multiplication results will be the same.

:) Happy Coding!

[embed]Numpy: Your Secret Weapon for Data Science and Machine Learning (Part I) Numpy Essentials: The Backbone of Efficient ML and Data Sciencepython.plainenglish.io