Nvidia cuda and TensorRT install on a Tesla GPU for Deeplearning Model inference

In this article we are going to install TensorRT on a Tesla T4 GPU with all the required packages like the cuda toolkit, nvidia drivers, cuDNN, Python packages, Deeplearning packages like the ONNX runtime etc…

Pre-requisites:

In this install I have used ubuntu22.04 on a NVIDIA Tesla T4 GPU chipset

python3 with pip is mandatory.

Step1:

Before we begin lets prepare the OS with required packages…

$ sudo apt-get install python3-pip
$ pip3 install torch
$ sudo apt-get update
$ sudo apt install nvidia-driver-535

With the above install we should be able to talk to the Nvidia GPU chip onboard. Lets checkit

Check 1:

The following checks are needed..

You can run the script, which can be found below

[**https://github.com/rangapv/Nvidia/blob/main/check1.sh](https://github.com/rangapv/Nvidia/blob/main/check1.sh)**

Clearly from the above output we have the Nvidia driver up and running!

Step 2:

Let install the cuDNN now. The cuDNN version I have used is 9.8.0

$ sudo wget https://developer.download.nvidia.com/compute/cudnn/9.8.0/local_installers/cudnn-local-repo-ubuntu2204-9.8.0_1.0-1_amd64.deb

$ sudo dpkg -i cudnn-local-repo-ubuntu2204-9.8.0_1.0-1_amd64.deb

$ sudo cp /var/cudnn-local-repo-ubuntu2204-9.8.0/cudnn-local-8138232B-keyring.gpg /usr/share/keyrings/

$ sudo apt-get update

$ sudo apt-get -y install cudnn

With the above steps we have installed cuDNN now lets verify

Check 2:

cuDNN samples is already installed lets build on it. Before we get there we need these libraries installed.

**$ sudo apt-get install libfreeimage3 libfreeimage-dev**

Now first move the samples

**$ cp -r /usr/src/cudnn_samples_v9/ $HOME**

and now the build….

~/cudnn_samples_v9/mnistCUDNN$ make clean
rm -rf *o
rm -rf mnistCUDNN
~/cudnn_samples_v9/mnistCUDNN$ make
CUDA_VERSION is 12080
Linking agains cublasLt = true
CUDA VERSION: 12080
TARGET ARCH: x86_64
HOST_ARCH: x86_64
TARGET OS: linux
SMS: 50 53 60 61 62 70 72 75 80 86 87 90
nvcc warning : Support for offline compilation for architectures prior to '<compute/sm/lto>_75' will be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
/usr/local/cuda/bin/nvcc -I/usr/local/cuda/include -I/usr/local/cuda/include -IFreeImage/include -ccbin g++ -m64 -std=c++11 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_53,code=sm_53 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_72,code=sm_72 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_87,code=sm_87 -gencode arch=compute_90,code=sm_90 -gencode arch=compute_90,code=compute_90 -o fp16_dev.o -c fp16_dev.cu
nvcc warning : Support for offline compilation for architectures prior to '<compute/sm/lto>_75' will be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
g++ -I/usr/local/cuda/include -I/usr/local/cuda/include -IFreeImage/include   -std=c++11 -o fp16_emu.o -c fp16_emu.cpp
g++ -I/usr/local/cuda/include -I/usr/local/cuda/include -IFreeImage/include   -std=c++11 -o mnistCUDNN.o -c mnistCUDNN.cpp
/usr/local/cuda/bin/nvcc -ccbin g++ -m64 -std=c++11 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_53,code=sm_53 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_72,code=sm_72 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_87,code=sm_87 -gencode arch=compute_90,code=sm_90 -gencode arch=compute_90,code=compute_90 -o mnistCUDNN fp16_dev.o fp16_emu.o mnistCUDNN.o -I/usr/local/cuda/include -I/usr/local/cuda/include -IFreeImage/include -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib64 -lcublasLt -LFreeImage/lib/linux/x86_64 -LFreeImage/lib/linux -lcudart -lcublas -lcudnn -lfreeimage -lstdc++ -lm
nvcc warning : Support for offline compilation for architectures prior to '<compute/sm/lto>_75' will be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).

And the output can be seen here…

$ ./mnsitCDNN

The TEST PASSED.

Step 3:

Now lets install the cuda toolkit the Star of the show. The cuda version I have used is 12.8

$ sudo wget https://developer.download.nvidia.com/compute/cuda/12.8.1/local_installers/cuda-repo-ubuntu2204-12-8-local_12.8.1-570.124.06-1_amd64.deb
$ sudo dpkg -i cuda-repo-ubuntu2204-12-8-local_12.8.1-570.124.06-1_amd64.deb
$ sudo cp /var/cuda-repo-ubuntu2204-12-8-local/cuda-*-keyring.gpg /usr/share/keyrings/
$ sudo apt-get update
$ sudo apt-get -y install cuda-toolkit-12-8

export PATH=${PATH}:/usr/local/cuda-12.8/bin
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda-12.8/lib64

With cuda installed and added to the path lets check it.

Check 3:

Lets verify if the cuda install was a success….

$ /usr/local/cuda-12/bin/nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0

Step 4:

Finally we are ready to install TensorRT. The version I have selected is 10.9.0

$ sudo wget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/10.9.0/local_repo/nv-tensorrt-local-repo-ubuntu2204-10.9.0-cuda-12.8_1.0-1_amd64.deb

$ sudo dpkg -i ./nv-tensorrt-local-repo-ubuntu2204-10.9.0-cuda-12.8_1.0-1_amd64.deb

$ sudo cp /var/nv-tensorrt-local-repo-ubuntu2204-10.9.0-cuda-12.8/nv-tensorrt-local-AD7406A2-keyring.gpg /usr/share/keyrings/

$ sudo apt-get update

$ sudo apt-get install tensorrt

Check 4:

Lets check the tensorRT install if successfull..

$ dpkg-query -W tensorrt tensorrt 10.9.0.34–1+cuda12.8

Also the sample….

python3
>>> import tensorrt
>>> print(tensorrt.__version__)
>>> assert tensorrt.Builder(tensorrt.Logger())

Step 5:

Now lets install the ONNX for deep learning requirement

$ pip3 install onnx

$ pip3 install onnxruntime-gpu

$ pip3 install tf2onnx skl2onnx

Keep Building!

If you have any issues, you can open an issue in the github repo page or alternatively contact me at rangapv@yahoo.com and you can also find me on X.com(twitter) Rangaswamy P V

Some of my other related articles….

[embed]Streamlit Frontend for your container statistics In this article we will deploy a python application using Streamlit as a frontend wrapper.rangapv.medium.com

[embed]Deploying Ray Cluster for AI/ML workloads on a Kubernetes Cluster In this article we are going to install a Ray operator in a Kubernetes cluster. We have looked into setting up a…rangapv.medium.com

[embed]Terraform scripts to create a K8s Cluster using “kubeadm” in AWS from scratch Here in this article we will create a custom Kubernetes cluster on the AWS Cloud and the following are the…rangapv.medium.com

[embed]Prometheus Install on a k8s Cluster in AWS In this article we will Install a Prometheus server to achieve the Observability goals for a Kubernetes cluster. To…rangapv.medium.com