How to Install and Adapt MuseTalk for NVIDIA’s RTX 5090 (Blackwell Architecture)

The NVIDIA RTX 5090, powered by the cutting-edge Blackwell architecture, represents a leap forward in GPU performance for AI and machine learning tasks. To leverage its capabilities for advanced AI-driven projects like **MuseTalk, a framework for generating realistic talking head videos, specific installation and configuration steps are required. This guide provides a comprehensive, tested approach to installing and adapting MuseTalk on the RTX 5090, based on the process I followed to successfully set it up. For reference, MuseTalk’s official repository is available at MuseTalk GitHub, and its research paper can be found at [arXiv:2410.10122](https://arxiv.org/abs/2410.10122).**

This tutorial assumes you have a Linux system (e.g., Ubuntu) with an NVIDIA RTX 5090 GPU and administrative privileges. Follow these steps carefully to ensure a smooth setup.

Prerequisites

Before starting, ensure the following:

• Hardware: NVIDIA RTX 5090 GPU.
• Operating System: Linux (Ubuntu recommended).
• Software: Anaconda installed for environment management.
• Dependencies: Access to the internet to download packages and weights.
• NVIDIA Drivers: Ensure the latest NVIDIA drivers compatible with the RTX 5090 are installed.

Step 1: Set Up the Conda Environment

To isolate dependencies, create a dedicated Conda environment for MuseTalk with Python 3.10.

Create and Activate the Environment:

conda create -n MuseTalk_5090 python=3.10
conda activate MuseTalk_5090

This ensures a clean environment tailored for MuseTalk and the RTX 5090.

Step 2: Install CUDA Toolkit

The RTX 5090 requires a compatible CUDA version to leverage its full computational power. Install CUDA 12.8.1, which is optimized for the Blackwell architecture.

conda install nvidia/label/cuda-12.8.1::cuda-toolkit

Note: If your system already has a compatible CUDA version installed globally, you can skip this step. Verify compatibility with the RTX 5090 using nvidia-smi.

Step 3: Install PyTorch

MuseTalk requires PyTorch with CUDA support. Install PyTorch 2.7 with CUDA 12.8 compatibility.

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

This command ensures PyTorch is configured to utilize the RTX 5090’s GPU capabilities.

Step 4: Clone MuseTalk Repository and Install Dependencies

Clone the MuseTalk repository and install its required dependencies.

Clone the Repository:

git clone https://github.com/TMElyralab/MuseTalk.git

Install Dependencies:

cd MuseTalk
pip install -r requirements.txt

This installs the core packages needed for MuseTalk to function.

Step 5: Install MMLab Packages

MuseTalk relies on MMLab packages (mmengine, mmcv, mmdet, and mmpose) for computer vision tasks. Install them as follows. Note that mmcv-full might take a while to install:

pip install --no-cache-dir -U openmim
mim install mmengine
mim install "mmcv-full==1.7.2"
mim install "mmdet==3.2.0"
mim install "mmpose==1.3.2"

Note: The specific versions listed are critical for compatibility with MuseTalk and the RTX 5090 setup.

Step 6: Modify MMLab Version Verification

To ensure compatibility, modify the version verification in mmdet and mmpose.

Locate the Files: Navigate to the following directories:

• YOUR_PATH_TO_ANACONDA/anaconda3/envs/MuseTalk_5090/lib/python3.10/site-packages/mmdet
• YOUR_PATH_TO_ANACONDA/anaconda3/envs/MuseTalk_5090/lib/python3.10/site-packages/mmpose

Edit __init__.py: In both directories, find the __init__.py file and set:

mmcv_minimum_version = '1.7.2'

This ensures mmdet and mmpose work with the installed mmcv-full version.

Step 7: Update MMCV Transforms Module

The mmcv transforms module for version 1.7.2 needs to be updated for compatibility.

1. Download MMCV Source Code: Download the source code from MMCV GitHub.
2. Copy Transforms Folder: Copy the mmcv/transforms folder to:

YOUR_PATH_TO_ANACONDA/anaconda3/envs/MuseTalk_5090/lib/python3.10/site-packages/mmcv/

This updates the transforms module to the latest version compatible with mmcv-full==1.7.2.

Step 8: Install FFmpeg

FFmpeg is required for video processing in MuseTalk.

sudo apt update && sudo apt upgrade -y
sudo apt install ffmpeg

Verify the installation with ffmpeg -version.

Step 9: Download Pre-trained Weights

MuseTalk requires pre-trained weights to function. Download them using the provided script:

chmod +x download_weights.sh
sh ./download_weights.sh

This script fetches the necessary model weights and places them in the appropriate directories.

Step 10: Modify MMEngine and MuseTalk Code

To ensure compatibility with PyTorch and the RTX 5090, modify the torch.load() calls to include the weights_only=False parameter.

Modify MMEngine:

• Locate checkpoint.py in:

YOUR_PATH_TO_ANACONDA/anaconda3/envs/MuseTalk_5090/lib/python3.10/site-packages/mmengine/runner

Edit line 347 to:

checkpoint = torch.load(filename, map_location=map_location, weights_only=False)

Modify MuseTalk Files:

Update the following files to include weights_only=False in all torch.load() calls:

• musetalk/models/unet.py (e.g., Line 44):

weights = torch.load(model_path, weights_only=False) if torch.cuda.is_available() else torch.load(model_path, map_location=self.device, weights_only=False)

• musetalk/utils/face_parsing/__init__.py
• musetalk/models/vae.py
• musetalk/utils/face_detection/detection/sfd/sfd_detector.py
• musetalk/utils/face_parsing/resnet.py
• musetalk/utils/training_utils.py
• scripts/realtime_inference.py
• musetalk/whisper/whisper/__init__.py

This modification ensures proper model loading on the RTX 5090.

Step 11: Test the Installation

Run the provided inference script to verify the setup:

chmod +x inference.sh
sh inference.sh v1.5 normal

This command runs MuseTalk with version 1.5 in normal mode. If successful, it confirms that MuseTalk is correctly installed and optimized for the RTX 5090.

Troubleshooting Tips

• CUDA Errors: Ensure the CUDA toolkit and NVIDIA drivers are compatible with the RTX 5090. Check with nvidia-smi.
• Dependency Conflicts: Verify that all MMLab package versions match those specified.
• Model Loading Issues: Double-check that all torch.load() modifications include weights_only=False.
• FFmpeg Issues: If FFmpeg fails, reinstall it or verify the PATH configuration.

Conclusion

By following these steps, you can successfully install and adapt MuseTalk for NVIDIA’s RTX 5090, leveraging the Blackwell architecture’s power for generating high-quality talking head videos. This setup has been tested and optimized to ensure compatibility and performance. For further details, refer to the MuseTalk GitHub or the research paper.

Happy creating with MuseTalk on the RTX 5090!