Intel’s Hidden GPU Power: 2x Faster PyTorch Training at No Extra Cost

Introduction

Neural Networks and PyTorch are the de facto standard in Data Science applications. One can run into an interesting problem. GPUs are considerably faster than CPUs for such tasks, but not everyone can afford an Nvidia GPU. What if, for some reason, the only thing you have is an Intel CPU and integrated GPU like Iris? Could we do something, and is it reasonable?

Actually, Intel has the IPEX library that makes it possible to run things like PyTorch and even Ollama on integrated GPUs. In this post, I will describe the prerequisites and installation process. Then, I will show you two PyTorch tests on two Intel platforms and compare them with Nvidia T4.

Test setups

I would not say that IPEX is a popular library. Nvidia CUDA is the king after all. That is why I will describe my installation experience and setups.

Setup 1:

• Intel 11th Gen i7 1185G7 4 cores 8 threads
• Intel Iris Xe iGPU with 96 cores 1.35 GHz
• 32 GB RAM SODIMM 3200 MT/s
• Windows 11 + WSL2 Ubuntu 22.04
• Python 3.11

Setup 2:

• Intel 13th Gen i7 1360p 12 cores 16 threads
• Intel Iris Xe iGPU with 96 cores 1.5 GHz
• 16 GB LPDDR5–5200 6000 MT/s
• Windows 11 + WSL2 Ubuntu 22.04
• Python 3.11

Setup 3 COLAB:

• Intel Xeon CPU (4 cores?)
• NVIDIA T4 GPU with 16GB
• 12 GB RAM
• Python 3.11

Installation and prerequisites

Unfortunately, installation is not so straightforward because even if you install everything correctly XPU device can be unrecognised.

General prerequisites:

• Update your Intel iGPU drivers
• Install WSL2 Ubuntu 22.04
• Install python 3.11 (I would recommend pyenv)
• Install oneAPI
• sudo apt install intel-oneapi-base-toolkit

Installation:

Activate your virtual environment e.g.

pyenv virtualenv 3.11 pytoch-intel
pyenv activate pytoch-intel

Install IPEX:

python -m pip install torch==2.5.1+cxx11.abi torchvision==0.20.1+cxx11.abi torchaudio==2.5.1+cxx11.abi intel-extension-for-pytorch==2.5.10+xpu oneccl_bind_pt==2.5.0+xpu - extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/

Run the code to count your XPU devices:

python -c "import torch; import intel_extension_for_pytorch as ipex; print(torch.__version__); print(ipex.__version__); [print(f'[{i}]: {torch.xpu.get_device_properties(i)}') for i in range(torch.xpu.device_count())];"

If you don’t have “XPU devices count is 0” then you are lucky. If you have then you should look deeply into the docs and install libraries. Unfortunately, no clear suggestions here. I will put some useful links in the appendix.

Testing improvements:

For testing, I asked Claude to generate two NNs for MNIST and CIFAR-10. You can run them on your machine to test and compare improvements.

MNIST

import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
import torchvision.transforms as transforms
import intel_extension_for_pytorch as ipex
from torch.utils.data import DataLoader
import time
from typing import Literal
from datetime import datetime

class SimpleCNN(nn.Module):
    def __init__(self):
        super(SimpleCNN, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(32, 64, 3)
        self.fc1 = nn.Linear(64 * 5 * 5, 128)
        self.fc2 = nn.Linear(128, 10)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.pool(self.relu(self.conv1(x)))
        x = self.pool(self.relu(self.conv2(x)))
        x = x.view(-1, 64 * 5 * 5)
        x = self.relu(self.fc1(x))
        x = self.fc2(x)
        return x

def train_model(
    device_type: Literal["cpu", "xpu"], num_epochs: int = 5, batch_size: int = 64
):
    print(f"\nTraining on {device_type.upper()}...")
    device = torch.device(device_type)

    # Load and preprocess MNIST dataset
    transform = transforms.Compose(
        [transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))]
    )

    trainset = torchvision.datasets.MNIST(
        root="./data", train=True, download=True, transform=transform
    )
    trainloader = DataLoader(
        trainset, batch_size=batch_size, shuffle=True, num_workers=2
    )

    # Initialize the model, loss function, and optimizer
    model = SimpleCNN().to(device)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.001)

    # Enable IPEX optimization for XPU
    if device_type == "xpu":
        model, optimizer = ipex.optimize(model, optimizer=optimizer)

    # Training loop with timing
    total_time = 0
    batch_times = []

    for epoch in range(num_epochs):
        epoch_start = time.time()
        running_loss = 0.0

        for i, (inputs, labels) in enumerate(trainloader):
            batch_start = time.time()

            inputs, labels = inputs.to(device), labels.to(device)
            optimizer.zero_grad()

            outputs = model(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            batch_time = time.time() - batch_start
            batch_times.append(batch_time)

            running_loss += loss.item()
            if i % 100 == 99:
                print(
                    f"[Epoch {epoch + 1}, Batch {i + 1}] "
                    f"Loss: {running_loss / 100:.3f}, "
                    f"Batch Time: {batch_time:.4f}s"
                )
                running_loss = 0.0

        epoch_time = time.time() - epoch_start
        total_time += epoch_time
        print(f"Epoch {epoch + 1} completed in {epoch_time:.2f} seconds")

    # Calculate statistics
    avg_batch_time = sum(batch_times) / len(batch_times)
    avg_epoch_time = total_time / num_epochs

    return {
        "device": device_type,
        "total_time": total_time,
        "avg_epoch_time": avg_epoch_time,
        "avg_batch_time": avg_batch_time,
        "model": model,
    }

def compare_devices(num_epochs: int = 5, batch_size: int = 64):
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    results = []

    # Train on XPU if available
    if torch.xpu.is_available():
        xpu_results = train_model("xpu", num_epochs, batch_size)
        results.append(xpu_results)
    else:
        print("\nXPU not available for comparison")

    # Train on CPU first
    cpu_results = train_model("cpu", num_epochs, batch_size)
    results.append(cpu_results)

    # Print comparison
    print("\n" + "=" * 50)
    print("Performance Comparison:")
    print("=" * 50)

    for result in results:
        device = result["device"].upper()
        print(f"\n{device} Results:")
        print(f"Total Training Time: {result['total_time']:.2f} seconds")
        print(f"Average Epoch Time: {result['avg_epoch_time']:.2f} seconds")
        print(f"Average Batch Time: {result['avg_batch_time']:.4f} seconds")

        # Save model
        model_path = f"mnist_model_{device.lower()}_{timestamp}.pth"
        torch.save(result["model"].state_dict(), model_path)
        print(f"Model saved to {model_path}")

if __name__ == "__main__":
    NUM_EPOCHS = 5
    BATCH_SIZE = 64

    print("PyTorch version:", torch.__version__)
    print("Intel Extension for PyTorch version:", ipex.__version__)
    print(f"XPU available: {torch.xpu.is_available()}")

    compare_devices(NUM_EPOCHS, BATCH_SIZE)

CIFAR-10

import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
import torchvision.transforms as transforms
import intel_extension_for_pytorch as ipex
from torch.utils.data import DataLoader
import time
from typing import Literal
from datetime import datetime
import numpy as np

class CIFAR10CNN(nn.Module):
    def __init__(self):
        super(CIFAR10CNN, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(32, 64, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )

        self.classifier = nn.Sequential(
            nn.Dropout(0.5),
            nn.Linear(128 * 8 * 8, 512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(512, 10),
        )

    def forward(self, x):
        x = self.features(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x

def calculate_accuracy(outputs, labels):
    """Calculate accuracy in a way that's compatible with both CPU and XPU"""
    _, predicted = torch.max(outputs.data, 1)
    predicted = predicted.to("cpu")
    labels = labels.to("cpu")
    total = labels.size(0)
    correct = (predicted == labels).sum().item()
    return correct, total

def train_model(
    device_type: Literal["cpu", "xpu"], num_epochs: int = 5, batch_size: int = 128
):
    print(f"\nTraining on {device_type.upper()}...")
    device = torch.device(device_type)

    # Data augmentation and normalization for training
    transform_train = transforms.Compose(
        [
            transforms.RandomCrop(32, padding=4),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
        ]
    )

    transform_test = transforms.Compose(
        [
            transforms.ToTensor(),
            transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
        ]
    )

    # Load CIFAR-10
    trainset = torchvision.datasets.CIFAR10(
        root="./data", train=True, download=True, transform=transform_train
    )
    testset = torchvision.datasets.CIFAR10(
        root="./data", train=False, download=True, transform=transform_test
    )

    trainloader = DataLoader(
        trainset, batch_size=batch_size, shuffle=True, num_workers=2, pin_memory=True
    )
    testloader = DataLoader(
        testset, batch_size=batch_size, shuffle=False, num_workers=2, pin_memory=True
    )

    # Class names for CIFAR-10
    classes = (
        "plane",
        "car",
        "bird",
        "cat",
        "deer",
        "dog",
        "frog",
        "horse",
        "ship",
        "truck",
    )

    # Initialize model, criterion, and optimizer
    model = CIFAR10CNN().to(device)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=5e-4)
    scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_epochs)

    # Enable IPEX optimization for XPU
    if device_type == "xpu":
        model, optimizer = ipex.optimize(model, optimizer=optimizer)

    # Training loop with timing
    total_time = 0
    batch_times = []
    best_acc = 0.0

    for epoch in range(num_epochs):
        epoch_start = time.time()
        model.train()
        running_loss = 0.0
        correct = 0
        total = 0

        for i, (inputs, labels) in enumerate(trainloader):
            batch_start = time.time()

            inputs, labels = inputs.to(device), labels.to(device)
            optimizer.zero_grad()

            outputs = model(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            # Calculate accuracy
            batch_correct, batch_total = calculate_accuracy(outputs, labels)
            correct += batch_correct
            total += batch_total

            batch_time = time.time() - batch_start
            batch_times.append(batch_time)

            running_loss += loss.item()
            if i % 50 == 49:
                print(
                    f"[Epoch {epoch + 1}, Batch {i + 1}] "
                    f"Loss: {running_loss / 50:.3f}, "
                    f"Acc: {100. * correct / total:.2f}%, "
                    f"Batch Time: {batch_time:.4f}s"
                )
                running_loss = 0.0
                correct = 0
                total = 0

        # Evaluate on test set
        model.eval()
        test_loss = 0
        correct = 0
        total = 0
        class_correct = np.zeros(10)
        class_total = np.zeros(10)

        with torch.no_grad():
            for inputs, labels in testloader:
                inputs, labels = inputs.to(device), labels.to(device)
                outputs = model(inputs)
                loss = criterion(outputs, labels)
                test_loss += loss.item()

                # Move tensors to CPU for accuracy calculation
                batch_correct, batch_total = calculate_accuracy(outputs, labels)
                correct += batch_correct
                total += batch_total

                # Per-class accuracy (on CPU)
                _, predicted = torch.max(outputs, 1)
                predicted = predicted.to("cpu")
                labels = labels.to("cpu")

                # Update per-class accuracy
                for label in range(10):
                    mask = labels == label
                    class_correct[label] += (predicted[mask] == label).sum().item()
                    class_total[label] += mask.sum().item()

        # Print per-class accuracy
        print("\nPer-class accuracy:")
        for i in range(10):
            if class_total[i] > 0:
                print(f"{classes[i]}: {100 * class_correct[i] / class_total[i]:.1f}%")

        acc = 100.0 * correct / total
        if acc > best_acc:
            best_acc = acc

        epoch_time = time.time() - epoch_start
        total_time += epoch_time
        print(f"\nEpoch {epoch + 1} Summary:")
        print(f"Average Test Accuracy: {acc:.2f}%")
        print(f"Best Accuracy: {best_acc:.2f}%")
        print(f"Epoch completed in {epoch_time:.2f} seconds")

        scheduler.step()

    # Calculate statistics
    avg_batch_time = sum(batch_times) / len(batch_times)
    avg_epoch_time = total_time / num_epochs

    return {
        "device": device_type,
        "total_time": total_time,
        "avg_epoch_time": avg_epoch_time,
        "avg_batch_time": avg_batch_time,
        "best_accuracy": best_acc,
        "model": model,
    }

def compare_devices(num_epochs: int = 5, batch_size: int = 128):
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    results = []

    # Train on XPU if available
    if torch.xpu.is_available():
        xpu_results = train_model("xpu", num_epochs, batch_size)
        results.append(xpu_results)
    else:
        print("\nXPU not available for comparison")

    # Train on CPU first
    cpu_results = train_model("cpu", num_epochs, batch_size)
    results.append(cpu_results)

    # Print comparison
    print("\n" + "=" * 50)
    print("Performance Comparison:")
    print("=" * 50)

    for result in results:
        device = result["device"].upper()
        print(f"\n{device} Results:")
        print(f"Total Training Time: {result['total_time']:.2f} seconds")
        print(f"Average Epoch Time: {result['avg_epoch_time']:.2f} seconds")
        print(f"Average Batch Time: {result['avg_batch_time']:.4f} seconds")
        print(f"Best Test Accuracy: {result['best_accuracy']:.2f}%")

        # Save model
        model_path = f"cifar10_model_{device.lower()}_{timestamp}.pth"
        torch.save(result["model"].state_dict(), model_path)
        print(f"Model saved to {model_path}")

if __name__ == "__main__":
    NUM_EPOCHS = 5
    BATCH_SIZE = 64

    print("PyTorch version:", torch.__version__)
    print("Intel Extension for PyTorch version:", ipex.__version__)
    print(f"XPU available: {torch.xpu.is_available()}")

    compare_devices(NUM_EPOCHS, BATCH_SIZE)

Here are my results:

Surprise-surprise switching workloads from CPU to GPU delivers up to 2x performance improvements. Also as you can see there is significant progress in calculations between two generations. I believe it is due to the higher RAM speed because the iGPU itself is almost identical. I also assume that new Intel chips will provide even larger improvements. Moreover, having RAM as a shared memory means that it is possible to avoid Nvidia memory bottleneck.

MNIST

In the case of the 11th generation, the CPU workload takes 30% longer. As of the 13th, there the difference is even clearer. It is almost 2x. So it is definitely worth some work on the installation and small code changes.

CIFAR-10

In the harder task, the difference becomes noticeable. Even in the 11th generation, we see 2x faster computations. And the 13th generation is even better.

The final showdown

I have promised to include a T4 Nvidia GPU that has a Free Tier.

MNIST

In this small task, XPU has won even in comparison with T4.

CIFAR-10

CIFAR-10 shows the advantages of having a dedicated GPU. Nevertheless, we have made great improvements on our machines. We should not forget that it is initially small and much less performant integrated GPU.

Conclusions

First of all, as you can see there is life beyond NVIDIA GPUs. Of course, iGPU themselves are much less performant, but it is definitely worth to try and install IPEX.

You can say that it is easier to buy even an old Nvidia card but keep in mind different circumstances:

• A person has a laptop. Not a PC.
• It is a low-on-budget student.
• Nvidia’s memory bottleneck. Even high-end cards have a shamefully low amount of RAM. 32 GB laptop will have on par or more.

P.S.1: If you have a newer chip let me know your results in the comments.

P.S.2: Are you interested in another post about running Ollama on IPEX?

Useful links:

[embed]Intel® Extension for PyTorch - Intel&#174 Extension for PyTorch 2.5.10+xpu documentation *This website introduces Intel® Extension for PyTorch**intel.github.io

[embed]Get the Intel® oneAPI Base Toolkit Select your operating system and distribution channel, and then download your customized installation of this toolkit.www.intel.com

[embed]Installing Client GPUs - Intel® software for general purpose GPU capabilities documentation Intel actively collaborates with various upstream projects to enable the best possible experience using the software…dgpu-docs.intel.com