NVMe Debugging in U-Boot Using QEMU and GDB Stub

Debugging U-Boot with QEMU is often much faster and more productive than debugging directly on hardware, especially during early driver development.

Debugging NVMe on real hardware typically requires a board with PCIe support and an NVMe device. Reproducing issues may involve repeated flashing, rebooting, and collecting serial logs.

QEMU combined with the GDB stub provides a much faster workflow for investigating low-level NVMe issues such as:

• PRP handling
• Queue initialization
• DMA alignment
• PCIe enumeration
• Identify commands
• Submission/completion queue handling

This setup enables full source-level debugging directly from GDB without requiring physical hardware.

Installation and Build

Install dependencies

sudo apt install gdb-multiarch qemu-system-arm gcc-aarch64-linux-gnu

Clone and configure U-Boot

git clone https://source.denx.de/u-boot/u-boot.git
cd u-boot
make qemu_arm64_defconfig
make menuconfig

Enable the following options:

Device Drivers — -> [] PCI support [] NVMe Support

Or directly edit .config:

CONFIG_NVME=y
CONFIG_PCI=y
CONFIG_DM_PCI=y
CONFIG_CMD_NVME=y
CONFIG_DEBUG_UART=y

Build with debug symbols

make CROSS_COMPILE=aarch64-linux-gnu- CFLAGS="-O0 -g3" -j$(nproc)

Using -O0 prevents compiler optimizations, making source-level debugging much easier.

Create a Virtual NVMe device in QEMU

qemu-img create -f raw nvme.img 1G

Run QEMU with NVMe and GDB stub

qemu-system-aarch64 \
 -machine virt \
 -cpu cortex-a57 \
 -m 2G \
 -nographic \
 -bios u-boot.bin \
 -drive file=nvme.img,if=none,id=drv0,format=raw \
 -device nvme,serial=deadbeef,drive=drv0,mdts=10 \
 -s -S

Important Options

• -s starts the GDB server on port 1234
• -S halts the CPU at startup until GDB connects

QEMU also allows experimenting with various NVMe controller parameters such as:

• Maximum Data Transfer Size (mdts)
• Maximum I/O queue pairs (max_ioqpairs)
• MSI-X vector table size (msix_qsize)

This makes it very useful for reproducing and debugging edge cases.

Start GDB

In another terminal:

gdb-multiarch u-boot

Inside GDB:

(gdb) set architecture aarch64
(gdb) target remote localhost:1234

Handle U-Boot Relocation

After startup, U-Boot relocates itself from SRAM/flash to DRAM.

Once relocation occurs, the symbols initially loaded by GDB no longer match the executing code addresses. To continue source-level debugging correctly, the symbol table must be reloaded using the relocated U-Boot base address.

Add a breakpoint at relocate_code:

(gdb) b relocate_code
(gdb) continue

Once the breakpoint is hit, obtain the relocated U-Boot address using:

(gdb) p/x ((gd_t *)$x18)->relocaddr

Example output:

(gdb) $1 = 0x7f68f000

Reload the symbols at the relocated address:

(gdb) add-symbol-file u-boot 0x7f68f000

After updating the symbol table, GDB will again be synchronized with the relocated U-Boot image, allowing debugging to continue normally.

Useful NVMe breakpoints

(gdb) b nvme_init
(gdb) b nvme_probe
(gdb) b nvme_identify
(gdb) b nvme_setup_prps
(gdb) b nvme_submit_sync_cmd
(gdb) b nvme_read

Since U-Boot execution is halted at the breakpoint, QEMU is also paused because the virtual CPU is stopped under GDB control.

PCIe enumeration and NVMe initialization will not proceed until execution is resumed.

(gdb) continue

After resuming, U-Boot continues with PCIe and NVMe initialization. Any breakpoints previously set in the NVMe driver will be hit as the corresponding code paths are executed.

Trigger NVMe from U-Boot shell

Once boot reaches shell:

=> nvme scan
=> nvme info
=> nvme device
=> nvme part

Example:

NVME scan: 1 NVMe Device(s) found

These commands trigger:

• PCI enumeration
• NVMe init
• identify controller
• identify namespace
• PRP setup
• DMA operations

Very useful for debugging.

Inspect PCI config space

In U-Boot:

=> pci enum
=> pci header

In GDB:

(gdb) p *ndev
(gdb) x/32x 0x40100000

Conclusion

QEMU combined with GDB remote debugging provides a powerful environment for debugging NVMe in U-Boot without requiring dedicated hardware.

It is particularly useful for investigating:

• PRP list handling
• Queue management
• DMA issues
• PCIe enumeration
• Controller initialization flows
• Command submission/completion handling

Once the workflow is set up, debugging becomes significantly faster and more iterative compared to traditional hardware-only development.