Mastering Nested Virtualization: Custom Linux Kernel with KVM Inside VirtualBox

Cloud computing enthusiasts and Linux lovers, get ready! In this guide, we’ll walk through building a custom Linux kernel inside a nested virtualization setup using VirtualBox and KVM/QEMU on Windows. The process may seem daunting, but with the right steps, anyone can do it — safely and efficiently. 🌟

📌 Why Nested Virtualization?

Before we dive into commands, understand the architecture:

Layers of Our Setup

LayerWhat It IsLives OnWindows PCYour physical computerHardwareVirtualBoxSoftware that pretends to be a computerWindowsUbuntu 22.04 (HOST VM)Linux environment for KVMInside VirtualBoxQEMU + KVMHypervisor for inner VMInside UbuntuInner Ubuntu VM (GUEST)Tiny VM for testing the kernelInside KVM/QEMUCustom Linux KernelCompiled and optimized kernelBoots inside GUEST VM

Nested virtualization allows running a VM inside a VM, letting us safely test kernel optimizations without touching your real Windows installation. 💻

⚡ Requirements

Minimum vs Recommended

• RAM: 8 GB | 16 GB
• CPU: Intel/AMD with VT-x | Modern quad-core
• Storage: 40 GB | 60 GB
• OS: Windows 10/11 | Windows 10/11
• Cost: 100% Free! 🎉

1️⃣ Enable VT-x / AMD-V in BIOS

This is critical — without hardware virtualization, KVM won’t run.

Steps:

• Open Task Manager → Performance → CPU → Check Virtualization
• If disabled, restart and enter BIOS (common keys: Dell F2, HP F10/Esc, Lenovo F1/F2, Asus F2/Del)
• Enable VT-x / AMD-V and save changes

Tip: Google “[your laptop model] enable VT-x BIOS” for exact instructions. 🔑

2️⃣ Install VirtualBox & Extension Pack

Steps:

1. Download VirtualBox and Extension Pack from VirtualBox Downloads 🌐
2. Install VirtualBox (leave default options)
3. Add the Extension Pack for USB support, NVMe performance, and shared clipboard

3️⃣ Download Ubuntu 22.04 ISO

• Get Desktop LTS edition (stable and GUI included)
• Save in Desktop or Downloads folder (~4.7 GB)

4️⃣ Create Ubuntu HOST VM in VirtualBox

Steps:

• Name: Ubuntu-KVM-Project
• Type: Linux → Ubuntu 22.04 LTS (64-bit)
• RAM: 4–8 GB depending on total system memory
• CPU: 2–4 cores
• Disk: 40–60 GB, dynamically allocated
• Enable Nested Virtualization → Settings → System → Processor → tick Enable Nested VT-x/AMD-V
• Configure Display: Video Memory 128 MB, Graphics Controller VMSVGA, 3D Acceleration ✔️

5️⃣ Install Ubuntu Inside VirtualBox

Walkthrough:

• Start VM → Boot from ISO
• Select language → Install Ubuntu
• Installation Type → Erase Virtual Disk Only
• Create user: student / Password: student123
• Reboot and skip welcome wizard
• Optional: Install Guest Additions for full-screen & clipboard

6️⃣ Setup Ubuntu HOST for Project

• Update system: sudo apt update && sudo apt upgrade -y
• Verify nested virtualization: grep -c 'vmx\|svm' /proc/cpuinfo
• Install KVM, QEMU, tools, and kernel build dependencies:

sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients \
bridge-utils virt-manager cpu-checker build-essential gcc g++ make \
flex bison libssl-dev libelf-dev libncurses-dev bc dwarves git wget curl
sudo usermod -aG kvm,libvirt $USER
newgrp libvirt
sudo systemctl status libvirtd

7️⃣ Create Inner VM (GUEST) with KVM

• Create folder for VM: mkdir -p ~/vms
• Download Ubuntu Server ISO (~2 GB)
• Create disk: qemu-img create -f qcow2 ~/vms/inner-vm.qcow2 15G
• Launch text-based installation via QEMU:

sudo qemu-system-x86_64 -m 2048 -smp 2 -enable-kvm -cpu host \
-drive file=~/vms/inner-vm.qcow2,format=qcow2,if=virtio \
-cdrom ~/vms/ubuntu-22.04.4-live-server-amd64.iso -boot d \
-net nic,model=virtio -net user,hostfwd=tcp::2222-:22 -nographic -serial mon:stdio

• Configure inner VM: username ubuntu, password ubuntu123, enable OpenSSH

8️⃣ Benchmark Stock Kernel

Metrics to capture:

• Boot Time: systemd-analyze
• CPU: sysbench cpu --cpu-max-prime=20000 --threads=2 run
• Memory: sysbench memory --memory-block-size=1K --memory-total-size=4G run
• Disk I/O: fio --name=disktest --filename=/tmp/fiofile --rw=readwrite --bs=1M --size=256M --numjobs=1 --runtime=20 --group_reporting

Take screenshots — this is your baseline. 📸

9️⃣ Download & Prepare Linux Kernel Source

• Kernel version: 6.6 LTS
• Download: wget [https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.6.30.tar.xz](https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.6.30.tar.xz)
• Extract: tar -xvf linux-6.6.30.tar.xz
• Copy current config: cp /boot/config-$(uname -r) .config
• Auto-fill new options: make olddefconfig

🔧 10️⃣ Configure & Optimize Kernel

Key optimizations:

• KVM Guest / Paravirtualization: CONFIG_KVM_GUEST, CONFIG_PARAVIRT, CONFIG_PARAVIRT_CLOCK
• VirtIO Drivers: CONFIG_VIRTIO, CONFIG_VIRTIO_PCI, CONFIG_VIRTIO_BLK, CONFIG_VIRTIO_NET, CONFIG_VIRTIO_BALLOON, CONFIG_VIRTIO_CONSOLE
• CPU/Timer: CONFIG_NO_HZ_FULL, CONFIG_HZ_1000
• Remove unnecessary drivers: WiFi, Bluetooth, Sound, USB, FireWire, PCCARD
• Disable debugging: CONFIG_DEBUG_KERNEL, CONFIG_KGDB, CONFIG_FTRACE

11️⃣ Compile Kernel

• Command: make -j$(nproc) 2>&1 | tee ~/kernel-build/build.log
• Success: arch/x86/boot/bzImage exists (~8-15 MB)
• Optional: Create .deb package: make -j$(nproc) bindeb-pkg

Time: 40–120 minutes depending on system ⚡

12️⃣ Install Custom Kernel in Inner VM

• Start inner VM → Copy .deb files via SCP → Install:

sudo dpkg -i ~/linux-image-*.deb
sudo dpkg -i ~/linux-headers-*.deb
sudo update-grub
sudo reboot

• Verify: uname -r → should show 6.6.30
• Check paravirtualization & VirtIO: dmesg | grep -i paravirt / dmesg | grep virtio ✅

13️⃣ Benchmark Custom Kernel

• Repeat same tests as stock kernel
• Expect improvements:

MetricStockCustomImprovementBoot Time20–40s8–18sFaster by removing unused driversCPUBaseline+10–20%Fewer VM exitsMemoryBaselineSimilar or betterParavirt memory opsDisk Read/WriteBaseline+15–25%VirtIO bypasses emulation

Save results, screenshots, and include in submission 📊

14️⃣ Troubleshooting

Common issues & solutions:

• VT-x greyed out: Enable in BIOS & disable Hyper-V
• kvm-ok fails: Enable Nested VT-x in VirtualBox
• Inner VM SSH fails: Wait longer, ensure QEMU running
• Compilation errors: Check disk space, install missing libraries

15️⃣ Viva Exam Prep

Key Questions:

• What is KVM?
• What is paravirtualization?
• Why VirtIO drivers are faster than emulated devices?
• What does CONFIG_KVM_GUEST do?
• Benchmark improvements & reasons

Practice running commands smoothly, show screenshots, and explain optimizations confidently 💡

✅ Final Submission Checklist

• VirtualBox + Extension Pack installed
• HOST VM with Ubuntu 22.04 created
• Nested virtualization verified (kvm-ok)
• Inner VM created & custom kernel installed
• Stock & custom kernel benchmarks saved
• dmesg proves paravirt & VirtIO active
• Screenshots labelled & ready
• Able to answer viva questions

💡 Pro Tip: Use a local network or tools like ngrok if your instructor requires remote access.