Fine-Tuning Small LLMs for eBPF: An Improvement Experiment

I write eBPF programs almost daily. Kernel tracing, XDP packet filters, TC hooks, LSM hooks. The work is technical and repetitive enough that I thought AI could help with boilerplate code.

I tried qwen3.5–4B, gemma-4–31B, and gpt-oss-20B. All three are solid at general programming tasks. All three confidently generate eBPF code that won’t compile.

Here’s a real example. I asked for a simple XDP program to drop packets based on source IP. The model generated this:

__u32 src_ip = bpf_get_packet_src_ip(ctx);
if (src_ip == blocked_ip) {
 return XDP_DROP;
}

The function bpf_get_packet_src_ip() does not exist. Never has. The correct approach involves parsing the Ethernet header, checking for IPv4, then reading from iphdr->saddr. The model invented a helper function that sounds plausible but is complete fabrication.

Other common failures: wrong map types (BPF_MAP_TYPE_HASHMAP instead of BPF_MAP_TYPE_HASH), invalid helper signatures, verifier-illegal pointer arithmetic, using deprecated APIs that were removed years ago.

Even with search tools enabled, the models don’t reliably find current eBPF kernel APIs. They pattern-match on old blog posts and outdated examples.

This matters because eBPF runs in kernel space. Code that compiles but violates verifier rules gets rejected at load time. Code with wrong helper signatures breaks in production. You can’t ship “close enough” eBPF programs.

The real problem: eBPF framework APIs are under continuous development. libbpf evolves, cilium/ebpf adds new patterns, aya introduces Rust-native APIs. The training cutoffs for these models mean they’re working with outdated snapshots of a moving target.

I also wanted to learn aya, the Rust eBPF framework. But asking models for aya code was worse than asking for C. They’d generate #[map] and #[xdp] attributes with completely fabricated syntax. Rust eBPF is even more underrepresented than C eBPF in training data.

The Hypothesis

I got access to an NVIDIA GH200 (97GB) through Supermicro Jumpstart. Heavy AI compute, free access window. I wanted to test something practical.

The idea: fine-tune a small model specifically for eBPF code generation. If the problem is stale training data, domain-specific fine-tuning should close the gap.

I already knew unsloth for efficient LoRA training and llama.cpp for fast inference. The question was whether it would actually work. Would a fine-tuned 4B model outperform a baseline 31B model on eBPF tasks?

Smaller models should benefit more from targeted fine-tuning. They have less eBPF in their pretraining, so fresh domain data should move the needle. Larger models have already seen more diverse code, so the boost should be smaller.

I decided to test two models:

• Qwen3.5–4B (small, conversational base)
• Gemma-4–31B (mid-range, strong baseline)

Same dataset, same training config, same eval. If the 4B model gets a measurable boost, it proves the methodology. If a conversational base benefits this much, a coder-specific base (Qwen2.5-Coder, DeepSeek-Coder) should do even better.

Building the Dataset

I needed current eBPF examples across all major frameworks. I scraped 19 open-source repositories:

C/libbpf: libbpf, libbpf-bootstrap, bcc, bpf-perf-tools-book, learning-ebpf Go/cilium: cilium/ebpf, tetragon, beyla, bpfman, retis Rust/aya: aya, libbpf-rs, redbpf, aya-rs/book Multi-framework: eunomia-bpf, bpf-developer-tutorial, tracee, deepflow

I pulled .rs, .c, .h, .go, .py, and .md files from each repo. Then chunked them by logical boundaries: function-level for code, section-level for documentation.

The signal filtering problem: Version 1 of the dataset had ~11k samples but included a lot of noise. Build scripts, CI configs, generic Go utility functions that had nothing to do with eBPF. They got included because they lived in eBPF repos but weren’t actually eBPF code.

For version 2, I added stricter filters. If a C file didn’t include <bpf/bpf.h> or <linux/bpf.h>, it got dropped. If a Rust file didn't import anything from aya or libbpf_rs, it got dropped. Go files needed cilium/ebpf imports. This cut ~5k chunks but the remaining 6,412 were clean eBPF code.

Generating Q&A pairs: I used a local Qwen3.5-Coder:80B model to convert code chunks into instruction-tuning format. Temperature 0.3 for code analysis (deterministic, focused), temperature 0.6 for conceptual explanations (slightly more varied).

Each sample got categorized by style:

• code_analysis: "Explain what this function does"
• api_explain: "How does this eBPF helper work?"
• concept: "What is CO-RE and why does it matter?"
• usage_example: "Show me how to attach a kprobe"

Output format was ShareGPT (system/human/gpt conversation turns). This is what most fine-tuning tools expect.

Deduplication: Ran MinHash LSH with Jaccard similarity threshold 0.85 to catch near-duplicates, then exact MD5 hashing for perfect matches. Some repos had copied examples from each other. No point training on the same pattern twice.

The entire pipeline took 22 hours to run. I ran Qwen3.5:80B locally because I didn’t want to pay Claude or OpenAI API costs for 6,412 generation calls. I also had limited VRAM, so I couldn’t run the larger open-source models that would have been better at this task. The 80B was the biggest I could fit. Scraping was fast, but generating Q&A pairs was the bottleneck.

Final dataset stats:

• 6,412 instruction pairs
• 19 source repositories
• 4 style categories
• Coverage: C (libbpf + CO-RE), Go (cilium/ebpf), Rust (aya)

The dataset is live at huggingface.co/datasets/Nikhil69/ebpf-instruct

Version 2 is smaller than version 1 but cleaner. Quality over quantity.

Training Setup

The GH200 is ARM64 architecture, So I cloned the repo and built from source for more comptability. Took about 20 minutes to compile with all the CUDA extensions.

I trained two models using the same config:

• Qwen3.5–4B (unsloth/Qwen3.5–4B base)
• Gemma-4–31B (unsloth/gemma-4–31b base)

LoRA configuration:

• Rank: 32
• Alpha: 32
• Dropout: 0 (no dropout for small dataset)
• Target modules: all linear layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)

Training hyperparameters:

• 3 epochs
• Context length: 2048 tokens
• Batch size: 4
• Gradient accumulation: 4 (effective batch size 16)
• Learning rate: 2e-4
• Scheduler: cosine with warmup
• Optimizer: AdamW 8-bit
• Warmup steps: 100

Hardware: NVIDIA GH200 with 97GB unified memory via Unsloth Studio. Qwen3.5–4B took 3.5 hours to train. Gemma-4–31B took 8 hours. Total training time: 11.5 hours.

Training loss for Qwen3.5–4B started at 0.98 and converged to 0.5–0.6 by the end of epoch 3. Gemma-4–31B started lower (0.65) and ended around 0.45. The larger model had better initial loss because it already knew more about programming in general.

Same dataset (ebpf-instruct-v2, 6,412 samples), same config, same hardware. The only variable was model size. Controlled experiment.

Evaluation Methodology

I needed a real measure of whether the models actually learned eBPF. Code that looks plausible but doesn’t compile is worthless for kernel programming.

I created 40 eBPF coding problems across 5 categories:

1. libbpf_c (10 problems): C programs using libbpf and CO-RE
2. cilium_go (10 problems): Go programs using cilium/ebpf library
3. aya_kernel (10 problems): Rust kernel-side eBPF with aya
4. aya_user (5 problems): Rust user-space programs loading eBPF
5. conceptual (5 problems): Explain eBPF concepts (no compilation)

Each coding problem has a compilation template. Cargo projects for Rust with proper dependencies in Cargo.toml. Go modules with cilium/ebpf imports. C Makefiles with clang, libbpf headers, and vmlinux.h generation.

The eval metric: pass@1

Does the generated code compile? Yes or no. No partial credit. The eBPF verifier doesn’t give partial credit either.

I set up an OpenAI-compatible endpoint so I could swap models easily. Same prompt format, same temperature (0.2 for deterministic code generation), same max tokens.

Two-stage evaluation:

1. Raw output: Take what the model generates, drop it into the template, try to compile
2. Post-processed output: If compilation fails, apply IDE-level fixes and retry

Post-processing handles things a developer’s editor would catch automatically. For C code: add missing #define macros for byte-order helpers (bpf_htons, bpf_ntohs) and network constants (ETH_P_IP, TC_ACT_OK). For Go code: remove unused imports and add missing cilium/ebpf sub-packages like rlimit or ringbuf.

I’m being explicit about this because it matters. The post-processed results show what happens when the model has correct API knowledge but misses syntactic details an IDE would fix. The raw results show pure model capability.

Why compilation-based eval matters:

eBPF code can look perfect and still fail. Wrong helper signature? Verifier rejects it. Missing bounds check on array access? Verifier rejects it. Invalid pointer arithmetic? Verifier rejects it.

Compilation is the first gate. If it doesn’t compile, it definitely won’t load. If it compiles, it might load. That’s the bar.

Results

Qwen3.5–4B: The Win

Raw fine-tuning gave a 10 percentage point improvement (12.5% to 22.5%). With post-processing, that jumped to 20 points over baseline (32.5%).

The big movement was in libbpf C code. Baseline couldn’t compile a single C program (0%). Fine-tuned model hit 30%. With post-processing (adding missing defines and constants), it reached 70%.

On conceptual questions, the model went from 83% to 100%. It learned to explain CO-RE, verifier constraints, and map types correctly.

The model learned correct API patterns. It stopped inventing helpers. It used the right map types. It knew which contexts get passed to which program types. Post-processing just handled syntactic gaps like missing #define bpf_htons(x) __builtin_bswap16(x) that an IDE would add automatically.

Gemma-4–31B: The Surprise

Fine-tuning gave zero improvement. 35% baseline, 35% after training.

The baseline was already strong. It could compile 60% of libbpf C problems and 20% of cilium Go problems out of the box. The larger parameter count meant it had already seen enough diverse code during pretraining to handle eBPF reasonably well.

The synthetic dataset I built was optimized for the failure modes of the 4B model. Those patterns didn’t help a 31B model that already knew the basics.

Rust eBPF (aya): 0% Everywhere

Every model, baseline or fine-tuned, scored 0% on aya kernel and user programs.

The problem is aya’s proc-macro attributes. #[map], #[xdp], #[kprobe], #[tracepoint]. These are Rust-specific compile-time macros that don't exist in C or Go eBPF frameworks.

None of the models had seen enough aya code during pretraining. Even after fine-tuning on my dataset, they couldn’t generate valid aya syntax. The dataset had aya examples, but not enough compilable templates showing the full kernel + user split with correct macro usage.

Rust eBPF is severely underrepresented in training data compared to C. This needs a v3 dataset with more complete aya projects.

The Takeaway

Small models benefit more from domain-specific fine-tuning than large ones. Qwen3.5–4B got a 20-point boost. Gemma-4–31B got nothing because it already knew enough.

If a conversational base model can improve this much, a coder-specific base model (Qwen2.5-Coder, DeepSeek-Coder) should do even better. Those models start with stronger code understanding, so the same eBPF dataset should push them further.

What I Learned

This was my first time training a model from scratch. I’m a systems engineer, not an ML researcher. I know eBPF verifiers and kernel hooks, not loss curves and gradient descent. Here’s what I figured out.

1. Small models are more data-hungry

Qwen3.5–4B needed the targeted eBPF dataset to improve. It didn’t have enough eBPF in its pretraining, so 6,412 domain-specific examples made a real difference.

Gemma-4–31B already had the knowledge. Larger parameter count means it had seen more diverse code during pretraining. The synthetic dataset didn’t teach it anything new.

This means if you’re working on a niche domain, start with the smallest model that can handle the task. Fine-tuning will help it more.

2. Compilation-based evals are critical

Generic benchmarks wouldn’t catch eBPF-specific failures. A model can score high on HumanEval and still generate code that fails the BPF verifier.

Testing whether code compiles forces the model to get details right. Helper signatures, map types, verifier constraints. “Looks plausible” doesn’t cut it for kernel code.

If you’re evaluating code generation for a specific domain, build domain-specific tests. Don’t trust vibes.

3. Dataset quality beats quantity

Version 1 had 11k samples with noise (build scripts, CI configs, generic utility code). Version 2 had 6.4k samples of pure eBPF code.

The smaller, cleaner dataset worked better. The model didn’t waste capacity learning patterns from non-eBPF files that happened to live in eBPF repos.

Stricter signal filtering is worth the time.

4. Rust eBPF is still a frontier

Even after fine-tuning, every model scored 0% on aya programs. The #[map] and #[xdp] proc-macro syntax is unfamiliar to all of them.

This tells me two things: Rust eBPF is genuinely underrepresented in training data, and I need more compilable aya examples in the next dataset version. Not just code snippets, but full kernel + user programs that actually build.

I still want to learn aya. The fine-tuned model didn’t help with that yet. But now I know what’s missing.

5. The methodology works

A conversational base model (Qwen3.5–4B) got a 20-point improvement on eBPF tasks. That proves the approach. If I had started with a coder-specific base model like Qwen2.5-Coder or DeepSeek-Coder, the results would be better.

Those models already understand code structure, compilation constraints, and API patterns. Adding eBPF-specific knowledge on top of that foundation should push pass@1 higher than 32.5%.

This experiment wasn’t about building a production tool. It was about proving that domain-specific fine-tuning works for niche coding tasks. It does.

Now What’s Next

This was version 1 of the experiment. It proved the methodology works. Now I want to push it further.

Better data, more of it

The current dataset has 6,412 samples from 19 repos. That’s enough to move the needle on a 4B model, but not enough to saturate a larger one.

I’m building v3 with stricter quality filters and more compilable examples. Target: 15k-20k samples with full end-to-end programs, not just code snippets. More aya projects showing complete kernel + user implementations. More cilium/ebpf examples with BPF tail calls, ring buffers, and perf events. More libbpf CO-RE patterns for real observability tools.

The goal is dataset quality high enough that even a 70B model learns something new.

Coder-specific base models

Qwen3.5–4B is a conversational model. It improved 20 points because it started from a weak base. A coder-specific model should start stronger and climb higher.

Next experiment: train Qwen2.5-Coder-7B and DeepSeek-Coder-6.7B on the same eBPF dataset. These models already understand code structure, compilation constraints, and API patterns. Adding eBPF-specific knowledge on top should push pass@1 past 50%.

If a conversational 4B hit 32.5%, a coding-focused 7B should break 50%. That’s the hypothesis.

Expand the eval to 100+ problems

40 problems is enough to see a trend. It’s not enough to measure fine-grained improvements. I want per-category breakdowns: XDP vs TC vs kprobes, CO-RE vs non-CO-RE, aya kernel vs user.

Building a 100-problem eval suite with compilation tests, verifier compliance checks, and runtime correctness validation. That’s the real benchmark.

Make aya work

Every model scored 0% on aya. That’s unacceptable if I actually want to learn Rust eBPF.

v3 dataset will have 50+ complete aya projects scraped from GitHub, not just the official aya repo. Community projects, production tools, real-world examples. If the models see enough #[map] and #[xdp] syntax patterns with correct macro usage, they'll learn it.

This is the part I care about most. I didn’t start this project to improve C eBPF generation. I started it because I wanted AI help learning aya. The fine-tuned model still can’t do that. v3 will fix it.

Open invitation

The dataset, model, and eval code are all public:

• GitHub: https://github.com/Nikhil690/ebpf-llm-training-experiment
• Model: https://huggingface.co/Nikhil69/Qwen3.5-4B-ebpf-it
• Dataset: https://huggingface.co/datasets/Nikhil69/ebpf-instruct

The Gemma-4–31B fine-tuned model isn’t uploaded. The GH200 access window had a time limit and storage constraints. I prioritized getting the 4B model out since it showed actual improvement.

If you’re working on eBPF tooling or kernel observability and want AI that actually knows current APIs, try it. If you have better ideas for dataset construction or eval design, open an issue. This is still early.

The answer to “can AI help with kernel-space development?” is yes, but only after you teach it your domain. General-purpose models won’t cut it. Domain-specific fine-tuning works.

Now I’m going to make it work better.