Unlocking the Linux Kernel: Real-Time Observability with eBPF

The Networking Black Box

For decades, the Linux Network Stack has been a “black box” for most developers. Packets go in, packets come out, and if something goes wrong (like a DDoS attack), we usually find out too late -after the CPU has already hit 100% and the system has frozen.

In the world of modern infrastructure, observability is no longer a luxury-it’s a necessity.

Imagine trying to debug a production issue where a specific syscall is causing latency spikes, or a network packet is being dropped deep in the kernel-only to realize your monitoring tools are blind to the real problem.

Why Traditional Tools Are No Longer Enough

For years, Traditional Linux observability tools like strace, perf, and ftrace have served us well, but they come with trade-offs -high overhead, limited programmability, or kernel recompilation requirements.

But what if you could sit inside the network driver? What if you could see every packet in nanoseconds and make decisions before the Operating System even knows they exist?

Welcome to the world of eBPF (Extended Berkeley Packet Filter) and XDP (eXpress Data Path).

Why eBPF Is Critical in 2026

• Mission-Critical Observability Without Compromise: In high-stakes environments (finance, healthcare, telecom, e-commerce), you need real-time, deep visibility without sacrificing performance.
• Performance at Hyperscale: AWS EKS made Cilium (eBPF-based) its default CNI. Companies like Meta, Netflix, Cloudflare, and Google run eBPF in production for packet processing, load balancing, and cost optimization.
• Runtime Security for the Zero-Trust Era: eBPF-powered tools like Falco and Tetragon detect threats at kernel speed-before malicious behavior reaches user space.

Ok, so what exactly is eBPF ?

eBPF (often just called BPF today) is a sandboxed virtual machine running inside the Linux kernel. It allows safe execution of user-supplied programs at near-native speed without kernel recompilation, modules, or reboots.

eBPF programs are event-driven, not continuously running like daemons. They attach to specific hooks in the kernel or user space. When the kernel (or an application) hits that hook point, the attached eBPF program executes.

Architecture

Key Architectural Components

1. eBPF Virtual Machine (VM)

• Register-based, 64-bit instructions, Limited stack, Instruction set: ~100 instructions (ALU ops, jumps, loads/stores, function calls to helpers)

2. The Verifier — The Safety Guardian

• Performs static analysis on all possible execution paths.
• Checks for: No infinite loops, no invalid memory access, no out-of-bounds, proper register usage, no blocking operations.

3. BPF Maps — The Communication & State Layer

• Maps are kernel data structures that persist independently of programs.
• They enable: Sharing data between kernel (eBPF programs) and user space, communication between multiple eBPF programs.

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, u32);          // PID
    __type(value, u64);        // Timestamp
} start SEC(".maps");

Common map types:

• BPF_MAP_TYPE_HASH / ARRAY: Key-value storage.
• BPF_MAP_TYPE_PERF_EVENT_ARRAY: For ring-buffer-like events (older).
• BPF_MAP_TYPE_RINGBUF: Modern, efficient zero-copy ring buffer.
• BPF_MAP_TYPE_PERCPU_HASH/ARRAY: For low-contention per-CPU data.

4. Helper Functions

• eBPF programs cannot call arbitrary kernel functions. Instead, they call a whitelist of helpers:

bpf_map_lookup_elem()
bpf_probe_read_kernel()
bpf_get_current_pid_tgid()
bpf_map_create()
etc...

5. JIT Compiler

• After verification, the bytecode is Just-In-Time compiled to native CPU instructions. This delivers near-native performance (often within a few % of hand-written kernel code).

6. BTF (BPF Type Format) & CO-RE (Compile Once — Run Everywhere)

• BTF: Compact, kernel-exported type information (like DWARF but smaller). Exposed at /sys/kernel/btf/vmlinux.
• CO-RE: Clang emits relocation records instead of hardcoded offsets. libbpf (loader) patches the bytecode at load time using the running kernel’s BTF.

This enables portable binaries across kernel versions/distros without recompilation. Modern production eBPF heavily relies on this.

7. Attach Points: Where BPF Programs Hook Into the Kernel

Dynamic instrumentation of any kernel function.

SEC("kprobe/tcp_sendmsg")
int BPF_KPROBE(trace_tcp_send, struct sock *sk, struct msghdr *msg, size_t size) {
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    // Log every TCP send with PID and size
    bpf_printk("tcp_sendmsg: pid=%d size=%d\n", pid, size);
    return 0;
}

Ok, so what exactly is XDP ?

XDP or eXpress Data Path provides a high performance, programmable network data path in the Linux kernel

The XDP packet process includes an in kernel component that processes RX packet-pages directly out of driver via a functional interface without early allocation of skbuff’s or software queues.

XDP (eXpress Data Path) — The Earliest Hook

Runs directly in the network device driver, before the kernel allocates sk_buff (socket buffer).

• Return Actions: XDP_DROP, XDP_PASS, XDP_TX, XDP_REDIRECT, XDP_ABORTED.

┌──────────────┬────────────────────────────────────────────────────┐
│ XDP_DROP     │ Packet silently discarded at driver. Zero cost.    │
│ XDP_PASS     │ Continue to normal kernel stack (sk_buff created). │
│ XDP_TX       │ Bounce packet back out same NIC. Hairpin routing.  │
│ XDP_REDIRECT │ Forward to another NIC, CPU, or AF_XDP socket.     │
│ XDP_ABORTED  │ Error path — drop + trace_xdp_exception event.     │
└──────────────┴────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  XDP Offload (NIC hardware)                                     │
│  └── BPF runs on SmartNIC ASIC/FPGA (Netronome, Mellanox)     │
│      Speed: line rate, zero CPU usage                           │
│      Limitation: restricted BPF subset, few NICs support it     │
├─────────────────────────────────────────────────────────────────┤
│  XDP Native (driver hook)                          ← preferred  │
│  └── BPF runs in NIC driver's NAPI poll loop                   │
│      Speed: ~24M pps per core                                   │
│      Support: mlx5, i40e, ixgbe, virtio_net, veth, bond        │
├─────────────────────────────────────────────────────────────────┤
│  XDP Generic (netif_receive_skb)                                │
│  └── BPF runs AFTER sk_buff allocation — fallback mode         │
│      Speed: ~5-8M pps (sk_buff already allocated, no savings)  │
│      Support: any NIC (but defeats the purpose)                 │
└─────────────────────────────────────────────────────────────────┘

Simple XDP Packet Dropper -

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

SEC("xdp")
int xdp_dropper(struct xdp_md *ctx)
{
    void *data_end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;
    struct ethhdr *eth = data;

    // Basic bounds check
    if ((void *)eth + sizeof(*eth) > data_end)
        return XDP_ABORTED;

    // Drop all TCP packets (example)
    if (eth->h_proto == __constant_htons(ETH_P_IP)) {
        struct iphdr *iph = data + sizeof(*eth);
        if ((void *)iph + sizeof(*iph) <= data_end) {
            if (iph->protocol == IPPROTO_TCP)
                return XDP_DROP;   // Drop TCP traffic
        }
    }

    return XDP_PASS;
}

char __license[] SEC("license") = "GPL";

XDP Hook — Packet Drop at NIC Level

#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <bpf/bpf_helpers.h>
#include <arpa/inet.h>

// Map of blocked IPv4 addresses
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10000);
    __type(key, __u32);         // IPv4 src address
    __type(value, __u64);       // drop counter
} blocked_ips SEC(".maps");

SEC("xdp")
int xdp_firewall(struct xdp_md *ctx) {
    void *data = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;

    // Parse Ethernet header
    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)
        return XDP_PASS;
    if (eth->h_proto != __constant_htons(ETH_P_IP))
        return XDP_PASS;

    // Parse IP header
    struct iphdr *ip = (void *)(eth + 1);
    if ((void *)(ip + 1) > data_end)
        return XDP_PASS;

    // Check if source IP is blocked
    __u32 src_ip = ip->saddr;
    __u64 *counter = bpf_map_lookup_elem(&blocked_ips, &src_ip);
    if (counter) {
        __sync_fetch_and_add(counter, 1);
        return XDP_DROP;  // Drop at NIC driver level — never hits kernel stack
    }

    return XDP_PASS;
}

TC (Traffic Control) / cls_bpf — Ingress & Egress

• Runs after XDP, in the Traffic Control layer (ingress and egress).
• Advanced traffic shaping, QoS, L4/L7 load balancing, packet mangling.

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

SEC("classifier")
int tc_ingress(struct __sk_buff *skb)
{
    void *data_end = (void *)(long)skb->data_end;
    void *data = (void *)(long)skb->data;
    struct ethhdr *eth = data;

    if ((void *)eth + sizeof(*eth) > data_end)
        return TC_ACT_OK;

    // Count packets or apply custom logic
    // Example: Drop UDP packets
    if (eth->h_proto == __constant_htons(ETH_P_IP)) {
        struct iphdr *iph = data + sizeof(*eth);
        if ((void *)iph + sizeof(*iph) <= data_end && iph->protocol == IPPROTO_UDP)
            return TC_ACT_SHOT;   // Drop
    }

    return TC_ACT_OK;
}

What is __sk_buff?

__sk_buff is the kernel’s internal representation of a network packet once it has entered the networking stack.

It is a C structure (struct __sk_buff) that contains:

• Pointers to packet data (data, data_end)
• Metadata: packet length, protocol, interface index, marks, priority, etc.
• Socket-related information (if associated)
• Routing, firewall, and connection tracking state

Unlike XDP’s lightweight xdp_md context, __sk_buff is a much heavier structure because the packet has already gone through several layers of the kernel network stack.

XDP vs sk_buff: Why XDP Is Dramatically Faster

The key to XDP’s speed is where it intercepts the packet. Traditional BPF hooks (TC, socket filters) operate on sk_buff — the kernel’s fully-allocated socket buffer. XDP operates on raw xdp_md before sk_buff is ever created.

Why does sk_buff allocation matter?

// XDP context — minimal, ~8 bytes effective
struct xdp_md {
    __u32 data;           // Pointer to packet start
    __u32 data_end;       // Pointer to packet end
    __u32 data_meta;      // Metadata area before packet
    __u32 ingress_ifindex;// Ingress interface
    __u32 rx_queue_index; // RX queue
    __u32 egress_ifindex; // Egress (for XDP_REDIRECT)
};
// Total: 24 bytes, no allocation — lives on stack

// sk_buff — full kernel packet descriptor
struct sk_buff {
    // ... 232+ bytes, heap-allocated via slab
};

sk_buff cost per packet:
├── slab_alloc()          ~30-50ns   (SLUB allocator)
├── memset 232 bytes      ~10-20ns   (zero initialization)
├── field setup           ~20-30ns   (protocol, device, headers)
├── skb_shared_info       ~40 bytes  (at skb->end, clones/frags)
└── cache pollution       variable   (evicts hot cache lines)
                          ─────────
                Total:    ~80-150ns  BEFORE any processing

At 10M packets/sec:
  = 10M × 150ns = 1.5 seconds of CPU time per second
  = 150% of one core JUST for sk_buff allocation

Getting Started

Prerequisites

# Ubuntu/Debian
sudo apt install -y linux-headers-$(uname -r) \
    libbpf-dev bpftrace bpfcc-tools linux-tools-$(uname -r)

# Verify BTF support
ls /sys/kernel/btf/vmlinux && echo "BTF available"

# Verify BPF JIT
cat /proc/sys/net/core/bpf_jit_enable  # Should be 1 or 2

First Program with bpftrace

# Count syscalls per second
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @ = count(); } 
interval:s:1 { print(@); clear(@); }'

# Trace process exec
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { 
    printf("%s -> %s\n", comm, str(args->filename)); }'

# TCP retransmit tracing
sudo bpftrace -e 'tracepoint:tcp:tcp_retransmit_skb {
    printf("retransmit: %s:%d -> %s:%d\n",
        ntop(args->saddr), args->sport,
        ntop(args->daddr), args->dport);
}'

Tracking TCP Connection Latency with Maps

Kernel space code

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, u32);      // Thread ID
    __type(value, u64);    // Timestamp
} start_times SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_PERCPU_HASH);
    __uint(max_entries, 10240);
    __type(key, u64);           // Latency in nanoseconds
    __type(value, u64);         // Count
} latency_hist SEC(".maps");

SEC("kprobe/tcp_connect")
int trace_tcp_connect(struct pt_regs *ctx)
{
    u32 tid = bpf_get_current_pid_tgid();
    u64 ts = bpf_ktime_get_ns();

    bpf_map_update_elem(&start_times, &tid, &ts, BPF_ANY);
    return 0;
}

SEC("kretprobe/tcp_connect")
int trace_tcp_connect_ret(struct pt_regs *ctx)
{
    u32 tid = bpf_get_current_pid_tgid();
    u64 *start = bpf_map_lookup_elem(&start_times, &tid);

    if (start) {
        u64 latency = bpf_ktime_get_ns() - *start;
        u64 slot = latency >> 10;  // Rough bucketing

        u64 *count = bpf_map_lookup_elem(&latency_hist, &slot);
        if (count) (*count)++;
        else {
            u64 one = 1;
            bpf_map_update_elem(&latency_hist, &slot, &one, BPF_ANY);
        }

        bpf_map_delete_elem(&start_times, &tid);
    }
    return 0;
}

char __license[] SEC("license") = "GPL";

User space code (using golang)

// main.go
package main

import (
 "context"
 "fmt"
 "log"
 "os"
 "os/signal"
 "syscall"
 "time"

 "github.com/cilium/ebpf"
 "github.com/cilium/ebpf/link"
 "github.com/cilium/ebpf/rlimit"
)

func main() {
 // Allow the current process to lock memory for eBPF maps
 if err := rlimit.RemoveMemlock(); err != nil {
  log.Fatal(err)
 }

 // Load the compiled BPF object
 spec, err := ebpf.LoadCollectionSpec("tcp_latency.bpf.o")
 if err != nil {
  log.Fatalf("Failed to load BPF spec: %v", err)
 }

 // Load the collection into the kernel
 coll, err := ebpf.NewCollection(spec)
 if err != nil {
  log.Fatalf("Failed to create collection: %v", err)
 }
 defer coll.Close()

 // Attach kprobe and kretprobe
 kp, err := link.Kprobe("tcp_connect", coll.Programs["trace_tcp_connect"], nil)
 if err != nil {
  log.Fatalf("Failed to attach kprobe: %v", err)
 }
 defer kp.Close()

 kretp, err := link.Kretprobe("tcp_connect", coll.Programs["trace_tcp_connect_ret"], nil)
 if err != nil {
  log.Fatalf("Failed to attach kretprobe: %v", err)
 }
 defer kretp.Close()

 fmt.Println("TCP Connection Latency Tracker Started (Press Ctrl+C to stop)")

 // Print latency histogram every 3 seconds
 ticker := time.NewTicker(3 * time.Second)
 defer ticker.Stop()

 ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
 defer stop()

 for {
  select {
  case <-ctx.Done():
   fmt.Println("\nShutting down...")
   return
  case <-ticker.C:
   printLatencyHistogram(coll.Maps["latency_hist"])
  }
 }
}

// printLatencyHistogram reads and displays the latency histogram
func printLatencyHistogram(m *ebpf.Map) {
 fmt.Printf("\n=== TCP Connect Latency Histogram (last 3s) ===\n")
 fmt.Printf("%-12s %s\n", "LATENCY", "COUNT")

 var bucket uint64
 var count uint64

 iter := m.Iterate()
 for iter.Next(&bucket, &count) {
  latencyNs := bucket << 10 // reverse the >>10 bucketing
  latencyMs := float64(latencyNs) / 1_000_000.0

  fmt.Printf("%8.2f ms    %d\n", latencyMs, count)
 }

 if err := iter.Err(); err != nil {
  log.Printf("Map iteration error: %v", err)
 }
}

Build & Run

# 1. Compile the BPF program
clang -g -O2 -target bpf -D__TARGET_ARCH_x86 \
     -I/usr/include -c tcp_latency.bpf.c -o tcp_latency.bpf.o

# 2. Install Go dependencies
go mod init tcp-latency
go get github.com/cilium/ebpf

# 3. Build and run
go build -o tcp-latency
sudo ./tcp-latency

Output

TCP Connection Latency Tracker Started (Press Ctrl+C to stop)

=== TCP Connect Latency Histogram (last 3s) ===
LATENCY       COUNT
    0.02 ms    1245
    0.08 ms     892
    0.15 ms     234
    0.45 ms      67
    1.25 ms      23
    3.80 ms       8
   12.50 ms       3
   45.20 ms       1

=== TCP Connect Latency Histogram (last 3s) ===
LATENCY       COUNT
    0.02 ms    1341
    0.08 ms     765
    0.22 ms     189
    0.65 ms      45
    2.10 ms      12
    8.50 ms       4
   28.40 ms       2

=== TCP Connect Latency Histogram (last 3s) ===
LATENCY       COUNT
    0.01 ms    2156
    0.04 ms    1023
    0.12 ms     345
    0.55 ms      78
    4.20 ms       9
   15.80 ms       2

Real-World Case Studies: eBPF in Production

1. Cloudflare: Terabit-Scale DDoS Mitigation & Observability

• Absorbs 2+ Tbps of DDoS attacks by dropping malicious packets
• XDP-based eBPF programs for packet filtering and “Magic Firewall”.

2. Netflix: Network Defense & Large-Scale Telemetry

• Deep visibility into network behavior without instrumenting every service.
• eBPF for network flow insights and performance troubleshooting.

3. Meta: The L4 Load Balancer (Katran)

• Strobelight (eBPF-based continuous profiler) + packet processing for every packet entering their data centers.
• Reduced CPU cycles and server load by up to 20%.

4. Major Cloud Providers: Google GKE, AWS EKS, and Azure

• Adopted Cilium (eBPF-based CNI) as the default or preferred networking solution.
• https://aws.amazon.com/blogs/opensource/getting-started-with-cilium-service-mesh-on-amazon-eks/

eBPF has fundamentally transformed how we observe, secure, and operate Linux systems.

We are no longer waiting for the OS to tell us what happened; we are observing reality at the very moment it hits the wire. This isn’t just a performance optimization; it’s a fundamental change in how we treat the network -moving from Software-Defined Networking to Hardware-Accelerated Intelligence.

In mission-critical environments -whether it’s a surgical display in a hospital or a high-frequency trading bot -CPU cycles are currency. Every cycle spent by a traditional security agent “scanning” a packet is a cycle stolen from the primary application.

eBPF is becoming the default foundation for Kubernetes networking (Cilium), security (Tetragon, Falco), observability (Parca, Beyla), and profiling.

It’s a paradigm shift — turning the kernel from a black box into a programmable, observable, and intelligent platform.

References —

[embed]Linux Observability with BPF Build your expertise in the BPF virtual machine in the Linux kernel with this practical guide for systems engineers…learning.oreilly.com

[embed]What is eBPF? An Introduction and Deep Dive into the eBPF Technology A detailed step by step introduction to the eBPF technology with lots of references for further reading.ebpf.io

If you would like to stay connected, follow this account and stay tuned.

Watch this space ! 👋