Your Kubernetes Cluster Got Hacked. Kyverno Didn’t Stop It. Here’s What Would Have.

Runtime security is the last line of defense you’re probably not running. This is a hands-on guide to deploying Falco — the CNCF’s eBPF-powered threat detection engine — with a working demo you can run on your laptop in 5 minutes.

The $10 Million Blind Spot

In January 2024, a cryptojacking campaign hit Kubernetes clusters across multiple cloud providers. The attackers didn’t need to bypass any admission controllers. They exploited a known vulnerability in a web application, got a shell inside a legitimate container, read the Kubernetes service account token, and pivoted to steal AWS credentials — all within a pod that had passed every policy check.

The cost? Hundreds of thousands in stolen compute for mining Monero. And most victims didn’t know it was happening until their cloud bill arrived.

The attack followed a textbook pattern:

1. ✅ Pod passed all admission policies (correct labels, non-root, signed image)
2. 💀 Attacker exploited an RCE vulnerability in the running app
3. 🐚 Attacker spawned a shell inside the container
4. 🔑 Attacker read the ServiceAccount token at /var/run/secrets/kubernetes.io/serviceaccount/token
5. 🌐 Attacker connected outbound to a C2 server
6. ⛏️ Attacker deployed a crypto miner on the cluster’s compute

Steps 3–6 happened inside a running container — completely invisible to admission control tools like Kyverno, OPA Gatekeeper, or Kubernetes PSA.

If you’ve read my Kyverno in Action article, you know how to lock down step 1. But what about steps 3–6?

You need runtime security. And the tool the industry has converged on is Falco.

What is Falco? (30-Second Version)

**Falco** is a CNCF graduated project — same maturity level as Kubernetes, Prometheus, and Envoy. It uses eBPF to monitor Linux kernel syscalls in real-time across every container on every node.

When a container does something suspicious — opens /etc/shadow, spawns a shell, connects to a mining pool — Falco detects it at the kernel level and fires an alert in milliseconds.

Kyverno = the bouncer at the arena door (checks IDs before you enter) Falco = the security camera system inside the arena (watches what you do once you’re in)

You need both. A bouncer stops the guy without a ticket. But the security cameras catch the fan who snuck in through the loading dock and is now running across the court.

The Demo: NBA Arena Security

I built a full working demo that simulates this exact attack chain. Following my *-in-action series* pattern, it uses an NBA arena as the analogy:

Everything runs locally on Minikube. No cloud account needed.

Setup (5 Minutes, Really)

git clone https://github.com/23seriy/FALCO-in-action.git
cd FALCO-in-action
chmod +x scripts/*.sh

./scripts/01-install-prerequisites.sh   # minikube, kubectl, helm
./scripts/02-start-cluster.sh           # minikube + Falco + Falcosidekick
./scripts/03-deploy-app.sh              # build images + deploy demo apps

Or run all 10 scenarios interactively:

./scripts/04-demo-scenarios.sh

Prerequisites: macOS with Docker Desktop running, ~8 GB RAM available.

7 Attack Scenarios (With Real Falco Output)

Each scenario maps to a MITRE ATT&CK technique — the same framework your SOC team uses to classify real-world threats.

🐚 1. Shell in Container — T1609

The attack: Spawning an interactive shell inside a running container. In production, this happens when an attacker exploits an RCE vulnerability.

kubectl exec -n falco-demo deploy/rogue-player -- /bin/sh -c "whoami && id"

Falco fires:

{
  "rule": "Terminal shell in container",
  "priority": "Warning",
  "output": "A shell was spawned in a container with an attached terminal
    (pod=rogue-player container=rogue-player shell=sh)"
}

🏀 NBA analogy: A fan jumping the barrier and running onto the court.

Why it matters: This is the most fundamental indicator of compromise. If someone has a shell in your production container, something has already gone very wrong.

📄 2. Read Sensitive File — T1552

The attack: Reading /etc/shadow to harvest password hashes for offline cracking.

kubectl exec -n falco-demo deploy/rogue-player -- cat /etc/shadow

Falco fires:

{
  "rule": "Read sensitive file untrusted",
  "priority": "Warning",
  "output": "Sensitive file opened for reading (file=/etc/shadow user=root pod=rogue-player)"
}

🏀 NBA analogy: Sneaking into the opponent’s locker room to steal their playbook.

💀 3. Write Below Binary Dir — T1525

The attack: Dropping a backdoor binary into /usr/bin for persistence.

kubectl exec -n falco-demo deploy/rogue-player -- \
  /bin/sh -c "echo '#!/bin/sh' > /usr/bin/backdoor && chmod +x /usr/bin/backdoor"

🏀 NBA analogy: Rigging the scoreboard with your own equipment.

📦 4. Package Management — T1105

The attack: Running apt-get inside a container — a red flag in production. Containers should be immutable.

kubectl exec -n falco-demo deploy/rogue-player -- apt-get --version

🏀 NBA analogy: A player bringing outside equipment onto the court mid-game.

🔑 5. Kubernetes Credential Theft — T1552.007

The attack: The first thing attackers do after getting a shell — read the ServiceAccount token to pivot within the cluster.

kubectl exec -n falco-demo deploy/rogue-player -- \
  cat /var/run/secrets/kubernetes.io/serviceaccount/token

Falco fires our custom rule:

{
  "rule": "Arena Pod Reading K8s Secrets",
  "priority": "Critical",
  "output": "K8s credential access detected in arena pod
    (pod=rogue-player file=/var/run/secrets/kubernetes.io/serviceaccount/token)"
}

🏀 NBA analogy: Stealing the coach’s iPad with all the play diagrams.

Why it matters: With the SA token, an attacker can query the Kubernetes API, list secrets, and potentially take over the entire cluster. This is why best practice is automountServiceAccountToken: false on every pod that doesn't need it. (Our compliant arena-security-api pod does this.)

🌐 6. Outbound Connection — T1041

The attack: A compromised pod phoning home to a C2 server.

kubectl exec -n falco-demo deploy/rogue-player -- python -c \
  "import socket; s=socket.socket(); s.settimeout(2); s.connect(('1.1.1.1',443))"

Falco fires our custom rule:

{
  "rule": "Arena Pod Making Outbound Connection",
  "priority": "Warning",
  "output": "Outbound connection from arena pod
    (pod=rogue-player dest=1.1.1.1:443)"
}

🏀 NBA analogy: A player sneaking out of the arena to meet their agent during the game.

⛏️ 7. Crypto Mining — T1496

The attack: Connecting to a mining pool — the #1 container attack in the wild.

kubectl exec -n falco-demo deploy/rogue-player -- python -c \
  "import socket; s=socket.socket(); s.settimeout(2); s.connect(('1.1.1.1',45700))"

🏀 NBA analogy: Using the arena’s generator to power your Bitcoin rig hidden under the bleachers.

Writing Custom Rules: Teaching Falco Your Arena’s Layout

Falco ships with 100+ built-in rules. But every environment has unique threats. Writing custom rules is straightforward — and this is where Falco really shines.

Here’s one of the 5 custom rules from this demo:

# MITRE ATT&CK: T1041 (Exfiltration Over C2 Channel)
- rule: Arena Pod Making Outbound Connection
  desc: >
    Detect when a pod in the falco-demo namespace makes an outbound
    connection to an external IP address.
  condition: >
    evt.type in (connect, sendto) and
    evt.dir = < and
    container.id != host and
    k8s.ns.name = "falco-demo" and
    fd.typechar = 4 and
    not fd.snet in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "127.0.0.0/8")
  output: >
    Outbound connection from arena pod
    (pod=%k8s.pod.name connection=%fd.name dest=%fd.sip:%fd.sport user=%user.name)
  priority: WARNING
  tags: [T1041, network, nba-arena, mitre_exfiltration]

The anatomy of a Falco rule:

Pro tip: Always tag rules with MITRE ATT&CK technique IDs. This gives your SOC team a common language and maps directly to their incident response playbooks.

Alert Routing with Falcosidekick

Falco writes alerts to stdout. That’s useful for kubectl logs, but useless for incident response. Enter Falcosidekick — it receives Falco events and routes them to 60+ outputs:

• Slack → #security-alerts channel
• PagerDuty → on-call rotation for Critical alerts
• AWS CloudWatch / S3 → audit trail
• Elasticsearch / Splunk → searchable alert history
• Webhook → your custom SOC dashboard (what this demo does)
• Falcosidekick UI → built-in web dashboard (included)

In this demo, after running all 7 attack scenarios:

curl http://localhost:9082/alerts/summary

{
  "total_alerts": 15,
  "by_rule": {
    "Arena Pod Making Outbound Connection": 2,
    "Arena Pod Reading K8s Secrets": 1,
    "Arena Pod Suspicious DNS Lookup": 4,
    "Read sensitive file untrusted": 1
  },
  "by_priority": {
    "Critical": 8,
    "Warning": 3,
    "Notice": 4
  }
}

Every alert routed automatically. Zero manual log parsing.

The MITRE ATT&CK Mapping

Every scenario in this demo maps to a real-world attack technique. This isn’t academic — it’s how your incident response team will classify and prioritize threats:

Kyverno + Falco: The Complete Defense

If you’ve followed my series, you know the punchline:

Kyverno prevents known bad configurations. Falco detects unknown bad behavior.

The attack chain from the opening story:

1. ✅ Pod passes all Kyverno policies → admitted to the cluster
2. 💀 Attacker exploits an RCE bug in the app
3. 🐚 Attacker spawns a shell → Falco alerts 🚨
4. 🔑 Attacker reads the SA token → Falco alerts 🚨
5. 🌐 Attacker connects to C2 → Falco alerts 🚨
6. ⛏️ Attacker deploys crypto miner → Falco alerts 🚨

Without Falco, steps 3–6 happen in complete silence. The first sign is your next cloud bill.

Key Takeaways

1. Admission control is necessary but insufficient. The 2024 Kubernetes security report found that 89% of clusters had at least one container with a known vulnerability. Compliant pods get compromised at runtime. Falco watches what containers actually do.
2. eBPF is the industry’s bet. Falco, Cilium, Datadog, and Tetragon all use eBPF for kernel-level observability. No kernel modules, no sidecars, no code changes. It’s the future of infrastructure security.
3. Custom rules are your competitive advantage. Built-in rules cover generic threats. The 20% specific to your environment — unusual outbound connections, credential access patterns, mining indicators — that’s where custom rules make the difference.
4. Alert routing makes detection actionable. Falcosidekick routes to 60+ outputs. Detection without notification is just expensive logging.
5. Map everything to MITRE ATT&CK. Tag your rules with technique IDs. Your SOC team speaks MITRE. Your incident response playbooks reference MITRE. Make Falco speak it too.
6. Defense in depth is not optional. Kyverno at the door. Falco inside. NetworkPolicies on the wire. Like an NBA arena: ID checks, security cameras, and a SOC watching the feeds.

Try It Yourself

The entire project is open source and runs on your laptop:

git clone https://github.com/23seriy/FALCO-in-action.git
cd FALCO-in-action
./scripts/01-install-prerequisites.sh
./scripts/02-start-cluster.sh
./scripts/03-deploy-app.sh
./scripts/04-demo-scenarios.sh    # 10 interactive scenarios

⭐ Star the repo if you found this useful: github.com/23seriy/FALCO-in-action

Resources

• Falco Documentation
• Falco Rules Reference
• Falcosidekick — 60+ Alert Outputs
• MITRE ATT&CK for Containers
• eBPF.io — What is eBPF?
• Sysdig 2024 Container Security Report

This is part of my “in Action” series — hands-on Kubernetes projects you can clone and run on your laptop. Each one takes a CNCF tool, wraps it in a practical demo, and explains it through an NBA analogy because security shouldn’t be boring.

More in the series:

• *Kyverno in Action — Admission control & policy-as-code*
• *Cilium in Action — eBPF networking, L7 policies, Hubble observability*
• *Crossplane in Action — Kubernetes-native infrastructure management*
• *Argo Rollouts in Action — Progressive delivery & canary deployments*
• *KEDA in Action — Event-driven autoscaling*

Follow me on Medium for the next one. 🏀

Tags:

falco, k8s, devops, kubernetes, cloud-native, infrastructure-as-a-code