Stop Encrypting Kafka PII in Every Microservice. Do It with a Kafka Proxy.

A hands-on guide to field-level encryption with a Kafka proxy — no per-application crypto, no broker changes.

At Current 2026 in London, an engineer told his team had a hard requirement to encrypt personally identifiable data (PII) flowing through Kafka. So they did what almost everyone does: they built it into the application layer (Java, Go etc.).

Every producer encrypts before it sends. Every consumer decrypts after it reads. It worked. It also took them massive amount of time to align teams, code, SDK, dependencies, CVE etc. and it has to be re-implemented in every new service, in every language, and they are now responsible of maintaining the whole thing.

Talking with us, he found out a Kafka proxy could have done the whole thing at the message level without changing any application code at all.

If you are doing the same thing on your side, this post is for you. We are going to encrypt specific PII fields in a Kafka record without touching your producers, your consumers, or your brokers. The encryption happens inside a proxy that sits between your clients and Kafka. Let’s build it.

The problem with app-layer encryption

Per-application encryption has many issues:

• Credentials and hard dependency to your KMS. Each team needs their own keys/tokens to access your KMS. AND you have to open the access to your KMS (network).
• It does not compose. Multiple services that produce to the same topic need five exact same implementations, whatever the programming language they are using or the versions. It means: strong technical alignment.
• It leaks across languages. Your Java service and your Go service now need compatible crypto libraries and identical key handling.
• Key access is everywhere. Every service that touches the data holds a key. Your blast radius is huge.
• You cannot change your mind. Want to rotate a key, add a field, or decrypt for one team but not another? That is a redeploy across every service.

The problem is that encryption logic lives in the wrong place now. It is a policy, and policy does not belong scattered across application code. A policy is meant to be central. This is why a proxy is the one place to put them.

How proxy-level encryption works

A Kafka proxy speaks the Kafka protocol. Clients connect to it instead of to the brokers, and it forwards traffic through. The Clients are not even aware they are talking to a proxy. It’s like nginx or traefik for reverse HTTP-proxy. It’s transparent.

Because every record passes through it, it can transform records in flight. Conduktor Gateway implements this with Interceptors: small plugins that fire on the produce path, the consume path, or both.

For encryption, two Interceptors matter:

• EncryptPlugin runs on produce. Records leave the client in plaintext, the plugin encrypts the fields you name, and the ciphertext is what lands in Kafka (your Kafka Provider cannot access the raw data). Data is encrypted before it ever enters the broker.
• DecryptPlugin runs on consume. The broker holds ciphertext; the plugin decrypts on the way out, transparently, for clients that are allowed to read it. Even if a user has access/ACLs, if they don't have the decryption key, they won't be able to read the data.

The keys live in a KMS, HashiCorp Vault for instance: never in the proxy and never in your apps.

TLDR: Data is encrypted at rest on disk (which generally is itself encrypted), and plaintext data is only ever visible to authorized clients. Producers and consumers stay completely unaware this is happening.

Prerequisites

You will need:

• A running Kafka cluster (MSK, Confluent, Red Panda, or local, Gateway does not care).
• Conduktor Gateway in front of it, with clients pointed at the Gateway bootstrap server.
• A KMS. We use Vault’s Transit engine here.
• The Conduktor CLI or Console to apply Interceptor configs.

We will encrypt a customers topic whose JSON records look like this:

{
  "userId": "sec-12345678",
  "fullName": "Ada Lovelace",
  "password": "admin123",
  "visa": "4111111145551142"
}

The goal: password and visa get encrypted at rest. userId and fullName stay readable. No producer or consumer code changes.

Step 1 — Encrypt specific fields on produce

Field-level encryption lets you target only the sensitive parts of a message. Create a file called encryption-interceptor.yaml:

apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: encrypt-customer-pii
  scope:
    vCluster: passthrough
spec:
  pluginClass: io.conduktor.gateway.interceptor.EncryptPlugin
  priority: 100
  config:
    topic: "customers"
    kmsConfig:
      vault:
        uri: http://vault:8200
        token: ${VAULT_TOKEN}
    recordValue:
      fields:
        - fieldName: password
          keySecretId: vault-kms://vault:8200/transit/keys/password-key
          algorithm: AES256_GCM
        - fieldName: visa
          keySecretId: vault-kms://vault:8200/transit/keys/visa-key
          algorithm: AES256_GCM

What are we doing here:

• topic: "customers" is a regex. You can scope a single topic, or match sensitive-.* to cover a whole naming convention.
• Each field gets its own key (password-key, visa-key). That means you can later decrypt one without exposing the other.
• fieldName supports nested paths with dots (education.account.username) and array indexes like banks[0].accountNo.
• priority: 100. Lower numbers run first. Spacing priorities (100, 200, 300) lets you slot other Interceptors in between later without renumbering.

Apply it with the CLI:

conduktor apply -f encryption-interceptor.yaml
# Interceptor/encrypt-customer-pii: Created

Produce a record through the Gateway exactly as you would to any Kafka cluster:

echo '{"userId":"sec-12345678","fullName":"Ada Lovelace","password":"admin123","visa":"4111111145551142"}' \
  | kafka-console-producer \
      --bootstrap-server gateway:6969 \
      --topic customers

Your producer did nothing special. It does not know encryption happened.

Step 2 — Verify the data is encrypted at rest

Consume without a decryption Interceptor in place, and you will see the ciphertext sitting on the broker:

{
  "userId": "sec-12345678",
  "fullName": "Ada Lovelace",
  "password": "AwAAAAEAEBm…<ciphertext>…",
  "visa": "AwAAAAEAEPk…<ciphertext>…"
}

userId and fullName are untouched. password and visa are encrypted, and they are encrypted on disk, anyone with raw broker or backup access sees ciphertext only. This is the property that app-layer encryption is supposed to give you, except here you got it without writing a line of crypto.

Step 3 — Decrypt transparently on consume

Now let authorized consumers read plaintext. Create decryption-interceptor.yaml:

apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: decrypt-customer-pii
  scope:
    vCluster: passthrough
spec:
  pluginClass: io.conduktor.gateway.interceptor.DecryptPlugin
  priority: 200
  config:
    topic: "customers"
    kmsConfig:
      vault:
        uri: http://vault:8200
        token: ${VAULT_TOKEN}
    recordValueFields:
      - password
      - visa

Notes:

• recordValueFields lists which fields to decrypt. Leave it empty and Gateway decrypts every encrypted field it finds, handy when you want all-or-nothing.
• There are matching recordKeyFields and recordHeaderFields if your sensitive data lives in the key or headers.
• errorPolicy defaults to return_encrypted: if decryption fails, the consumer gets ciphertext rather than an error. You can switch it to fail_fetch if you would rather fail hard.

Apply it, consume, and you get your original record back:

{
  "userId": "sec-12345678",
  "fullName": "Ada Lovelace",
  "password": "admin123",
  "visa": "4111111145551142"
}

The data is still encrypted at rest. If you delete the decryption Interceptor, every consumer immediately goes back to seeing ciphertext. That switch is your access control surface — and it is one config change, not a fleet redeploy.

Step 4 — Decide who gets to decrypt

This is where the proxy model is so much ahead of app-layer encryption. Interceptors are scoped. Instead of vCluster: passthrough (everyone), you can bind a decryption Interceptor to a specific service account or group:

metadata:
  name: decrypt-customer-pii-fraud-team
  scope:
    group: fraud-analytics

Now only the fraud-analytics group sees decrypted visa numbers. Every other consumer on the same topic reads ciphertext. More specific scopes override broader ones, and Gateway applies the right policy dynamically based on which client connects, so different teams get different views of the same topic without separate clusters. Try expressing that with per-application encryption.

More flexibility: Field-level with full-payload fallback

The real world is messy. Some records have the sensitive fields; some do not. You can run field-level encryption first, then a full-payload encryption Interceptor as a catch-all, using priorities to order them:

# Priority 1: encrypt known PII fields
apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: fieldLevelEncrypt
  scope:
    vCluster: passthrough
spec:
  pluginClass: io.conduktor.gateway.interceptor.EncryptPlugin
  priority: 1
  config:
    topic: "sensitive-.*"
    kmsConfig:
      vault:
        uri: http://vault:8200
        token: ${VAULT_TOKEN}
    recordValue:
      fields:
        - fieldName: password
          keySecretId: vault-kms://vault:8200/transit/keys/password-key
          algorithm: AES256_GCM
        - fieldName: visa
          keySecretId: vault-kms://vault:8200/transit/keys/visa-key
          algorithm: AES256_GCM
---
# Priority 2: anything not already handled gets full-payload encryption
apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: fullPayloadEncryptFallback
  scope:
    vCluster: passthrough
spec:
  pluginClass: io.conduktor.gateway.interceptor.EncryptPlugin
  priority: 2
  config:
    topic: "sensitive-.*"
    errorPolicy: skip_already_encrypted
    kmsConfig:
      vault:
        uri: http://vault:8200
        token: ${VAULT_TOKEN}
    recordValue:
      payload:
        keySecretId: vault-kms://vault:8200/transit/keys/full-payload-key
        algorithm: AES256_GCM

skip_already_encrypted keeps the fallback from double-encrypting records the first Interceptor already handled. e.g. a record with a visa field gets field-level treatment; a record shaped differently still does not leak, because the whole payload gets encrypted. No record in plaintext, ever.

What you actually built

Encryption is no longer application logic: it is a few YAML files applied to a proxy.

• Keys live in Vault, not in your services.
• Data is ciphertext at rest.
• Plaintext visibility is a configurable, revocable policy you change in one place.

Add a new microservice tomorrow and it inherits all of this for free, in any language, because it just talks Kafka.

The platform team can have this without rewriting a single producer or consumer. That is the part developers were excited about at Current 2026 when talking with us: most people think a proxy just passes packets between legacy clients and the cluster. What it actually does is hold the policy: encryption, masking, validation, access control, and much more in the one configurable place.

Deeper reference: our Gateway encryption docs that cover schema-based encryption, multiple KMS providers, and crypto-shredding for GDPR erasure.

If you’re curious about other Kafka learnings from the community: What We Learned at Current 2026.