From JSON to Avro: Fixing Schema Chaos in Our Kafka CDC Pipeline

The Real Problem

Our CDC pipeline using Debezium + Apache Kafka + Spark was working fine.

But scaling exposed the real issue:

👉 Schema management was completely broken

Why JSON Didn’t Scale

A typical event:

{
  "payload": {
    "after": {"id": 123, "name": "John"},
    "op": "c"
  }
}

Looks simple… but:

❌ Schema not in message ❌ Stored separately (S3) ❌ Manual sync required

Old Architecture (JSON + S3 Schema)

Problems We Faced

Tipping Point

A single table onboarding took 3 days:

• Day 1 → Schema upload
• Day 2 → Consumer broke
• Day 3 → Fix + missing column again

👉 That’s when we knew: This approach won’t scale

The Solution

👉 Move to Avro + AWS Glue Schema Registry

Before vs After

How It Works

1. Debezium → Avro

{
  "value.converter": "io.confluent.connect.avro.AvroConverter",
  "value.converter.schema.registry.url": "glue-endpoint"
}

2. Message Structure

[Header][Avro Payload]

Header:
- Schema ID
- Version

3. Consumer Change (Game Changer)

Before:

df.select(from_json(col("value"), schema_json))

After:

df.select(from_avro(expr("substring(value, 19)"), schema))

👉 No manual schema fetching

Schema Evolution (Finally Solved)

ALTER TABLE users ADD COLUMN phone_verified BOOLEAN;

✔ New schema auto-registered ✔ Consumers auto-adapt ✔ Old data → NULL ✔ No deployment needed

What We Gained

• 🚫 No schema drift
• ⚡ Faster processing (binary Avro)
• 📦 Smaller messages (~40% reduction)
• 🤝 No cross-team coordination
• 🧠 Zero schema headaches

Key Takeaways

• JSON CDC pipelines break at scale due to external schema handling
• Schema Registry = single source of truth
• Avro = performance + evolution
• Removing manual steps = massive productivity boost

Final Thought

If your pipeline “works” but needs:

• manual schema updates
• coordination across teams
• frequent debugging

👉 It’s already slowing you down.

Move to Schema Registry early — you’ll thank yourself later.