Practical Guide to Schema Compatibility in Kafka

Schema compatibility sits at the core of building reliable and evolvable Kafka-based systems. Every schema change — adding a field, renaming an existing one, altering optionality — can impact downstream consumers. Without clear rules, even a small schema update can break dozens of services.

A schema registry protects your system by enforcing these rules during schema evolution. Understanding how these rules work and how to configure them effectively is essential for designing stable, long-lived event pipelines.

Overview

When you register a new version of an existing schema, the Schema Registry validates it against earlier versions according to the compatibility mode you have configured. Each mode defines how the latest version must relate to previous ones, ensuring that producers and consumers — some of which still rely on older schemas — continue to interoperate safely. Compatibility checks apply only when a subject already has earlier versions.

Schema Registry offers several compatibility modes that govern how schemas may evolve.

Compatibility Modes

Backward

Ensures that new schemas remain readable by older consumers. This protects long-running or legacy services whenever producers evolve.

Forward

Ensures that old data remains readable by consumers using the new schema. Useful when upgrading consumers first, while producers still emit older versions.

Full

Requires compatibility in both directions — new schemas must be compatible with existing consumers, and new consumers must work with old schemas. This offers the strongest safety guarantees but is also the most restrictive.

Fine-Grained Compatibility: scope

In addition to selecting a compatibility mode (Backward, Forward, or Full), Schema Registry allows you to define how far back compatibility checks must reach. This creates a two-dimensional model of schema validation:

• Compatibility Type: BACKWARD, FORWARD, FULL, NONE
• Compatibility Scope: latest version only (non-transitive) or all previous versions (transitive)

This second dimension — the number of historical versions the registry checks — is often overlooked but is crucial for systems that must evolve safely over long periods.

Non-Transitive Compatibility (latest version only)

In this scope, the registry validates a new schema only against the most recent version of the subject.

Use when:

• Deployments are coordinated or predictable
• Schema evolution happens gradually
• Consumers do not depend on very old schema versions
• You want faster validation and more flexibility

Transitive Compatibility (all previous versions)

In this stricter scope, the registry validates a new schema against every historical version of the subject.

Use when:

• Consumers may deserialize years-old or archival messages
• You store long-lived data in compacted or immutable topics
• Many services rely on the same topic for long lifecycles
• You need strong, long-term guarantees

Transitive compatibility prevents systems from accumulating many small schema updates that are individually safe but collectively incompatible with older versions.

Mapping to Real Schema Registry Settings

Compatibility scopes map directly to Schema Registry configuration values. Transitive variants extend how far back compatibility checks apply:

• BACKWARD_TRANSITIVE: backward compatibility with all previous versions
• FORWARD_TRANSITIVE: forward compatibility with all previous versions
• FULL_TRANSITIVE: backward and forward compatibility with all previous versions

Their non-transitive counterparts (BACKWARD, FORWARD, FULL) validate compatibility only against the latest version.

Transitive variants do not introduce new types of compatibility; they refine the scope of validation for long-term schema evolution.

Practical Configuration Options in Schema Registries

Understanding compatibility modes and scopes conceptually is one part of schema evolution. The next step is to understand how to apply these settings in real systems. Schema registries expose compatibility configurations through APIs, letting you control both the compatibility type and scope.

Setting Compatibility Modes (Including Transitive Variants)

# Set Backward compatibility for a subject (latest version only)
curl -X PUT \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "BACKWARD"}' \
  http://localhost:8081/config/customer-events

# Set Full Transitive compatibility (check against ALL previous versions)
curl -X PUT \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FULL_TRANSITIVE"}' \
  http://localhost:8081/config/customer-events

# Get the current compatibility mode
curl -X GET \
  http://localhost:8081/config/customer-events

These examples show how to enforce both standard (latest-only) compatibility modes and their stricter transitive variants.

Registering Schemas via the API

curl -X POST \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{
    "schema": "{ \"type\": \"record\", \"name\": \"UserCreated\", \"fields\": [ { \"name\": \"id\", \"type\": \"string\" } ] }"
  }' \
  http://localhost:8081/subjects/customer-events-value/versions

The registry validates this schema based on the subject’s configured compatibility mode.

Subject-Level Compatibility

Each subject (typically a topic’s key or value) can define its own compatibility mode. This allows different event streams to evolve independently.

Examples:

customer-events  → FULL_TRANSITIVE
debug-events     → NONE
analytics-events → BACKWARD

This flexibility allows for strict guardrails on critical topics and a relaxed approach elsewhere.

Schema Version Retention Policies

Schema registries store all schema versions as they evolve, providing historical visibility for auditability, debugging, and rollback. Retention does not affect compatibility checks, but it determines how far back teams can inspect or restore schema history.

Different schema registry implementations offer different retention controls. For example, Confluent Schema Registry supports:

• Soft deletion (versions are hidden but still retained for compatibility checks)
• Permanent deletion (versions are removed entirely when required)
• Storage-level retention (e.g., Kafka topic retention on the _schemas topic)

Other registries may use different mechanisms, such as artifact-based retention (Apicurio) or simplified version storage (AWS Glue).

General guidance:

• Use high retention (i.e., keep schema versions) for long-lived data pipelines, production topics, analytics workloads, and any environment where auditability or rollback is important.
• Use low retention (deleting older schema versions) only for development or ephemeral topics where schema history is not important.

High retention ensures long-term observability and prevents compatibility issues caused by missing historical versions.

Additional Validation Rules

Beyond compatibility checks, schema registries also enforce structural validation rules that ensure each schema is internally correct. These checks apply even when compatibility is satisfied, protecting the system from registering schemas that are malformed, incomplete, or violate registry-specific constraints.

Typical validation rules include:

• Ensuring correct use of logical types
• Rejecting invalid or ambiguous union definitions
• Enforcing namespace and type consistency
• Validating that referenced types exist and are resolvable

Compatibility governs how new schemas relate to older versions, while validation governs whether the schema itself is structurally safe and adheres to the registry’s rules.

How to Choose the Right Compatibility Mode

Choosing a compatibility mode is not only about what each mode ensures, but also when each mode is appropriate. The following guidance helps align your schema strategy with real deployment patterns.

Backward Compatibility

Use when producers evolve faster than consumers, such as in systems with legacy services or long-running batch jobs.

Why it matters

Older consumers expect certain fields and data types. Backward compatibility guarantees that they can still deserialize the new schema.

Safe schema changes

• Adding optional fields with defaults
• Adding nullable fields
• Adding new enum symbols

Breaking changes

• Removing required fields
• Changing field types
• Renaming fields without aliases

Forward Compatibility

Use when new consumers deploy before producers, and older schemas will continue being produced.

Why it matters

New consumers must remain able to read older schema versions until producers catch up.

Safe schema changes

• Removing fields unused by consumers
• Making fields optional
• Adding default values

Breaking changes

• Removing fields still required by consumers
• Changing field types
• Making optional fields required

Full Compatibility

Use when producers and consumers may update in any order.

Why it matters

In distributed environments with multiple teams, rollout order is often unpredictable. New schemas must be compatible with old consumers, and old schemas must work with new consumers.

Safe schema changes

• Adding optional fields with defaults
• Adding nullable fields
• Adding enum values
• Reordering fields (Avro only)

Breaking changes

• Removing required fields
• Changing field types
• Making optional fields required
• Removing default values

For highly critical topics, combine Full with Transitive scope to enforce safety across the entire schema history.

None (No Compatibility Enforcement)

Use only for early development or non-critical topics.

Why it matters

This mode allows any schema changes, with no protection. Consumers depending on older schemas can break immediately.

Safe schema changes

• All changes are allowed.

Breaking changes

• All changes can break consumers.

This mode is intentionally unsafe and should never be used in production.

Final Thoughts

Schema compatibility is more than a safety mechanism — it is the foundation of long-term stability in event-driven systems. By understanding compatibility modes, selecting the right scope, and applying the appropriate configuration, teams can confidently evolve schemas without breaking consumers. Whether you optimize flexibility, strict safety, or long-term data retention, Kafka’s Schema Registry offers the tools needed to balance innovation with reliability. Thoughtful compatibility choices today prevent costly downstream failures tomorrow.