Streaming as a Service: From ksqlDB to Flink

When we first started building a self-service streaming platform at Riskified, the idea was simple: let product teams define their streaming logic in SQL, and we’ll take care of everything else — infra, serialization, deployment, and monitoring.

We began with ksqlDB. On paper, it looked like a great fit. SQL interface? Check. Kafka-native? Check. But once we started using it in production, the cracks started to show, especially around schema evolution, schema control, and resource isolation.

After a lot of trial, error, and lessons learned, we transitioned to Apache Flink, which gave us the flexibility and control we needed. More than just swapping out one tool for another, this shift changed how we think about building data platforms: the platform should adapt to the developer — not the other way around.

This post walks through our journey — what we built, what broke, and how we rebuilt it with AWS Managed Flink to deliver a better developer experience backed by strong schema and data contract enforcement.

The ksqlDB Experience: Lessons Learned

At first, our ksqlDB-based platform worked well for simple use cases, especially with JSON topics. Users would define their sources and sinks, and the backend would auto-generate the necessary queries. We even wrapped it in Kubernetes with all the bells and whistles — metrics, alerts, Schema Registry.

But as things scaled and teams started working with Avro and Protobuf, the problems piled up.

Schema Management was a Headache

ksqlDB enforces Confluent’s topic-naming strategy, which clashes with our internal conventions. To make things work, we had to duplicate schemas just to satisfy naming requirements. It felt hacky and added unnecessary friction.

Even worse, ksqlDB auto-generates sink schemas, removing our ability to explicitly define schemas for downstream consumers. As a result, even simple transformations — like filtering records — produce entirely new schemas, making schema evolution and consistency increasingly difficult to manage.

Surprise Transformations

• Enum fields automatically convert to strings, requiring additional transformations to maintain type integrity.
• Double values are downcast to float, leading to potential precision loss.
• All fields become optional, altering the expected data model and requiring extra validation.

Schema Evolution? Not Really

• ksqlDB does not support dynamic schema evolution, making iterative changes difficult.
• Adding new fields inside nested structures causes deserialization errors, breaking compatibility.
• The only way to apply schema updates is by manually tearing down and recreating queries, as ksqlDB locks in schemas at query creation — disrupting production pipelines.

These issues made ksqlDB unsustainable for production. We needed an alternative that offered better schema management, schema evolution, and operational stability.

Adopting Apache Flink

After recognizing that ksqlDB’s limitations made it unsustainable for our needs, we turned to Apache Flink. But this transition wasn’t just about switching technologies — it was about redefining our approach.

Rather than self-hosting Flink and managing its operational overhead, we chose AWS Managed Flink, which allowed us to focus on functionality rather than infrastructure. This shift enabled us to build a seamless user experience where teams could effortlessly define and deploy streaming jobs while we handled the complexities behind the scenes.

Platform Considerations

To uphold our “Streaming as a Service” vision, we developed a generic, Java-based Flink application using Flink’s Table API, enabling users to:

• Define SQL-based streaming jobs that process structured data with minimal friction.
• Leverage Avro schemas with explicit control over both source and sink schemas.
• Register Avro schemas as tables, allowing records to be processed using GenericRecord.
• Seamlessly integrate with our data catalog and data contracts, ensuring schema consistency and enforcing data integrity.

Execution & Isolation

Beyond schema management, another critical aspect of building a streaming-as-a-service platform is how jobs execute and scale. Here, the differences between ksqlDB and Flink became even more evident.

ksqlDB: Shared Execution, No Isolation

• Single cluster, shared resources — ksqlDB distributes queries as Kafka Streams tasks across nodes, but all queries within the cluster still share compute and memory resources, leading to potential resource contention and performance degradation as the workload scales.
• Lack of isolation — A misconfigured or resource-intensive query can degrade the performance of all other running jobs.
• Continuous execution model — Every persisted query runs continuously, regardless of whether data is actively flowing, leading to unnecessary compute costs.
• Scaling challenges — Scaling up a single query requires scaling the entire ksqlDB cluster (horizontally or vertically), making resource optimization difficult.

Flink: Per-Job Execution, Strong Isolation

• Dedicated execution per job — Each Flink job runs as an independent application with its own execution plan and resource allocation.
• Dynamic scaling — Flink applications scale dynamically based on workload demands rather than being constrained by a static cluster setup.
• Optimized execution — Flink’s query planner optimizes execution plans to reduce compute costs while maximizing efficiency.
• Improved fault tolerance — If a Flink job fails, it restarts independently using checkpoints without affecting other running applications.

This shift meant users didn’t have to worry about other streaming jobs affecting their performance. It also aligned better with our cost and performance goals. Instead of keeping an entire shared cluster running 24/7, Flink applications consume minimal resources during low traffic and scale automatically as demand increases.

Cost & Efficiency

One major benefit of moving to Flink was the ability to control execution costs more effectively:

• Resource utilization — Unlike ksqlDB, where queries run continuously, Flink dynamically scales jobs based on workload demand, avoiding unnecessary compute costs.
• Granular scaling — Flink scales individual jobs independently rather than requiring an entire cluster to scale, optimizing resource allocation.
• Isolation for stability — Each Flink job runs separately, preventing resource-intensive queries from affecting other jobs.

This execution model was a game-changer for us. Instead of operating a large, always-on shared cluster, we now run self-contained, efficient Flink applications that scale dynamically.

Developer Experience

With Flink as our new foundation, we focused on developer experience, ensuring that streaming jobs could be defined and deployed effortlessly.

CI/CD: Safe & Reliable Deployments

• Flink applications pull artifacts from S3 but only reload them on an update.
• We automate artifact deployments across environments.
• A workflow ensures safe, gradual rollouts by validating applications before updating them.

Query Validation: Catching Errors Before Deployment

One of the most significant pain points with ksqlDB was debugging failing queries at runtime. If a user submitted an invalid query due to syntax errors, referencing nonexistent fields, or using unsupported SQL functions, the system would fail only after deployment, causing unnecessary downtime and requiring a manual fix.

Additionally, failing queries could sometimes hang indefinitely, consuming resources and adding unnecessary costs. This led to noisy, failing jobs cluttering the system, impacting both stability and performance.

While ksqlDB provides stateless pull queries that can be used for basic validation, they do not support all operations used in stateful queries, limiting their effectiveness for comprehensive query validation.

We wanted a frictionless developer experience where users could catch issues before deploying their streaming job. This led us to implement a dry-run validation mechanism using Apache Calcite, the same SQL parser used internally by Flink’s Table API, ensuring that queries were syntactically and semantically correct before execution.

How It Works

Before a Flink application is deployed, the system:

1. Parses the SQL query to check for syntax errors.
2. Validates schema references to ensure all fields exist in the source schema.
3. Verifies SQL compatibility with Flink’s execution engine.
4. Surfaces errors immediately, preventing misconfigurations from making it to production.

This approach eliminates runtime failures, allowing users to fix issues early and confidently deploy their streaming applications.

Under the Hood: Apache Calcite

Since we exclusively use Avro for structured data, we needed to ensure that our validation fully understood Avro schemas, including:

• Primitive types (e.g., int, string, boolean).
• Logical types (e.g., timestamp-millis, decimal).
• Complex/nested structures (e.g., record, map, array).
• Union types (e.g., nullable fields).

To achieve this, we extended Apache Calcite’s validation by building a custom data type catalog reader that translates Avro schemas into structured table schemas. This enabled query validation without a Flink execution environment, allowing users to reference both root-level and nested fields naturally using dot notation (e.g., order.id), ensuring a seamless SQL experience.

import org.apache.avro.Schema
import org.apache.calcite.rel.`type`.{RelDataType, RelDataTypeFactory}
import org.apache.calcite.schema.impl.AbstractTable

class AvroTable(avroSchema: Schema) extends AbstractTable {
  override def getRowType(typeFactory: RelDataTypeFactory): RelDataType = {
    avroToCalciteSchema(avroSchema, typeFactory)
  }
}

We configured the SQL parser with explicit casing and quoting rules to ensure correct query validation.

SQL Configuration

We set the parser to be case-sensitive while preserving the original casing of quoted and unquoted identifiers. We used backticks (`) for field names to maintain consistency in referencing structured data.

val sqlConfig = SqlParser.Config.DEFAULT
  .withCaseSensitive(true)
  .withQuoting(Quoting.BACK_TICK)
  .withUnquotedCasing(Casing.UNCHANGED)
  .withQuotedCasing(Casing.UNCHANGED)

Query Parsing & Validation

We parsed and validated queries using this configuration without requiring a Flink execution environment. The validation step ensured all referenced fields existed in the Avro schema and complied with Flink SQL syntax.

val sqlParser = SqlParser.create(sqlQuery, sqlConfig)
val parsedQuery: SqlNode = sqlParser.parseQuery()
val validator = SqlValidatorUtil.newValidator(
  SqlStdOperatorTable.instance(),
  catalogReader,
  typeFactory,
  SqlValidator.Config.DEFAULT
)
validator.validate(parsedQuery)

Monitoring & Alerting

To provide users with complete visibility into their streaming applications, we integrated a comprehensive monitoring and alerting stack:

• Grafana Dashboard — We built a detailed application-level dashboard to track key performance indicators in real-time.
• Preconfigured Metrics — We monitor essential aspects such as:
• Application status & checkpoints — Ensuring job stability and fault tolerance.
• Throughput & consumption — Tracking event processing rates and data ingestion.
• Resource utilization — Monitoring CPU, memory, and disk for optimal performance.
• Lag monitoring — Detecting delays between data ingestion and processing.
• Automated Alerts — When jobs fail, experience high latency, or produce unexpected results, alerts are automatically routed to application owners.

Since AWS Managed Flink natively exposes metrics through CloudWatch, we exported these metrics to Prometheus to create a richer, application-centric Grafana dashboard.

In addition to visualization, we leveraged Prometheus alerting rules to proactively detect and escalate issues. These rules trigger alerts based on predefined thresholds for lag, error rates, resource exhaustion, and checkpoint failures, ensuring quick responses to anomalies before they impact production workloads.

Schema & Data Contracts

One of the biggest challenges we faced with ksqlDB was the lack of schema control. To address this, we implemented strict schema enforcement in Flink:

• Both source and sink topics must have a data contract to ensure schema consistency.
• Schemas are explicitly defined, preventing unexpected transformations.
• Flink applications use Avro natively, avoiding the serialization issues we encountered with ksqlDB.

We integrated our data contract framework into the UI to further enforce data quality standards. Users are presented with the dataset quality score when they select a Kafka topic. This score is calculated based on a weighted combination of several factors:

• Field-level documentation completeness.
• Defined quality assertions (e.g., required fields, valid value ranges).
• Freshness — how frequently the dataset is updated.
• Metadata completeness — ensuring key attributes like ownership and descriptions are well-defined.

Datasets must meet minimum quality thresholds to be eligible for streaming, ensuring that only well-defined and reliable data sources are used in production streaming jobs. This approach not only improves data consistency and reliability but also encourages teams to enrich their data contracts, fostering better data governance across the organization.

Key Takeaways

What We Learned from ksqlDB

• Schema management — While ksqlDB handles schema creation automatically, the lack of control led to unexpected changes, inconsistencies, and additional operational overhead.
• Schema evolution — ksqlDB’s strict limitations on evolving schemas made it impractical for real-world production use cases.

Why Flink Was the Right Choice

• Explicit schema control allows us to define both source and sink schemas.
• Full SQL support, including complex operations, windowing, and aggregations.
• Query validation with Apache Calcite ensures errors are caught before deployment.
• AWS Managed Flink allows us to focus on functionality rather than infrastructure.

Final Thoughts

This project taught us a lot — not just about tools but also about designing platforms that truly empower developers.

We went into this thinking ksqlDB would give us a quick win. And it did, for a while. But once we needed real schema control, better validation, and isolation between jobs, we found ourselves fighting the system more than working with it.

Switching to Flink wasn’t just about getting new features — it was about rethinking how we support teams working with streaming data. Now, developers can define SQL jobs with confidence, and we handle the rest: schema enforcement, validation, deployment, scaling, and monitoring.

In the end, what made this successful wasn’t just the tech stack — it was treating platform design as a product, focusing on user experience, and building trust in every layer.