Apache Kafka & FinOps: it’s complicated

Kafka can be expensive and your FinOps team might take a look. But the only person who can change the trajectory is the engineering leader.

You use Kafka in your company, great. Who pays the bill?

The platform team pays it. Finance challenges it.

An engineering leader described it that his Kafka bill had grown faster than usage for several quarters. Finance was asking sharper questions:

• why is this climbing so fast
• can we make it predictable
• is the spend producing value?

So they did the normal thing: they asked the platform team to fix it by cleaning stuff.

The platform team can tell where the waste is (if they have a solution like Conduktor Insights). What they can’t do is remove topics, that would be unsafe. They can see a dead topic or an over-provisioned partition count on another team’s resources. They can’t unilaterally delete it, that would be wrong and could cause business issues or interruption in the worse case.

The gap between who can see the cost and who can act on it is why Kafka spend can increase faster than expected.

You can’t tune your way out of an ownership problem, and ownership is not something a platform team can grant itself.

”How do we decrease cost?” is the wrong question

Cost initiative is about reducing numbers.

This is done under the assumption that someone exists whose job is to reduce it, with the authority and the information to do so. In most organizations, this is not one person.

The better question is who should own the cost, and what would it take for them to act on it.

The invoice always lands on the platform team (more on this later), but they need to make people accountable of their usage of Kafka. The answer is to know if the people positioned to act have both the data and the authority.

The three stages of cost ownership

• Stage 1: Visibility without authority: everything on the platform team: budget, tooling, vendor relationship, and the bill. Project teams use the platform freely, costs grow with adoption, and the platform team carries full accountability without a single lever on resources other teams created. This is almost how all platform are working today.
• Stage 2: Authority without visibility. The organization pushes responsibility onto project teams, like chargeback with a flat split of the bill. Teams now “own” cost. Do they? They receive numbers with no way to tell value from waste, and no tooling to act at the source.
• Stage 3: The partnership. Project teams pay in proportion to their actual usage (usage: to be defined, production, consumption, storage etc.), and they have the visibility and self-service tooling to act on it. The platform team brings the expertise: which patterns matter, where guardrails belong. This is the only configuration that reliably make sense at scale.

Why “we need chargeback” always fails

The first reaction at Stage 1 is to get into this initiative: “we need chargeback/FinOps”. It does not work like this.

Stage 3 depends on scaffolding that can take time to deploy and adopt: defensible cost attribution, per-team visibility, self-service tooling, guardrails that stop bad practices/new waste from being created.

This scaffolding is mandatory to get it right, if you don’t think about how to attribute cost (which ones):

• Project teams receive a chargeback bill they can neither understand nor influence
• ➡ they challenge the methodology (the attribution IS shaky)
• ➡ the initiative loses credibility, no body trust sit makes sense
• Months later the bill is still growing and the word “chargeback” is an internal private joke

You need cultural changes + behavioral changes + technical changes.

You enable responsibilities to shift.

This is change management

A dashboard nobody is accountable is useless. The hard work, the part leaders tend to underestimate, is rewiring who owns what and make sure it’s owned.

It helps to split these three responsibilities:

• Paying the bill. Platform team, at every stage.
• Managing the budget. Platform team at Stage 1, flat allocation at Stage 2, proportional per-project ownership at Stage 3.
• Identifying waste and acting on it. Platform-team expertise on what matters, project-team context and action on their own workloads.

The last one cuts across team boundaries and it competes with everything those project teams are being asked to ship.

It requires someone with the authority to say “this is now part of your job”. The platform team has visibility but no authority. The project teams have authority over their workloads but no visibility into cost. Only the engineering leader sits above both.

Diagnose first, avoid numbers

This is typically how we help our customers maturing their Kafka platform:

1. What stage are we actually at? Showback on paper but never reaches the teams to act on it is Stage 1.
2. What’s the next stage we can reach in two quarters? No cost sharing at all? Shared but not proportional, or attributed but unactionable?
3. Who owns each piece today? Vague aspirations toward “more accountability” produce nothing. Name the owners.
4. What supporting structure does the next stage require? Proportional attribution, self-service tooling, guardrails, a governance cadence. Whatever’s missing is the actual project.

Most platform owners running Kafka at scale, can see total spend but not what is driving it. Only a smaller group had visibility down to team and resource.

The leader’s call

Kafka costs feel expensive because everyone keeps solving them at the wrong level: the technical level.

• Engineers tune partitions.
• Platform teams build dashboards.
• Finance asks for a forecast.

It’s all of useful, but these are local initiatives only. The real solution is to fix the mismatch between who can see cost and who can change it. This is an org-design decision, and org-design decisions belong to leaders.

You don’t need to re-architecture things. You need:

• the proper scaffolding in place (software to get the useful insights that can move the needle)
• the right people in place
• the mandate

The platform team can’t fix Kafka costs alone. They were never positioned to. The teams creating the cost (the producers & consumers) can, once someone gives them both the visibility and the mandate.

This piece is adapted from Conduktor’s writing on Kafka cost ownership. For the underlying cost patterns this diagnosis tends to surface, see Conduktor’s Kafka cost optimization overview and the original article, Your Platform Team Can’t Fix Kafka Costs Alone.