Your Best Data Engineer Is About to Become Your Worst Manager

Being the best pipeline builder on the team was never the same job as leading the people who build it.

Non-members: Click here to read

Photo by Austin Distel on Unsplash

A senior data engineer got promoted to manager last year. Six months in, she was still writing half the DAGs herself, reviewing every PR before her team could merge, and jumping into Slack to fix pipeline failures at 11 PM. Her team called her the best manager they’d had. Her director called her a bottleneck.

But what no one told her was that those were two different jobs.

This keeps happening in data teams, and it keeps getting mistaken for a good problem to have. Someone is excellent at building pipelines, modeling data, and tuning Spark jobs, so the company hands them a team. The assumption is that if you can design a bronze-to-gold medallion architecture, you can probably lead the four people building it. That assumption is wrong more often than anyone wants to admit.

Data engineering makes this worse than most technical fields, because the work is unusually visible. A broken pipeline shows up as a red X in Airflow at 3 AM. A schema change breaks a downstream Power BI report and the business team notices within the hour. Everyone can point to the artifact and say, this person fixed it, this person built it. Management does not work that way. Nobody sees the meeting where a manager talked two engineers out of quitting during a rough migration. Nobody puts that in a sprint report.

Why we promote the wrong skill

Look at how most data teams actually build their leadership pipeline. A junior engineer becomes a senior engineer. A senior engineer who ships good code becomes a lead. The lead who keeps shipping good code eventually becomes the manager. At every step, the promotion is based on the same skill: can this person write correct, performant code and design systems that don’t fall over.

Then one day the job changes completely. The manager is no longer responsible for whether the ADF pipeline runs on schedule. They’re responsible for whether the two engineers who disagree about partitioning strategy can work in the same room without it turning into a standoff. They’re responsible for explaining to a VP why the Databricks compute bill tripled, in language that doesn’t include the word “shuffle.” None of that was tested during the promotion.

This is the trap Laurence Peter described decades ago. People get promoted based on success in their current role until they land in a role that needs a completely different set of skills, and then they stop getting promoted, because now they’re bad at the job. In data engineering specifically, this means we take someone who is brilliant at optimizing a slow Delta Lake merge and put them in charge of a team’s morale, growth, and priorities, and then act surprised when attrition goes up.

The technical manager who never lets go

Here’s a pattern I’ve seen on more than one data team. The manager still owns the trickiest part of the pipeline personally. Maybe it’s the CDC ingestion from the legacy Oracle system or the identity resolution logic that nobody else fully understands. Every time something breaks there, it goes straight to the manager, because they’re the only one who’s touched that code in two years.

That feels like strength. It is actually a warning sign. If the manager is still the only person who can safely touch a critical piece of the pipeline, the team has not grown, and the manager has not done their job. The point of leading a data team is not to remain its best engineer. It’s to make sure the team doesn’t need one irreplaceable person at all.

I watched this play out at a mid-size company migrating from an on-prem Hadoop cluster to Databricks. The manager, a former principal engineer, wrote the trickiest part of the migration script himself instead of pairing it out to his team. The migration finished on time. Eighteen months later, he left for another company, and the team spent three weeks just understanding what his script did before they could safely modify it. His technical excellence had quietly created a single point of failure with his name on it.

What management actually looks like in this job

Photo by Yosep Surahman on Unsplash

Good data engineering managers spend their time on things that don’t show up in a commit history. They notice that one engineer has been assigned every messy, undocumented source system for the last three sprints and nobody asked if that was fair. They catch that the data quality checks nobody wanted to own are quietly not being run before they turn into a bad number on an executive dashboard. They sit between the analytics team asking for a new gold table by Friday, and the engineer who knows that table will break three others if it’s rushed, and they make a call that keeps both sides functional.

None of that gets a Jira ticket. All of it prevents the kind of failure that does get a Jira ticket, six weeks later, after everyone’s forgotten the original decision that caused it.

There’s also a real cost to skipping this. I’ve seen teams where the manager was hands-off but also untrained, and the result wasn’t autonomy; it was drift. Engineers used inconsistent naming conventions across four Synapse workspaces because no one was aligning the team on a standard. That’s not a technical failure. That’s a leadership failure wearing a technical costume.

The AI angle everyone’s using as an excuse

Photo by Immo Wegmann on Unsplash

Some leadership teams have started using AI-assisted status updates and automated pipeline monitoring as justification for flattening manager ratios, one manager for fifteen engineers instead of six. The logic is that if AI can summarize a standup and draft the sprint report, the manager has less to do, so they can manage more people.

This confuses two different kinds of work. AI can write the summary of why yesterday’s load failed. It cannot sit down with an engineer who’s burning out from being the only person who understands the CDC pipeline and figure out how to fix that before they quit. It cannot navigate the disagreement between two senior engineers who each think their approach to slowly changing dimensions is the right one. Cutting the administrative load doesn’t reduce the human load. If anything, more direct reports means less time for the part of the job that was never administrative to begin with.

What actually needs to change

Photo by Towfiqu barbhuiya on Unsplash

None of this means managers should be technically clueless. A manager who can’t tell the difference between a schema evolution problem and a genuine data quality bug will make bad calls and lose the team’s trust fast. Enough technical grounding to ask sharp questions and evaluate real tradeoffs is not optional.

But that’s different from requiring the manager to still be the strongest engineer in the room. Most data teams train engineers for years through conferences, certifications, mentorship, and deliberate practice on hard problems and then hand someone a team of people and assume leadership will just show up because they were good at SQL. It doesn’t work that way. It never has.

We keep asking whether the data engineering manager is technical enough. That’s the wrong question. The better one is whether we ever actually taught them how to manage at all.

I write about data engineering and the people problems that show up around it. ***Follow*** if that’s useful.