I Paused a Podcast 23 Times. This Is What I Found.

I have a bad habit. Whenever I find a podcast episode that feels genuinely useful — not the kind where someone says ship fast and iterate for 45 minutes and calls it wisdom — I stop whatever I am doing and start taking notes like it is my board exam.

That is what happened with Lenny Rachitsky’s conversation with Cat Wu, PM at Anthropic working on Claude Code.

I am going to try to write down what I actually took away. Not a summary. Not timestamps. Just — what it felt like to watch someone who has clearly figured something out talk about her craft.

First, the thing nobody says out loud: speed is not about hustle.

Everyone in product says move fast. Most of them mean: work more hours, cut corners on thinking, and call the chaos agility.

What Cat was describing was different. She was talking about shortening the feedback loop — the distance between having an idea and knowing whether it was a good one. That is a structural thing, not a vibe.

The way she put it at Anthropic was almost embarrassingly simple: find the key user, find the primary problem, reduce your prompt to the minimum testable version, and ship something within a week.

“PRDs should be reserved for features that are genuinely ambiguous. For everything else, default to shipping and learning.” — Cat Wu, paraphrased

I have been on teams where we wrote 12-page PRDs for a button rename. I have also been on teams where we shipped the wrong thing beautifully because nobody slowed down enough to ask what does good actually look like here? Cat’s point is that most things do not need a PRD. They need a working prototype and three real users.

Product taste is the moat. And taste is terrifying to admit you have or don’t.

She said something that I have been thinking about for days: as writing code becomes cheap, the scarcest thing in product development is judgment. Knowing what to build. Knowing what the most delightful version of it looks like. Being able to say — this feels off — and actually being right.

The honest version of this is that taste cannot be templated. It shows up in the micro-decisions. Which interaction feels obvious. Which edge case deserves handling. Which feature is technically functional but quietly ruins the experience.

The uncomfortable part? Taste is developed by using products obsessively, by shipping things that fail, and — I think this is the part nobody wants to say — by caring enough about the person on the other side to feel it when something is not right.

“Demonstrating product taste is what the world is looking for right now.” — Cat Wu

I keep coming back to that line. Not because it is flattering to PMs. Because it implies that taste is demonstrable — you either show it in the work or you don’t

Wear every hat. Leave the ego at home. (This one stung a little.)

She talked about the best AI PMs being shape-shifters. Writing a spec, interrogating a model, reading a diff, running a user interview — sometimes in the same afternoon.

Engineering fluency helps. Not because you need to write the code, but because understanding what makes something hard leads to better prioritization and more honest conversations.

But here is the bit that stayed with me. She said the point is not technical skill. The point is adaptability combined with zero ego. People who insist on staying in their lane slow teams down. People who fill the gap — wherever the gap is — move products forward.

And then she said something about staying sane in constant chaos. Calmness under ambiguity as a superpower. Optimism that does not tip into denial. I wrote this is the whole job in my notes and underlined it twice.

Evals. That’s it. That’s the unlocking skill.

If I am being honest, I went into this episode expecting to take notes on roadmapping and stakeholder management. I did not expect to spend twenty minutes pausing the video because someone was explaining how to measure whether your AI product is actually getting better.

Cat’s argument is direct: LLMs are inherently general-purpose, which means they introduce significant ambiguity into every product decision. Without structure, teams default to the vibe check. Does this feel better? And feelings, as it turns out, do not scale.

“Evals act as a bridge, allowing PMs to concretely define what success looks like for a specific feature. Without them, teams are forced to rely on subjective vibe checks rather than rigorous data.” — Cat Wu

What an eval actually is, in its simplest form: a structured test that tells you whether a model or prompt change is making your product better — or quietly making it worse. The key word is quietly. That is the scary part. You can regress without knowing.

Ten great evals beat a hundred mediocre ones. Start there.

I was expecting her to talk about massive eval frameworks. Hundreds of test cases. Research infrastructure. She went the other way entirely.

“Creating even 10 great evals is sufficient to help a team quantify progress, identify where the model is failing, and measure the impact of improvements or regressions when updating prompts or models.” — Cat Wu

Ten. Ten well-chosen, genuinely representative eval cases. That is the starting point.

The word she kept using was curated. The best eval cases come from real user interactions — the edge cases your users actually hit, the queries that break things in production, the high-stakes inputs where failure is costly. Synthetic cases are fine for coverage. But real examples are gold.

She described five steps that I wrote down almost verbatim:

• Spend time with the model and identify where it fails — this is product intuition, not engineering
• Define success concretely. Not ‘better.’ Something specific: ‘includes price breakdown, no hallucinated product names’
• Curate from real interactions, not generated test cases
• Automate the boring cases; build human review into the ambiguous ones
• When something fails, trace back to root cause. Patch. Then add a new eval to prevent regression.

That last one hit different. When the model fails an eval, do not just log it. Close the loop. Use the failure to patch the weakness, then add a new case so that weakness never sneaks back in.

The part about evals as communication infrastructure. (This is the underrated bit.)

Most people think of evals as a testing tool. Cat reframed it entirely.

“Evals serve as a vital communication tool between product and research teams, transforming fuzzy goals into measurable technical outcomes.” — Cat Wu

When a research team asks what does good look like for this feature, your eval suite is the answer. Not a slide. Not a paragraph. A test that either passes or does not.

She described a small pod at Anthropic that works closely with the research team. Their evals are the primary communication mechanism. That structure — product and research aligned around a shared definition of success — is what allows them to move without spending half their meetings arguing about whether something is better.

There is a specific example that stuck with me. Claude Code makes a front-end change but fails to run the necessary tests afterward. The failure is logged. The eval isolates the failure mode — not the model is bad at coding, but the model skips test execution after UI changes. That specificity is what makes the fix tractable.

“When a model makes an error — such as making a front-end change but failing to run the necessary tests — I use evals to isolate the specific failure mode. This allows the team to update the system prompt or the harness to specifically patch that weakness.” — Cat Wu

The threshold question. The one that removes politics from shipping.

She mentioned a practice that I want to immediately steal: set an eval threshold before you ship.

Decide in advance what score a model or prompt needs to hit before it goes to production. If you enforce 85% pass rate across your eval suite, you create an objective gate. The shipping decision is no longer a debate about whether something feels ready. It is a comparison of numbers.

I have been in too many conversations where the question of whether something is good enough becomes a room full of people defending their intuitions at each other. An eval threshold does not eliminate judgment. It grounds it.

The PM who ships fast, has taste, and builds rigorous evals will consistently win.

Not because those three things sound impressive in a job description. Because they map to the three actual constraints in AI product development right now: time, judgment, and measurement.

Code is increasingly cheap. Inference is fast. The constraint is no longer execution. It is knowing what to build, building it quickly, and actually knowing whether it worked.

Cat did not say any of this dramatically. She said it like it was obvious. Like these were just the things you did when you were serious about the craft.

I think that is the thing I found most useful about the episode. Not any single framework. But the feeling of watching someone who gives a damn talk about giving a damn.

Start with ten eval cases. Raise the bar from there.