What Watching People Taught Me About Product Development

And what running coffee shops taught me about interpreting data.

Source: Coolaboola Coffee

I’ve led product teams and brought digital products to market across the US, Europe and Middle East and built product strategy for £2bn businesses, but so much of my product sense comes from owning a small chain of coffee bars.

The first Coolaboola Coffee site opened in 2006 and the following year made The Independent’s UK’s Top 50 Coffee Shops.

Within 48 hours, we knew we had found something people wanted. The right product, right price, right part of the city, the right moment just as third wave coffee was cresting, so we surfed it.

The Bet That Looked Right on Paper

Our second Coolaboola site was Central Station in Newcastle. On paper it was an obvious bet. The official footfall shared by the landlord was almost ten times our first site and the demographics looked great. I spent time watching the footfall patterns on site, but I didn’t want to acknowledge what my gut was telling me. Because the data is always correct, right? I placed a bet with our retained profits and our reputation.

What I failed to openly acknowledge was a nuance that no footfall figure will ever surface. In public transport there’s a critical difference between being a destination and being a point of departure. When people are moving through a space to catch a train, they think about time completely differently. They’re not in a treat-yourself mindset, they’re in an afraid of missing the connection mindset. We weren’t selling coffee to people arriving somewhere, we were selling it to people leaving, and they didn’t want to risk a delay by ordering a coffee at that exact location.

I can’t blame anyone else for that (although I do still question the validity of those footfall figures). It was my business, my call and my read of the data and I got them wrong.

Within two weeks of opening I knew we weren’t going to hit product market fit at that location. What we could do was make enough to cover costs, ensure the site washed its own face, and move on to site number three. Which is what we did.

There’s a real difference between that (figuring out how to keep the lights on after a bet doesn’t land) and running a post-mortem on a 2.03.17 app release. Both are legitimate forms of learning, but only one requires you to open up again at 06:00 the next morning regardless of how you feel about it. I’m not criticising solid product work here, but it’s an observation about what gets burned in when the consequences are that real.

That experience taught me something I’ve never unlearned. Data tells you what happened, but it rarely tells you ‘why’…and the ‘why’ is almost always a person doing something the system didn’t expect.

When the Numbers Were Right but the Story Was Wrong

Years later I was working on a customer-facing retail tech product. The data was telling a strange story; user adoption varied wildly across shop locations with some performing significantly better than others at peak times. The variance was too wide for me to ignore but the numbers across multiple sources all tallied up. I went to one of the locations, ordered some food and watched what unfolded. Before I left I had a conversation with staff about the new tech solution; “it’s a nightmare…”, someone told me within minutes.

The nightmare wasn’t the technology solution though, it was the operations; they weren’t setup to succeed with the additional order throughput so staff had found a way to disable the system during their busiest periods because it was generating additional sales and therefore putting more pressure on the team, not less.

The data had been accurate, but the story it was telling was completely wrong.

Source: Coolaboola Coffee

The Photocopy of a Photocopy

A couple of years back I worked with an early-stage startup, who had an excellent customer service agent who read every review, triaged every support ticket and passed summaries up to the leadership team. They weren’t ignoring their users, they just had made assumptions on what the users actually needed.

Every human handoff degraded the signal; specific complaints became general themes and emotional language got edited out. By the time user feedback reached a product decision it had been summarised twice, paraphrased once and was competing with fourteen other priorities. They were reading a photocopy of a photocopy.

We fixed it by connecting the support system and app store reviews into a single pipeline, and set up ML tooling to detect patterns across hundreds of data points with the original language preserved. Suddenly we could see not just what was being complained about but how often, in what language and alongside what other issues. The data was always there, it was the signal that was dying in translation.

Sometimes the problem isn’t that you’re not watching people closely enough, it’s that you’ve got systems that don’t let good signal survive the journey to a decision. But that can be fixed once you understand what gets lost without it.

Sometimes You Have to Leave the Dashboard

I didn’t learn these instincts at a product conference or from a framework. I learned them standing in a train station in 2007, watching people walk past something I’d built for them and realising that the numbers had not lied to me. I had simply expected them to answer a question they could not answer on their own.

The tools available to product managers are becoming more powerful, and we are all changing how we work. We can build and test products faster than ever, but the judgement required to understand what the data is really showing us has not changed. If anything, moving faster makes that judgement even more important.

When I see an unexplained variation in the data, I now resist creating an explanation from the dashboard alone. I go to the environment where the behaviour is happening, watch how people interact with the product and speak to those closest to the work. I then compare what I observe with the assumptions behind the numbers.

The goal is not to replace data with anecdotes or allow one conversation to outweigh a broader pattern. It is to uncover the context needed to interpret the data correctly.

Sometimes that context is a customer worried about missing a train. Sometimes it is a member of staff disabling a system because the operation cannot handle the demand it creates. The dashboard can show you that something is happening, but it cannot always explain why.

Before you draw too many conclusions from the data, find a way to watch what is actually happening. The numbers will still be there when you return, but you will have a deeper and more useful understanding of what they mean.

Many thanks to Tremis Skeete, Executive Editor of Product Coalition, for his invaluable contributions to this story. A special shout-out to founder Jay Stansell for creating an environment that enhances product management education.