Between “Reaction” and “Observation” — We Look, But Do We Truly See?

Over the past few years, the main stage for usability testing and user interviews has shifted almost entirely online. Remote research platforms are genuinely convenient — as long as you can recruit participants, you can run a session with ease. As a learning opportunity, there’s every reason to embrace and scale it.

But this time, I wanted to bring less experienced designers — and teammates who had never actually met a user face-to-face — to the best seat in the house. So we deliberately built a makeshift UX lab inside Tokyo Gas’s own office space, hauling in equipment ourselves. For me personally, it meant leading an on-site (in-person) usability evaluation for the first time in roughly four years, and it brought back that same overwhelming feeling I remembered from before.

We ran the study across two major projects: 16 participants total, roughly 80 minutes each, combining evaluative testing with depth interviews. What we captured went far beyond audio — subtle hand movements, shifting eye gaze, changes in facial expression, the nuance in someone’s voice. The resolution of first-hand data we gathered was on an entirely different order of magnitude, and it’s safe to say the team members in the room were just as overwhelmed by the sheer density of signal as I was.

The sessions were, by any measure, a design win: together with users, the team was able to confirm the product had reached a release-ready quality bar. But that’s not really what I want to write about here. What stayed with me was something that surfaced through the act of facing users and evaluating alongside them — a deeper question about how genuinely difficult “observation” actually is.

We Look, But We Don’t Really See: A Feed Full of “Reactions,” and a Missing “Question”

For this study, we set up a live channel on Slack and Teams connecting the moderators (myself and one other) with observers in real time — some watching from behind a partition on-site, others joining the stream remotely. The idea was that anyone could ask a question at any moment.

Beforehand, I asked everyone — on-site or remote — to do one thing: focus on the interview footage and the participant’s words, observe the facts, take notes, and if a question emerged from those notes, fire it off to Slack or Teams immediately, no need to overthink it.

During the sessions, the timeline filled up fast. But most of what came through wasn’t a question at all — it was commentary. “Looks like they’re stuck here.” “Ah, so that’s how they’re interpreting it.” Observations about what was happening, but rarely anything that pushed toward a deeper “why.” That gap turned out to be far more telling than I expected.

When people watch a session through a screen, with a chat window open and ready for quick comments, it’s easy to feel like you’re “participating.” But reacting to something and truly observing it are two fundamentally different acts.

If you’re genuinely immersed in a user’s cognition and behavior — really watching, not just looking — hypotheses form on their own. Why did they do that just now? What prior assumption is getting in their way? And it’s exactly those hypotheses that turn into sharp, specific requests to the moderator: “Can you probe deeper on this?”

Plenty of reactions, but almost no questions worth probing — this might just be an extension of a familiar office pattern: someone shares their screen, and the room quietly listens to one person talk. It may be evidence that even the people closest to the product — the ones who should be most invested — aren’t fully immersing themselves in the user’s world. The normalization of online meetings, and the ease of remote research, might be quietly stripping away something essential to good research: the discipline of holding your breath and losing yourself in someone else’s behavior.

Reconfirming the Weight of First-Hand Data at an Entirely Different Scale

This is exactly why building our own lab and insisting on being physically present with users mattered so much.

The subtle rise and fall in a user’s voice as they handle a prototype. A meaningful filler word. The intake of breath when they get stuck mid-sentence. Eyes wandering with hesitation as they say, “This is nice… but — .” And then that moment — “Oh, I get it!” — where a few more taps later, the air in the room visibly shifts as a sense of self-efficacy clicks into place. The resolution and texture of first-hand data you get from 80 dense, uninterrupted minutes is on a completely different level from anything a screen can deliver. It hit with the same weight it always did, back when on-site was the only option there was.

A Real Sense of “Team Building” Through Being Present With Users

This effort also happened to be Tokyo Gas’s first substantial on-site usability test.

From planning and building the lab, to mapping out hypotheses, designing the 80-minute session structure, running the moderation itself, and finally the analysis and reporting — every teammate who ran the full process together walked away with more than just a new method in their toolkit. Encountering the same first-hand data together, and sharing deep insight with the user genuinely at the center, is the kind of experience that will show up in how this team works going forward. Watching the organization’s research capability grow, and a real design culture start to take root through the process — that, for me as a design leader, was the biggest win of all.

Now I want to take the insight and energy this on-site study generated and put it straight into finishing the product. And I want to keep amplifying the business impact that design work at Tokyo Gas can have — stronger, and further, than before.

And I’d add one more thing: the more reps we put in, the more our ability to truly see — and to generate the questions worth asking — will grow with it.