Behavioral Prototyping with Voice AI

Last week, my classmates and I were tasked with building a Wizard-of-Oz prototype: one that functioned like an actual product, but actually did not work. The intention of this was to explore and test out ideas without being constrained by production costs and feasibility of building.

Designing a Wizard-of-Oz game

My group and I chose to build a voice AI prototype. We created a game called “Word World”, where the user would get to choose one out of four categories (Food, Movies, Countries, and Animals). They would then select a mode (Alphabetical or Single Letter), and based on their selection, they would have 5 seconds to come up with a word for the category they chose. For instance, if they picked Food and Alphabetical, they might say, “Apple, Banana, Cantaloupe, etc.”

The game included a training mode, in which we walked our users through how to play. It then went on to play the game with the users (giving them positive or negative sound effects based on their answer), and included a portion at the end when the user had the option to make the game harder or easier.

Testing a Wizard-of-Oz prototype

To test out the prototype, we put a phone in a black box, and connected that phone to our Wizard (Ryan) through Zoom. Then, we asked a super kind student who we found outside our classroom if he would like to test it out. I acted as a facilitator while our user tested out the game with Ryan (who he thought was actually programmed AI!). Han also acted as a Wizard and produced sound effects (a “ding” and an error sound) per answer our user gave. I can’t attach the video here because we don’t have consent for that, but here’s a picture of what our set up looked like.

The black box hid the phone which our Wizard was using to communicate with our user via Zoom.

We went into user testing to find these measures:

• The number of times our participant had to ask the facilitator (me) a question because something was unclear: 3
• Learnability of game: rated 5 on a scale of 1–5 (with 1 being difficult and 5 being easy)
• Enjoyment of game (likeliness of playing game again): rated 4 on a scale of 1–5 (with 1 being not at all and 5 being extremely likely)

Analysis

I learned quite a few things about behavioral prototyping from this.

1. Some of our questions were really confusing. Our user visibly looked confused and had to ask me to clarify how to answer some questions. For instance, we asked our user how easy the game was, which he found difficult to answer (he changed his mind and went back and forth). We could’ve asked him to pick a mode “easy/moderate/difficult” instead.
2. In terms of facilitating, in the beginning I didn’t clarify that the user could end the game at any point. So when our Wizard asked him “Would you like to play again”, our participant said “Yes” three times. I had to tell him that it was okay to end the game, which he promptly did after I explained that he was welcome to say “No.”
3. In the future, I would’ve liked to explore more ways to measure how much a user liked a game. I know we asked a question about if they would play it again at the end, but people lie sometimes. I wish we could’ve gotten more feedback on if the user found our AI to be too repetitive, etc.

Overall, behavioral prototyping, especially for voice, was super helpful because it allowed for us to explore different responses and use cases for how participants would interact with our game; I would be curious to see what another users experience might look like.