mobile icon.png
focus icon.png

Personal Focus




lead designer

ideation &

prototyping lead

VUI Assistant

My student team was challenged to utilize AI in a new way to improve an existing experience. 

My team loves food, so we chose to improve the general restaurant experience though aiding in food discovery.


I led the ideation of our VUI Assistant and created the prototypes used to test the experience Wizard of Oz style. Our testing sought to validate our designs and reveal flaws and pain points with the experience.


This project taught me the intricacies of AI designs as well as helped me understand AI-interactive experiences.

project overview


Explore and understand the space within the context of our idea and build a foundation for our design process.

We have placed our design in a near and realistic future (5 years or so) in order to account for the current public unfamiliarity with VUI. One of our key assumptions about this future is that Alexa and Siri have only become more advanced and more popular, as well as the rise of other possible assistants not on the market yet. This means that the public has become VUI-familiar through use. In addition, we assumed that the technology for VUI and AI has advanced to reduce a lot of the errors that cause frustrations with VUIs (reducing the need to repeat oneself or speak loudly or slowly).

Secondary Research

In order to give our design a strong footing, we performed secondary research and found that adults aged 35-45 are the most likely to go out to eat and have the most disposable income.

We realize the importance of identifying and designing for the correct audience and used secondary research as a tool to understand who we should design for. With this in mind, this age group would be our primary user base and therefore is who we decided to design for.


By finding and interviewing users in our target user group, we were able to identify goals and concerns they might have both with going out to eat as well as their eating habits.

Because our research defined a specific user base for our solution, we decided to take a look at attaining qualitative data through interviews with this user base. We performed four interviews with 3 males and 1 female from this age group. Our goal for these interviews were three ideas: the ordering process, food interests, and new food discovery.

Service Blueprint

We used a service blueprint to map out how the dynamic in a restaurant might change and discover how the waitstaff and the VUI can cooperate with each other to create a better experience for customers. 

Since a VUI ordering experience can take some of the waitstaffs’ responsibility, like suggesting and submitting the order, we envision that the dynamic between the waitstaff and customers will be changed by our design.

research & understanding

! icon.png

Generate ideas about the details of the structure and functionality of our device.

Our VUI Assistant is called Loomy, and I will be referring to it as Loomy moving forward.


Our product is primarily a physical object that will be present on tables in restaurants. Therefore, our sketches and ideation were done at said platform level. This helped us better visualize our solution and allowed us to rapidly ideate and test.

Loomy is in the small box on the side of the table.

This simple prototype used a small box with a phone inside to mimic Loomy's interactions. One of my team members was on another phone, called in, pretending to be Loomy. This worked well enough to simulate VUI interactions, but we found a few problems in our testing, outlined later.


This is one side of our second prototype.

The point of this was to stimulate engagement with users, prompting them to talk with Loomy by initiating conversation with "Hey Loomy!" This worked well enough, but we found a digital prototype to have a few more affordances in relation to prompting users.

This is the second screen of our second prototype.

Our primary goal with this sketch was to show users when Loomy is listening to what they have to say. Once again, a digital prototype was able to share this feeling much easier.



! icon.png

Experience our prototype in action and identify points of improvement.

We were unsure about what the VUI device will look like during our initial ideation. Therefore, we decided to conduct a wizard of oz testing to answer our questions:

  • Should it be a device that sticks on the table, embedded in the table or a portable device that people can carry around in the restaurant? 

  • Does it need a screen or any other display components?

  • If a screen is necessary, what would it display?

Initially, we intended our participants to order food three times with different types of devices and rate each type based on ease of use, comfortableness, and intuitiveness, so we can compare them. However, when we tested with our first group of participants we discovered that this didn’t work out well. It doesn’t make sense to ask them to go through the same scenario three times. Therefore, we modified the testing set up after each group of participants based on their feedback.

Wizard of Oz Round 1

Discover initial pain points in the ordering process and with our prototype.

We put a phone in a cardboard box and had a team member on the other side of the phone to pretend to be the VUI. Additionally, I acted as the waitstaff aiding our VUI. Participants were in a group of two and were told to pretend to be customers at a restaurant. 

Group 1

For the first test, we used a plain cardboard box and the device was stuck on the table. Participants were clearly confused about the state of when Loomy is listening and when Loomy is inactive. They thought Loomy, like other smart speakers in the market, was always listening. They even asked us if they are able to remove the device from the table to avoid any extra listening. Also, we realized the importance of onboarding - since ou VUI is a new type of device, it is important for the waitstaff to onboard customers successfully.

Instead of leaving it stuck on the table, we made it portable and onboarded our customers with Loomy before getting to the table. If the participants agreed to use it, they held the device and placed it on the table themselves. 

Group 2

By implementing all these adjustments, our second group of participants seemed much more comfortable to talk to Loomy and have fewer concerns. Then we decided to test one more group of people with the same setup.

This group was not very talkative with Loomy, and we were unsure whether or not that was because of the testing of the personalities of the testers involved.

Group 3

For the third group, at first, we used the same process as we did for group 2. However, from their feedback, we realized that it didn’t make much sense to ask customers to hold the device and place it on the table themselves. We wanted this group to talk as much as possible to Loomy so we encouraged them to talk with Loomy as much as they could.


We discovered that all three groups of participants wanted Loomy to be more personable and descriptive.


Also, we finally decided to end up with a portable device, which will be placed on the table as the customers are seated. A screen is also necessary for our smart speaker to inform when it is listening, talking, or inactive. The screen would also be used to display both what the user says and what Loomy says because, in a restaurant setting, it can be crowded and the customer may not be able to hear Loomy clearly. We also realized another critical responsibility of the waitstaff, the importance of correct onboarding.

We wanted to take our feedback from our first round of testing into consideration and ideate solutions to our problems.

The waiter introducing diners to Loomy is one of the most crucial points in our process. We wanted to nail down exactly what was said and cut down on anything unimportant. In our first testing, the onboarding process was done on the fly, but for this test, a small script of talking points was created.


In regards to an active/listening/zero state, this feature was much more easily prototyped digitally than physically. We were manually switching paper mockups for our first testing, but we created this experience prototype in Keynote, allowing for simple animations to convey feelings. We introduced a green/white circle to indicate listening(white) and talking(green). The zero state was expressed by Loomy’s circle staying low on the bottom edge of the screen.

Experience Testing

We wanted to determine if users can identify Loomy’s zero, listening, and talking states, and compare whether or not Loomy is an improvement on the current dining experience.

This testing was done using Keynote as a quick and easy to use method of prototyping an interface with animation. Our most consistent feedback was that Loomy was not humanized to the point that it would naturally fill in the role of a waiter. We learned that there was also confusion identifying when Loomy was listening, which was one of our main goals.  We needed a better cue for when Loomy was listening and when it was going to speak. 

Our largest part of the feedback that had an impact on our final prototype was to make Loomy more human-like.  Some of the more particular quotes are below: 


“It needs to be more human-like and descriptive.”

“What about changing or customizing my order?”

“The ‘total balance’ makes me think I have to pay.”

“I think there should be a better cue when Loomy is listening.”

Our project ended with this last experience testing but moving forward our most important point of concern lay with the humanization of Loomy to make the experience as comfortable as possible.

! icon.png



Understand areas for improvement and lessons learned.

Through this process we learned about the unique challenges of designing for voice user interfaces, for example:

  • It is difficult to figure out how to onboard users about how to interact with a VUI

  • Conversational structures are infinitely complicated and figuring out how to map this out for a VUI is very time consuming

  • Mixing technologies (VUI and screens) can add functionality but also adds complexity.

There are a few things that I would have changed, however.

We were really focused on turning the human into a robot and that was evident within our presentation feedback.  We should have focused on using waitstaff to add the “humanization” aspect. For example, as one of our professors mentioned, we could have some of the restaurant staff recommend their favorite meal that they serve, etc. 

Another piece of feedback we got was to add some sort of “wait time.”  We thought this might help the “humanizing” of Loomy as well. For example, when you are waiting for someone to respond in Slack or a message, you will often see some sort of notification or some indication of wait time.  This also gives more visual cues to the diner when Loomy is listening or speaking.


! icon.png