Rebooting Product Usability Testing for Much Better Insights

rebootresearchContributed by Mitch Solomon

Usability Testing – Something Has Always Seemed Off…

Something has never seemed quite right to us about how product usability testing is done. No matter what is being tested, from software to cereal to screwdrivers, there is always a herd of elephants – in the room. With their rows of booths and test kitchens and cameras and glass, usability labs and focus group facilities are nothing like the places people actually use products. And everyone involved in the process knows it. Asking people to use a product at a specific time because that’s when the research team could get access to a facility (or arrive at the participant’s home or workplace) we know results in forced and unnatural behaviors that are divorced from the real reasons and context associated with real product usage. And the presence of the moderator or facilitator we know creates biases that get baked into the findings and virtually impossible to filter out. Still this is how we tested products because, well, it was how everyone tested products.

In Search of A Better Way

Leaving a facility in Boston on a raw October evening last year, after working with a client on the testing of a popular mapping app, a colleague and I were convinced there had to be a better way. We knew what we had just witnessed, with painstaking detail, was flawed. Back in the office, whiteboard markers in hand, we listed the problems we had identified with traditional approaches to usability testing.

  • Labs are very different from users’ real usage environments
  • Appointment-based / lab-based studies are disconnected from real context that drives useIts very hard to study repeat / long term product usage, which can be invaluable
  • Moderators / observers (especially the bad ones) can bias participant behavior
  • It’s hard to do iterative / agile testing due to costs (i.e. only get one crack at the participant)
  • Projects are slow and expensive, driven in large part by travel
  • It’s hard to get certain participants to come to facilities; millennials, seniors, B2B, other
  • Sensitive / personal products present real recruiting and data reliability challenges
  • It’s hard to accurately compare usage of new products to participants’ existing solution

Could we solve them all? We weren’t sure. Ultimately we decided to focus on developing a usability testing method that:

  • Put the user in the real usage environment, not in a lab, to learn about real world usage and context
  • Eliminated the moderator / human observer, to eliminate the bias that s/he can introduce
  • Accelerated the time-to-insight, so we could study more users across more market segments or geographies, more quickly

Mobile Was Definitely Going to Be A Big Part of the Solution

Being a mobile and digital-centric research firm, we felt intuitively that the smartphone held the answer. Portable, personal, ubiquitous, and with a built-in video camera, still camera, microphone, and keyboard we knew that all the tools were present for participants to share enormous details about product usage in the real world, in ways they never could before. But without a researcher present to continuously observe and direct participants over the course of the study, we would need very good, highly customizable mobile software that engaged participants and held their hands as they shared their usage experiences with us. Having already worked with a powerful yet highly intuitive smartphone research app called Over The Shoulder for a variety of other study types – including customer needs assessment, customer journey mapping, and more – we believed we had the tool that could address our top three problems, and possibly others as well.

After considerable trial and error, we confirmed that the Over The Shoulder smartphone qualitative research app would allow us to design studies that could enable user to use any product out in the real world, without a researcher present, and share their experiences with us in a wide variety of media appropriate to what we wanted to know. And we knew that we could get to many more participants much more quickly than we ever could with traditional in-home or lab-based methods. After working out the kinks, we completed our first highly successful study in the home coffee brewing market. Here’s how it worked.

Coffee, Coffee Everywhere…

First, we recruited 20 participants in three different market segments and shipped the product to them in packaging that simulated the final packaging for the product. Next, these research participants, who had been carefully screened and recruited, downloaded the Over The Shoulder research app onto their phones, which provided custom-built assignments and instructions for them over the course of a two week period. During the first week, they made coffee normally using their traditional non-espresso coffee maker, and during the second week they were instructed to switch to the client’s new product. Each morning at their specified wake-up time (and in the evening for some participants as well), they were reminded by the app to engage with study and complete specific tasks and assignments, and then share their experiences with us via video, photos, and open-ended text responses. Among these assignments were:

  • Show us how you make a pot of coffee using your existing machine, and tell us about what you’re doing and why.
  • What are the joys and frustrations associated with the use of your current machine?
  • Show us what is involved in cleaning-up and how you feel about it?
  • Please get the box with your new machine in it, and have someone record you opening it and setting it up.
  • Please make a pot of coffee using your new machine, and tell us how the experience compared to using your own machine?
  • Tell us why you’re making coffee now?

..and much more.

Over the course of the two week study, we were able to see people make coffee over 600 times using their existing coffee maker and the client’s product. Every single one was made in the user’s own usage environment, with no bias-inducing observer present. And while the study focused on natural usage habits, a range of assignments ensured that each participant tested out every feature of the product by the conclusion of the study. We were able to see challenges that persisted throughout repeated usage, as well as challenges that users were able to resolve on their own, sometimes through creative adaptations that we never would have anticipated. We were able to see the interplay between multiple users of the product at home, many of whom would not have participated in a study that required participants to come to a lab. We were able to see coffee being made at all different times of the day, including late at night and at times not typical for a given participant, that an appointment-based home ethnography or lab-based study would have missed. And we were able to do all of this, and more, over the course of two short weeks of field work, with no travel time or expenses for the client, the researchers or even the participants.

A Huge Success That Became A Standard Part of The Client’s Process

All of the parties involved in this project were astounded by the insights that were obtained, and mobile-based product usage studies have since become a staple in this client’s product development process. In fact, select videos and other data have periodically been requested by the company’s CMO to illustrate the organization’s innovative approach to customer-centric new product development, something we are very excited to have played a material role in facilitating.

To sum up, mobile-based product usage testing has some enormous benefits that every company, no matter what they make and who they sell it to, should be using. And while traditional studies using labs and in-home / workplace observation certainly have their place in the product testing toolkit (a full exploration of this topic is beyond the scope of this post), we strongly believe that mobile-based studies deserve to be a part of nearly every product development process. Best of all, the fact that most companies have yet to discover the amazing power of mobile-based product usage testing makes it a real competitive weapon for those who make the decision to use it. Now ask yourself: why shouldn’t your company be one of them?

Feature image from Flickr Commons


mitch-solomon-photo-standing-wood-background-t-shirtMitch Solomon is President of ThirdSlice Research, a Boston-area consultancy specializing in the use of novel mobile and digital to tools to provide clients with faster access to better customer insights in support of their new product development and innovation initiatives. ThirdSlice has assembled a dream-team of leading research industry professionals who are pioneers in nextgen research tools and methods, with decades of experience, and who represent the best-of-the-best in the research profession. By combining the best people, tools and methods, ThirdSlice is uniquely qualified to help clients generate big ideas and insights that can serve as the foundation for customer-centric product development and innovation.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s