Digital Design Blog

User Testing: A Case for an Informal Approach

I’m not one for simplistic maxims. Early in my career, however, I became convinced of this one: “Test early and test often”

I don’t think I need to convince anyone of the value of user feedback. However, as I’m exposed to various projects, there seems to be a few myths about user testing that persist — even in the most sophisticated projects. These myths include:

• User testing costs too much
• User testing is too time consuming
• User testing will require too many resources (people)

If unchecked, these myths cause a project team to believe that user testing is an impractical “nice-to-have” and steer the project away from gathering a much needed “outsider” perspective in the name of saving money or time. Through exploring each of these myths, I-ve identified one myth that encompasses them all. I call it THE Myth. My goal in writing this post is to first, draw out and identify THE Myth and second, counter THE Myth and show how user input is much more accessible and viable than many of us realize.

THE Myth

When we hear the term “user testing” (or its sister term “usability testing”) too many of us are still stuck with dogmatic images of the high-tech usability lab or the market testing facility completely “pimped-out” with hidden cameras, expensive recording equipment, specialized tracking devices and the ubiquitous two-way mirror. What an intimidating mental picture! I can just see the project manager now, desperately trying to keep scope, budget, resources and timelines in check, and thinking: “Something this involved must require more time and resources and come with a hefty price tag!” However, our project manager is beginning to succumb to THE Myth. What is THE Myth? THE Myth is the assumption that all user testing is, by nature, this involved.

THE Counter

Let me introduce an idea that is not new to the world of user experience practitioners but nonetheless a potent, thought-provoking, conversation-starting proposition. Ready? Not all user testing requires the facilities or rigor of the formal lab/facility approach to produce useful and relevant feedback. While the formal approach has it place and definitely has its advantages, there is another type of testing that can be applied when a project team does not have the time or resources required for this approach. I call this type of testing informal user testing.Informal user testing — also known as “guerilla testing” is a highly practical approach which reduces the test to its core components. In the last few years, I have conducted user/usability testing both in formal and informal settings. In most cases, I have seen no significant differences in the quality or quantity of user feedback but enormous differences in the cost, time, and resources required. Let’s break it down.

Cost

Recruiting Participants
The formal approach to user testing hires a third-party to recruit and screen potential participants. This method is costly as third- party providers charge for their efforts with both those who participate and those who are screened out. The informal approach eliminates the third party and employs a loose recruiting strategy — often pulling from the personal network of the facilitator. The assumption is that while greater rigor in recruiting is good, it should not decide whether or not the team will be conducting user testing. When results are viewed, the fact that these participants do not strictly match the target is taken into consideration.

Conducting the Test
The formal approach uses a third-party service/facility with systems and tools to both conduct and record each session. These spaces, technologies and services can be incredibly helpful but are not inexpensive. The formal approach also requires 2 to three individuals time (one to facilitate the test and 1-2 more to observe and take notes). The informal method trades the professional facility for the participants’ home, a conference room in the facilitator’s office, or even a table at the local library. The high-tech equipment is traded for a simple software program installed on the facilitator’s machine (such as Camtasia or Morae) that records participant interactions, facial expressions and complete audio of the entire session. The additional, note-taking personnel either view footage later and provide commentary or are eliminated altogether due to the facilitator’s ability to review footage for herself/himself.

Participant Incentive
In order for research facilities to remain competitive, high incentive rates are used. This means that the formal approach to participant incentive can be quite costly. Since the informal approach calls on the direct or indirect network of the facilitator (i.e. friends, colleagues and family) the incentive can be smaller with no negative impact on participation.

Time

The formal approach to user testing can be quite time consuming with all of the planning and coordination that needs to happen. The formal approach also requires everyone involved (especially the facilitator) be able to devote several days to testing.The informal approach is much more agile and can be done “at the drop of a hat” so to speak. Since the facilitator (one of the project team members) has the software installed on her/his machine, and has a network of potential participants who are willing to “get paid for their opinions”, recruiting is quick and testing can happen virtually anywhere at any time. Because results are recorded on the facilitator’s machine, a report can be turned around in as quick as 24 hours of testing.

Resources

In the formal approach, the ability to conduct a successful round of testing depends on all the roles being filled (i.e. facilitator, observers/note-takers). This requires the coordination of multiple schedules. Additionally, the time needed to execute the formal approach can remove the focus of some of the project team from their current project responsibilities or require an entirely new project member for the duration of the study.The informal approach eliminates the need for the coordination and sometimes even eliminates all roles except the facilitator. Additionally, since the turnaround time from recruiting to testing to reporting lasts a few days at most, members of a project team can weave these efforts into their current responsibilities with only minor impacts to overall effort.

Results

If you’re the facilitator, you’ll be impressed with how insightful your colleagues, friends and family will prove to be. As a project stakeholder, you’ll immediately see the benefits of user insights influencing the decision-making process. Ultimately you’ll see that all user testing does not have to be “that involved”.

Conclusion

I first loosened my adherence to THE Myth in favor of employing an informal approach a little over 2 years ago. Since then, I have not seen significant differences in the quality or quantity of user feedback but enormous differences in the cost, time, and resources required. The formal approach has its place and some clear benefits. However, if a project has no budget or time for the formal approach, instead of succumbing to THE Myth and scoping user testing out, consider taking an informal approach.