Conducting User Testing

Home / Conducting User Testing

User testing is often an afterthought, considered only after the design and development of an app. Our team at Savvy Apps believes in evaluating and validating your app from the outset and throughout the creation process. This allows us to iterate on what we learn and make sure your app appeals to users and meets their expectations before its launch. Without user testing, there’s no way to know if you’re on the right track until it’s too late.

As there’s no one-size-fits-all approach to user testing, we created this guide to teach you the fundamentals and strategies that will help you create your own testing plan specific to your business needs. We examine everything from defining your user goals, choosing between remote and in-person testing, recruiting your participants, preparing your testing environment and materials, to determining the type of data to collect.

Regardless of whether you use this resource to create your own in-depth user testing approach or contact us for help, by the time you finish reading you’ll understand each of the core user testing steps, the best tools to use, and the special considerations you need to get started.

Define User Testing Goals For Your App

The first step in user testing is deciding exactly what it is you want to get out of your testing. Your goals determine everything from your testing methods to the exact environment, tasks, and questions you ask your participants. This impact makes it essential to determine your goals before going any further.

Your user testing goals will change depending on where your app is in its lifecycle. For example, if you plan on conducting user testing before your app’s development phase, your goals will be more focused on exploration and discovery. That is, finding out what your target users need, what similar apps/products they already use, and what pain points they need help with. During this phase you won’t have a functioning version of your app to test with, but you don’t need one to achieve your user testing goals. Interviews with users, focus group discussions, and competitive analysis work well to get at the heart of your target users’ wants and needs.

User testing goals during development tend to lean more towards validation and evaluating. That is, finding out if users understand and enjoy your app, and whether the features meet their needs. For this, leverage early versions of your app, prototypes, and mockups. We prefer card sorting and concept testing to zero in on how your users understand and interact with your app’s hierarchy, structure, and feature set.

Testing doesn’t stop once your app is live in app stores. It’s important to continue testing to make sure you didn’t miss any opportunities with your users and to keep your app aligned with evolving user expectations. Your goals in the post-launch phase will focus on measurement and further validation. That is, how your app is performing, opportunities for optimization, and whether your user experience appeals to users. For this, give your participants your actual app and try A/B testing to narrow in on what resonates best with your users.

Once you establish your user testing goals, you need to go a step further and determine clear objectives for individual tests. Try to hone in on the most important insights you want to gather rather than trying to test everything at once. Remember, you’re not looking for generic feedback, you’re looking to uncover specific problems. When you try to discover too much at once, you run the risk of conducting a long and unwieldy test that fatigues your testers. It’s also easier to confuse your results and next steps when you try to tackle too much.

Your main goals and specific test objectives come together to form the foundation of your user testing strategy. For example, consider how those elements impacted our user testing for Share Our Strength’s healthy eating app, Cooking Matters.

Since we conducted our user test later on in the development of Cooking Matters, we used a stable, pre-launch version of the app. Our main goals focused on evaluating the intuitiveness of the app’s navigation and core features. We wanted to observe users’ natural interactions with Cooking Matters to identify any red flags before launch. To make sure we established clear objectives for this test, we outlined a few key questions to answer at the end:

  • Is the navigation intuitive?
  • Can users easily find recipes?
  • Can users easily add ingredients to their shopping list?
  • Can users schedule and add meals to their meal plan?

By answering these questions and sticking to our main goal throughout the testing process, we discovered a few easy-to-fix issues that we likely would not have noticed before launch.

Choose Between In-Person and Remote Testing

User testing can occur either in person or remotely. For both types of testing, you will want to observe participants using your product and uncover any usability issues. Where in-person and remote testing differ is in their pros and cons.

With in-person testing, you get the opportunity to establish a real connection with your participants and see their reactions from up close. It’s much easier to read someone’s body language in person than it is over a video conference. In-person testing also makes it easier to use a participant’s body language cues to gauge when to ask questions for clarification or dig for more feedback. These in-person interactions come with a cost, however. They often take more time because they involve traveling to a location, setting up a testing environment, and being present for each participant session. You may also incur a cost if you need to rent physical space for the testing.

Remote testing works well for unmoderated sessions (discussed below), as well as when budgets are more limited. This type of testing is also ideal for tighter timeframes as there’s no need for travel or testing facilities, and because there is less pressure on test moderators. The major downsides, however, are that you can’t interpret body language and it is more difficult to follow up or do a deeper dive into your participants’ feedback.

Decide Between Moderated and Unmoderated Remote Testing

Say you decide to go the remote testing route. Then you have one more big decision to make — whether you want to conduct moderated or unmoderated sessions.

Moderated sessions often offer more insights because you have the ability to ask questions in real time. If a user gets stuck on a particular task, you can redirect them or get more information about what may be causing confusion. You also have the option to ask more questions as they arise. For example, someone might say “I wasn’t expecting that” and leave it at that. In a moderated session, you could follow up by saying “What were you expecting?” or “Why were you expecting that instead of what you saw?” to further investigate the user’s experience.

Moderated remote testing comes with a few downsides. Like in-person testing, there’s a higher burden on time and effort as you need to schedule and be present for each session. You also run the risk of receiving less than candid results. Some participants may feel more pressure with someone observing and asking them questions than if they were interacting with your testing materials on their own.

While you don’t have to worry about scheduling, moderating, and pressuring participants with unmoderated, remote testing, you lose the ability to ask follow-up questions and redirect participants as needed. You also run the risk of not being there to troubleshoot tech issues or explain tasks to confused participants. These scenarios could result in testing sessions that provide less meaningful insights.

Unmoderated sessions do, however, give you the added benefit of being able to test your app in a unique context or environment. This is important for apps used under specific circumstances. For example, we asked our test participants to use the on-demand ride service, Sprynt, in the most natural context — when requesting a ride. It would have been impossible (or at least, very awkward) to try to simulate similar circumstances with a remote moderator.

Select and Recruit Your Participants

If you recruit the wrong participants, or if you don’t get enough participants, your user testing will not generate the results you need. To find the right participants for your user test, you first need to identify your app’s target audience. We often dive into recruiting participants during the discovery and user research process.

Behavior Versus Demographics

It’s more important to recruit participants based on their behavior rather than demographics. In choosing your participants, select for behavior more than age, sex, income, location, education, or marital status. The best way to do so is by finding people who are already active in similar apps. If you don’t have clear competitors, look for people who do activities similar to your app’s feature set. These participants are more like your target user base in that they likely have some prior experience and interest in what your app is trying to achieve. This is often more helpful than a participant who falls into a specific gender or age group.

That doesn’t mean you should dismiss demographics altogether. Your user research should indicate the common characteristics of your target users. Selecting participants who match those demographics will help you arrive at more useful insights. How picky you should be in matching those characteristics, though, depends on your target users and testing goals. You will want participants who match a very specific demographic if you have a niche product or user base. This is also important for complicated apps, apps that need a certain level of prior knowledge or experience, or apps with direct competitors. Demographics aren’t as important if your app appeals to a wide variety of users. This is also true for apps that anyone can use with minimal instructions or experience, and apps that you can test in 15 minutes or less.

There are several other considerations to keep in mind when selecting the right participants for your user testing. Before reaching out, be sure to determine how technically proficient your participants should be and whether they need access to a specific platform (Android, iOS, or web). Also stay away from recruiting people you know or contacts from social media to keep your user testing results as unbiased and accurate as possible.

How Many Participants Do You Need?

The number of participants you need for your test also depends on your testing goals. You should recruit 5-7 participants to test usability problems. While the Nielsen Norman Group asserts that you only need five participants to uncover 85% of a product’s issues, we prefer to schedule seven participants per test. The two extra participants account for no-shows.

Tests that investigate trends and opinions require a larger sample size to gather enough quantitative findings. We segment those users into groups of 5-7 participants based on different characteristics. You might separate them based on their knowledge related to the topic of your product or experience with your existing product. For example, when testing trends for BoatUS, we segmented a large pool of test participants into two groups based on their experience with the existing app. One group of participants had no prior experience with the BoatUS app, while the other had used the app before. We then compared results to see if that prior experience made a difference in how the participants used the app and how it made them feel.

Incentivizing Participation

Once you identify the 5-7 (or more) participants you want based on their behaviors and demographics, you need to get them on board. For in-person testing, we generally pull from our existing users. If you don’t have existing users, you can lean on your network. We’ve found particular success by tapping into community organizations and professional associations interested in the problems our apps try to solve. To help entice candidates, we provide compensation like gift cards, but the incentive will depend on what appeals to potential recruits. We often use an incentive that’s valued at $15 up to $50. That’s up to your budget and how vital feedback is to you. We also try to be flexible about days, times, and the lengths of testing sessions to cater to our participants’ busy schedules.

Recruiting With Tools Like User Interviews

There are several effective and affordable services that can help you recruit users to participate in remote testing. We prefer User Interviews. This tool allows us to target specific demographics and pose questions to narrow down the pool of potentials. Each candidate provides a profile and completes a survey for you to review before deciding whether they make a good fit. You provide a description of your project, any requirements, and any compensation. Once you invite a candidate, it’s then up to you to provide more details and either run the testing session or provide a link to the remote test.

We leverage services like User Interviews when we do not have an existing pool of users to recruit from for testing. For example, we used User Interviews to set the demographics for ideal Jellies users, provided a screener survey to weed out candidates, and then reviewed the remaining potentials to find our user testing participants.

Recruiting participants takes 3-5 days. This is an acceptable amount of time, especially for the tool’s less-expensive price point. User Interviews costs $20 per participant, plus any incentive you want to provide. It can take a lot of time and resources to recruit testing participants on your own, and it’s expensive to use a firm that specializes in recruiting.

Still, there are downsides to using User Interviews. As with any outreach, you’re at the mercy of the information you’re provided. With User Interviews you have to rely on your survey to make sure the candidates match up with your target pool’s behaviors and demographics. You don’t speak with the participants before the sessions, so you have to trust that they were truthful in their responses.

Schedule Your User Testing Sessions

Scheduling every user session for every test can become a logistical nightmare depending on the number of tests you decide to run. We simplify the process by using a tool called Calendly to eliminate the hassle of scheduling each individual testing session.

Calendly eliminates the confusion and hassle of trying to nail down session dates and times for each participant. It prompts each candidate to select one of your available time slots. When one participant selects a certain date and time, Calendly removes that option for everyone else. This saves you time and headaches by removing the need to reschedule double-booked sessions. We find it especially helpful when dealing with large numbers of testers for moderated remote testing. If you’re conducting unmoderated testing via a tool like Lookback (more on that later), your participants need only run through the tasks at a time convenient to them.