Table of Contents

Why we have this Process

Usability is a major factor in the success of any product. User testing is a technique for evaluating the usability of a product or prototype and discover design issues. It creates a deep understanding of the user and usually it is experienced as eye-opening to all who observe a user test.  At a basic level, a user test usually involves a facilitator asking a user to complete a series of tasks while observing the interaction, noting any problems the user encounters, errors made, and sometimes the amount of time it takes to complete a task. 

Context and Scope

User testing can be performed on anything from early stages designs up until production systems. It can be helpful to start by performing user tests with paper prototypes and continuing to test more interactive clickable prototypes as a product iterated upon.

The user test findings are fed back into the design process in order to improve the prototype or product.

User testing is usually combined with some User Interviewing in the form of some questions before the user test. Combining user testing with questions allows us to also answer the question if the user need is big enough to motivate the user to start the task in the first place. 

User testing is planned using the User Testing Backlog

When should the process be performed?

User testing is an ongoing process with test always planned every two weeks. In this way we make user testing a habit. We test the top item from the User testing Backlog, or if there is no we fall back to testing the (core workflow) on the production web app. 

As soon the design of an Epic is done and internally reviewed, although multiple epics can be grouped into one user test for efficiency. (Where Epic is defined as: A big version of a user story, that groups a set of user stories, that together achieve a user goal that has actual business value.  Good examples:"Manage Todo's", "Create Timesheet for hours worked this Week", "Email Users Updates in the Wiki", "Buy Product", "Browse Articles", "Manage Profile", "Hello World Username". "Browse my music library and lay music" is Spotify and is too big.)

Who performs the process

Who performs the process: UX Designer
Who monitors if the process is done (correctly): Head of Product
Who participates in the process: PM, sample of developer(s)/testers(s)/other colleague(s) (max 3). 

Process steps

  1. Prepare the test goal, plan test days,  tools, using this template Prep: Overview & Participants . Plan User Test days (recommendation: Fixed day(s) every 1 or 2 sprints)
  2. Determine Test hypothesis, i.e. what Epics to test and/or what to interview users about, using this template: Prep: Hypotheses To Test (tip: Start a Topics to test backlog)
  3. Recruit testers, i.e. mail your Customer Panel or customers (see FAQ below for more tips).
  4. Schedule tests with the users (90 minutes recommended. Offer them timeslots via www.calendly.com )
  5. Prepare a the User Test, see section "Prepare User test" below.
  6. Run a User test, see section "Run a User test" below.
  7. Summarize and share your findings with the team, using this template: After: Test Round Conclusion Template


Please note the many templates linked in the process above. You can also find them in the menu to the left, as subpages of this page.


Quality criteria

With every user test you should also be looking for opportunities to remove friction in the process (things that hold the user back from completing the task or make it harder) and for clues on how to encourage the user to use the feature more (i.e. by adding a trigger or explaining the benefits better). This can be achieved by adding interview questions before and during the user test, i.e.: How often do you travel with friends or family by train? Have you ever booked online train tickets? Let's say you want to travel on Friday evening with your family of 4 to city X. How would you proceed? Why do you prefer online booking or phone booking? In what case would you not use online booking? 

FAQ

Isn't User Testing too much work?

Developing features that nobody uses because the user finds them too hard to understand is the real waste of time. User testing is an essential part of the UX designer's role. Designs need to be tested just like software, and finding issues in this stage saves a lot of development work. To convince the PM or CEO that time and money for User testing is needed is relatively easy: Invite them for the first user test. 

How to find testers?
Email your customers, your network, your Facebook or use a recruitment service like testingtime.com.  

How long does a User test take?
I recommend 90 minutes for concentration reasons, with a coffee break halfway. Also, see the example agenda.

How many people to invite per user test round?
I recommend 4 users per round, and to do them in one day. 2 for very early stage designs and your first round. Start with a dry run using i.e. the secretary/colleague

How to reward testers?
A 20 Euro/Dollar amazon.de voucher is common, up to 50 Euro/Dollar for hard to get audiences.

How do we show the app to the user and record the screen and audio?

For web apps

For mobile apps


Where to do the testing? In office or Remote?

Related Tools

https://www.confirmkit.com/  - Create Interview scripts and organize findings

Further reading



User Testing Template


Prepare a User Test

General Preparation tasks before Test Day

Advice for Interviewer to keep in mind


Scenario specific Preparation tasks

Scenario 1 prep tasks


Scenario 1 prep tasks Tester Handouts

Using tester handouts is optional, but it often is more convenient to share some details like password than to have to say them out loud. 

Background info: The fictional case

You are an Agent who works at a small 3 person company in New York. You signed up for our service because it offers potential clients to you (you are paying a certain service fee). They told you in an email that you can use their service via the app or via an iPhone app and you decided to give the iPhone app a go. The login details that they emailed you are below. For this test please ignore spelling errors and the fact that many words are not translated into German yet. Also, ignore the menu at the bottom of the screen.  Remember to think out loud.

Login data

Username:

Password: 

Search parameters: 2M-3M , lower Manhattan, 27 bedrooms. 


Run a User Test

General Introduction Script

Welcome and Purpose

Thank you for agreeing to participate in this app/site evaluation. Today we are asking you to serve as an evaluator of this app/site and to complete a set of scenarios. Our goal is to see how easy or difficult you find the app/site to use. We will record your reactions and opinions; so, we may ask you to clarify statements that you make from time to time.

Test Facilitator’s Role

I am here to record your reactions and comments of the app/site you will view. Our other colleagues will act as observers helping me take notes and observe your interaction with the site. During this session I will not be able to answer any questions or offer any suggestions or hints. However if a question comes to mind please say it out loud. There may be times, however, when I will ask you to explain why you said or did something.

Test Participant’s Role

I will ask you to search for information in this app/site to learn if it works well for you. We will do this by giving you scenarios or tasks to complete in the app/site. As you will interact with the product I kindly but strongly request you to think aloud while you complete the tasks and scenarios. Basically you should be talking most of the time. You also will be asked a couple general questions before the before the testing session and a series of questions about your experience at the end of this session.

Things to Keep in Mind

Here are some things that you should know about your participation:

Do you have any questions before we begin?

Two alternative intro scripts can be found here.


Scenario Specific Script

Scenario 1 

The process to be used during the actual test when the tester has arrived. 

Agenda

  1. People Introduction Round (5 mins)
  2. General Instruction, see above (5 mins)
  3. User Interview Questions (20 mins)
  4. Scenario 1  (40 mins)
  5. User Interview Questions (15 mins)
  6. Ending (5 mins)

Total: 90 minutes

User Interview questions (Agenda point 3)

Tip: You have to rapid fire the user interview questions (and sometimes cut of the tester) to get through them in time. It's recommended to warn the tester about this beforehand!

  1. What company do you work for?
  2. How many employees work there?
  3. Do you use any similar software?
  4. (if yes) What device(s) do you use it on?
  5. How often do you do X?

Scenario 1 - Facilitator part (Agenda point 4)


Scenario 1 Instructions 

Introduction of what the user is going to do.


Task 1

PagesHomepage, Page X, Page Y. (all the pages/screens you prepared. Include links to all pages, or alternatively only to the first page if all other pages are clickable from the first page). 
StoryYou just installed the app and want to try it out. Or: You are a recurring user, and you have setup the system to import your google contacts and you flagged certain contacts as high prio. 
Instruction Try to login to the system. Please remember to think out loud. 
After task questions
  1. What would you do now since the password is not working?
  2. Was there anything in the system that confused you or took a bit longer to understand? Did the system behave as you expected?
Observations






Task 2

PagesLogin page, forgot password page, change password page, dashboard
Story
Instruction

The correct password is the one we gave you plus a 0 at the end. Log in to the system and tell us your first impression. If any questions or any concerns let us know. Do not yet Edit or Add something yet.

After task questions
  1. On the To-do screen, in the top right, there are some icons. What do you think they do?
  2. On the left menu, there are toggles labeled Business and Privat. What do you think they do?
  3. What would you click if you wanted to change your password?
Observations






Task 3

PagesHomepage
Story
Instruction


After task questions
Observations






User Interview questions (Agenda point 5)

It is common to also ask some user interview questions after the test, since A) some questions might be related to what the user just tested, B) You might not want to influence the user with certain questions before the user test. 


Ending (Agenda point 6)

After Tester left