Why we have this Process
Usability is a major factor in the success of any product. User testing is a technique for evaluating usability of a product or prototype and discover design issues. It creates deep understanding of the user and usually it is experienced as eye opening to all who observe a user test. At a basic level, a user test usually involves a facilitator asking a user to complete a series of tasks while observing the interaction, noting any problems the user encounters, errors made, and sometimes the amount of time it takes to complete a task.
Context and Scope
User testing can be performed on anything from early stages designs up until production systems. It can be helpful to start by performing user tests with paper prototypes and continuing to test more interactive clickable prototypes as a product iterated upon.
The user test findings are fed back into the design process in order to improve the prototype or product.
User testing is usually combined with some User Interviewing in the form of some questions before the user test. Combining user testing with questions allows us to also answer the question if the user need is big enough to motivate the user to start the task in the first place.
User testing is planned using the User Testing Backlog.
When should the process be performed?
User testing is an ongoing processes with test always planned every two weeks. In this way we make user testing a habit. We test the top item from the User testing Backlog, or if there is no we fall back to testing the (core workflow) on the production web app.
As soon the design of an Epic is done and internally reviewed, although multiple epics can be grouped into one user test for efficiency. (Where Epic is defined as: A big version of a user story, that groups a set of user stories, that together achieve a user goal that has actual business value. Good examples:"Manage Todo's", "Create Timesheet for hours worked this Week", "Email Users Updates in the Wiki", "Buy Product", "Browse Articles", "Manage Profile", "Hello World Username". "Browse my music library and lay music" is Spotify and is too big.)
Who performs the process
Who performs the process: UX Designer
Who monitors if process is done (correctly): Head of Product
Who participates in the process: PM, sample of developer(s)/testers(s)/other colleague(s) (max 3).
- Plan User Test days (recommendation: Fixed day(s) every 1 or 2 sprints)
- Determine Test goals, i.e. what Epics to test and/or what to interview users about. (tip: Start a Topics to test backlog)
- Recruit testers, i.e. mail your Customer Panel or customers (see FAQ below for more tips).
- Schedule tests with the users (90 minutes recommended. Offer them timeslots via www.calendly.com )
- Prepare a User Test (see template below)
- Run a User test (see below)
- Store your detailed findings on the wiki.
- Summarize and share your findings with the team.
- The participants represent real users.
- The participants do real tasks.
- The moderator does not steer the user by giving hints or answering questions.
- You observe and record what participants do and say.
- You analyze the data, diagnose the real problems, and recommend changes to fix those problems.
With every user test you should also be looking for opportunities to remove friction in the process (things that hold the user back from completing the task or make it harder) and for clues on how to encourage the user to use the feature more (i.e. by adding a trigger or explaining the benefits better). This can be achieved by adding interview questions before and during the user test, i.e.: How often do you travel with friends or family by train? Have you every booked online train tickets? Let's say you want to travel Friday evening with your family of 4 to city X. How would you proceed? Why do you prefer online booking or phone booking? In what case would you not use online booking?
Isn't User Testing too much work?
Developing features that nobody uses because the user finds them too hard to understand is the real waste of time. User testing is an essential part of the UX designer's role. Designs need to be tested just like software, and finding issues in this stage saves a lot of development work. To convince the PM or CEO that time and money for User testing is needed is relatively easy: Invite them for the first user test.
How to find testers?
Email your customers, your network, your facebook or use a recruitment service like testingtime.com.
How long does a User test take?
I recommend 90 minutes for concentration reasons, with a coffee break halfway. Also see the example agenda.
How many people to invite per user test round?
I recommend 4 users per round, and to do them in one day. 2 for very early stage designs and your first round. Start with a dry run using i.e. the secretary / collegue
How to reward testers?
An 20 Euro/Dollar amazon.de voucher is common, up to 50 Euro/Dollar for hard to get audiences.
How do we show the app to the user and record the screen and audio?
For web apps
- Using a clickable prototype, i.e. Using Figma or by publishing your (Sketch) design to Invision.
- Record using a tool like www.lookback.io
For mobile apps
- Option 1:Test on a mobile device connected to a PC or Mac and record from the screen using lookback.io (lookback recognizes the device, no need for extra software)
- Option 2: For Android Lookback.io now also supports guided user tests out of the box. For IOS too, but it requires some lines of code in your IOS app.
- Option 2: Demo via a web testing tool like https://kobiton.com/ and for recording use lookback.io
- Option 3: Use the built in screen recorder of IOS or Android. (recommended)
Where to do the testing? In office or Remote?
- Recommended option: In office or a (quiet) coffeeshop, for rich dialog.
- Remote is also possible, ie via a trail of www.lookback.io that allows you to join in live and also records the screen.
- Alternative: Use an online user testing service (not ideal, this usually does not allow you to ask live questions to the user)
- "Don't Make Me Think" by Steve Krug
- "Handbook of Usability Testing" by Rubin J. and Chisnell, D.
- "A Practical Guide to Usability Testing" by Dumas, J. and Redish, J.
- "Usability Engineering" by Jakob Nielsen
Cheatsheet introduction text for before test: PDF Think Aloud pretest - version 09/06/17
Helpful usability.gov instructions to remote usability testing.
How to really observe users behaviour instead of relying on their words: NNgroup - First Rule of Usability? Don't Listen to Users
Different techniques of talking in order not to bias a user and to take the most from the observation (not a conversation): NNgroup - Talking with Participants During a Usability Test
User Testing Template
Prepare a User Test
General Preparation tasks before Test Day
- Decide if you’re using the introduction script (click to download) or use the points in the section "General Tester Instruction at Start of the Test". Print it if used. Translate it if needed.
- Prepare the Scenario's you want to test, by adjusting the templates below. A scenario is 1 persona in 1 point in time in 1 system state (i.e. new user in empty system). (see examples below) (i.e. think of Scenario's (Scenario's) and tasks per Scenario, and what test data you need for that). By searching for the word "todo" you will find all area's to fill in.
- Determine what Test data you need for each Scenario and write it down below under "Scenario specific Preparation tasks before Test Day".
- Setup the system with this test data.
- If you are testing on a Design or Product that's actively beind developed you might want to ask for a seperate stable version for the test period.
- Select / setup Recording software (i.e. Lookback.io or Quicktime on mac for screen and audio recording)
- Gather Laptop(s) or mobile devices as needed
- Adjust the Agenda paragraph to the test length (i.e. 90 mins, 60 mins, etc)
- Print the "How to run a test" section below. Make sure the tester parts are on seperate pages and take some blank sheets of A4.
- Do a dry run with colleague who ideally doesn't know the app
- Read “Advice for Interviewer” section below.
- Buy Amazon gift card or other gift if applicable
Advice for Interviewer to keep in mind
- The focus is only on the pure user intentions and thoughts, reduce bias as much as possible:
- Don't explain, only ask questions. However, avoid straightforward questions which can be biasing as well.
- Don't answer questions during the test. Apologise and say that this is exactly the question which cannot be answered right now, but we can discuss it in the end of the test.
- Ask a little as "if" questions as possible (for example, "if this won't be here, if you were needing this")
- Avoid tech or designer talk (shortcuts like UI, terms and specifications).
- Don't interrupt the tester if he is reading or thinking, but sometimes do ask questions, especially if you see facial expressions.
Scenario specific Preparation tasks before Test Day
- Add a User with no purchased objects and login as below. And no profile image. One user for each tester that is coming in and one backup.
- For each user: 4 objects to explore, 2 outside New York.
- (TODO: fill in your actual test (data) to prepare before)
Run a User Test
Process to be used during the actual test when the tester has arrived.
- Introduction Round (5 mins)
- General Instruction(5 mins)
- General User Interview Questions (20 mins)
- Scenario's (40 mins)
- General After Scenario questions (15 mins)
- General After Entire Test script (5 mins)
General Tester Instruction at Start of the Test
- Thank user for his/her help.
- Explain why we are doing this test
- Get recording permission and start recording
User Interview questions
- What company do you work for?
- How many employees work there?
- Do you use any similar software?
- (if yes) What device(s) do you use it on?
Scenario 1 - Facilitator part
General Tester Instruction at Start of the Scenario
- Explain they can't do anything wrong
- Explain we want honest opinions, don’t worry about hurting our feelings
- Explain that they are testing a prototype
- Explain to really click on things, but not everything works, but we want to learn so it's good
- Explain to think out loud
- Explain the role of the user - NOT NEEDED: Is on tester handout.
- Explain the task of the user - NOT NEEDED: Is on tester handout.
- Handout the Scenario Tester handout.
- Cover up tasks 2 and further (we reveal them 1 by 1).
- Ask the user to read and start with task 1 afterwards.
Questions after a Task
Question for after a Task was completed. The numbers of the questions below are the same as the task numbers in tester handout
- What would you do now the password is not working?
- What’s your first impression? Is it clear to you what the different parts of the application are for (objects/wishlist/etc)?
- Did you notice what cities the objects are from?
- Is it clear to you what you bought exactly? Did you have all the information you needed? What do you expect will happen next?
- What do you think the difference is between the 2 emails? What do you think the picture to the top left is for (on objects screen)?
- (TODO: replace these examples with your actual questions)
Questions after Scenario
General questions for after the scenario was completed all Scenarios
- Ask about the overall impression of the tested feature (or have him/her fill in the SUS (System Usability Scale) first and then discuss it).
- Do you think you would like to use this feature frequently?
- Was the process complex or straightforward for you? Could you predict well the result of the actions?
- How would you assess the integration of the various functions in the tested feature?
- Was the system consistent?
- In your opinion, how easy for other people would be to learn to use this feature?
- Did you feel confident using the feature?
- How would you improve this app?
- (TODO: replace these examples with your actual questions)
Scenario Specific questions for after the scenario was completed
- (TODO: replace these examples)
(TODO: Copy paste the next scenario here if there is one, otherwise remove this line.)
After all Scenario's have been completed
- Thank user, tell him/her how this helps
- Tell him/her what we're going to do with is, when it goes live
- Handout the gift voucher or tell him/her when it will be mailed.
After Tester left
- Reinstall app to clear out login data.
- Make sure everything is ready for next test
- If time allows: Evaluate.
Scenario 1 - Tester Handout
You are a Agent who work at a small 3 person company in New York. You signed up with us because offer potential clients to you (you are paying a certain service fee). They told you in an email that you can use their service via the app or via an Iphone app and you decided to give the Iphone app a go. The login details that they emailed you are below. For this test please ignore spelling errors and the fact that many words are not translated to German yet. Also ignore the menu in the bottom of the screen. Remember to think out loud.
Remember: Please think out loud the whole time. No worries, you cannot do or say something wrong.
- Try to Login to the system.
- The correct password is xxxxxxx. Login to the system and tell us your first impression. If any questions or concerns popup let us know. Do not yet Edit or Add something yet.
- Compare objects and tell us which one looks most interesting to you
- Try to buy an object you like. You want to pay by invoice. Go through it step by step and tell us what you are thinking.
- You want change the email that you use to login. Find out if its possible (no need to actually change it, if it’s possible).
Table of Contents