User Testing on a Shoestring
🎉🎉
YOU MADE AN APP!!!
🎉🎉
"Dad tried it out and it was like watching someone who doesn't know how to use technology try to do something on their phone." - my sister
"The most frustrating part about it is that Dad is not tech illiterate so idk what was going on there." - my sister, again
Let's fix this with user testing
Slides? I gotchu
Just the slides please
The blog post, please
Or: https://rachelkaufman.dev
What is user testing?
Official definition
In a usability-testing session, a researcher (called a “facilitator” or a “moderator”) asks a participant to perform tasks, usually using one or more specific user interfaces. While the participant completes each task, the researcher observes the participant’s behavior and listens for feedback.
Unofficial definition
Putting your app in front of people because you can't learn how people use it unless you watch them
We're gonna need..
some tools
some ideas
some people
Who is this talk for?
You are...
- A solo dev working on a solo project, OR..
- A dev at a tiny nonprofit, OR...
- Volunteering your time for the greater good
You aren't...
- A UX pro (I'm not either)
- Flush with cash
Who am I?
- Software engineer (not UX pro)
- Initiated and ran a user testing project for Code for America's National Brigade Network (sounds fancier than it was)
- <-- My proudest moment
The many types of user testing
Why user test when we have a/b tests?
Great question!
A/B testing is great for answering quantitative questions.
- Does a red checkout button lead to more conversions than a blue one?
- Which subject line gets more people to open our email?
User testing is great for answering more qualitative questions
- Can our users successfully check out?
- Where are they encountering friction?
This is wildly oversimplified, see
https://www.invespcro.com/blog/ab-vs-usability-testing-friends-or-foes/ for more nuance
OK BUT WHAT IS IT?!
Get a person.
Record them using your product.
Think about what problems they surfaced.
Fix them.
Repeat.
So is this, like... a lot of work?
Sounds like it would take forever
No! 5 users can uncover almost 80% of the problems in your design.
Source: https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
How much does this cost?
| Original usability cost estimate by [Mantei and Teorey 1988] | $128,330 |
| Scenario developed as paper mockup instead of on videotape | - $2,160 |
| Prototyping done with free hypertext package | - $16,000 |
| All user testing done with 3 subjects instead of 5 | - $11,520 |
| Thinking aloud studies analyzed by taking notes instead of by video taping | - $5,520 |
| Special video laboratory not needed | - $17,600 |
| Only 2 focus groups instead of 3 for market research | - $2,000 |
| Only 1 focus group instead of 3 for accept analysis | - $4,000 |
| Questionnaires only used in feedback phase, not after prototype testing | - $7,200 |
| Usability expert brought in for heuristic evaluation | + $3,000 |
| Cost for "discount usability engineering" project | $65,330 |
Our cost: a couple hundred bucks maybe
But I don't have any fancy tools :(((
Let's try it!
Gather your peeps
Ok, what do I actually do?
- Make it clear THEY are not being tested
- Load the site
- Remind them to speak out loud
Then what?
- "What can you do here?"
- "Try to find a brown purse that costs less than $200 that you could take to a business meeting."
- Let 'em cook
Some tips
- Remind them THEY are not being tested
- One task at a time
- Open-ended, not closed-ended, questions
- Word your tasks carefully (try to avoid using the feature name in the question)
- Believe what they do, not what they say
What does it all mean?
- Why didn't they use the filter?
- Is it in a weird place?
- It has too many options and is overwhelming?
- You only have 3 purses on the site so you don't need to filter?
- Are all your testers stupid? This is a joke I promise
How to find people
who are as excited to test your app as these guys
Things you will probably learn
- Nobody knows what your site is
- Make buttons bigger
- People notice everything you stopped noticing
"Many ... have described the almost religious effect ... the first time [people] try running a user test and see with their own eyes the difficulties perfectly normal people can have using supposedly "easy" software."
Now convince the boss
Thank you
Get the slides:
Photo credits