How much can we expect to learn from one person?
Paying close attention to user feedback during design iterations is a learning opportunity like none other. That’s why every serious UX designer will consistently test their wireframes and apps with at least five users.
Finding five people to test with can often be a challenge, especially when you are doing several design iterations. Bugging five people with every iteration is not only a chore, but you start to hear the same things over and over.
The amount of feedback repetition is actually quite consistent. In fact, Nielsen Norman Group says you can get 33% of all your feedback from just one user. To discover all usability problems, they say you’ll need to interview at least fifteen users. So where’s the practical middle ground in this?
The answer is in distribution. Test with fewer users early on, then include more users in later iterations. In my project sprints, I stagger the scale as we move from testing wireframes to testing the actual product.
My typical test user distribution looks something like this:
- Design sprints:
- Lo-fi designs: 1 tester
- Med-fi designs: 3 testers
- Hi-fi designs: 5 testers
- Development sprints: testing done by dev team
- QA sprints:
- QA: 1-3 testers
- UAT: 5 testers
- Beta deployment: at least 15 testers
Eliminate redundant feedback by beginning with just one tester
Reduction of redundancy is the number one reason for scaling the number of test users this way. You can be confident that you’ll get 33% of your feedback from that first tester, as this feedback is likely to reflect your most glaring issues.
Cutting out all that noise in the beginning is essential if you want efficient testing. This also protects a major weak spot in your dev team. If you really want to get under a developer’s skin, report the same defect multiple times, then run to a safe space.
UX versus Functional Testing
My points above are mainly related to evaluating the usability of user experience features. When it comes to functional testing of technical requirements, that’s another ball of wax. That’s a case of objective versus subjective requirements.
For testing objective requirements we recommend none other than CapIO, Cerna’s amazing automated testing framework.