After @joel’s presentation at the conference last week I came across this article (which may have informed his presentation?).
That is right! I have it linked in the slides along with the link to the paper it was referring to. It is a pretty interesting piece of work.
The headline and strapline of the article don’t reflect the content very well at all. So here’s a brief summary in case it’s helpful. It’s best to test with between 3-5 users then fix the issues before testing again. If there are different kinds of users, e.g. translators, consultants etc, then each of those groups need 3-5 test users.
It also helps to test your test with a few users. If you line up 20 user tests and discover significant issues with your methodology after you complete the test, that’s a lot of wasted effort! Especially when you’re first getting started, it helps to reevaluate after the first test or two in case you’ve set up your test wrong.
Testing 3–5 people at a time is a compromise that gives you actionable data while reducing your risk of putting in effort to get bad or redundant data.
The more formal and susceptible to bias your test is, the more you can expect to throw out the first pancake.
@Alex_Mercado, do you have any examples of issues in the methodology of setting tests up?