Agile usability testing: saving costs


Reducing risk for Agile projects

Usability testing still has an aura of complexity around it – people think of it as time and cost-consuming, and often dispensable. "Why test this, we should know what’s best, after all, we are the experts and it will add cost!"

Truth is, people who conceive applications usually don’t know what’s going on in the users’ minds. Why? Because they're not the users. No matter how sure we are about our design and concept, the users will always surprise us. As for the cost argument, you will see below how usability testing can actually save a significant amount of cost if done early enough in the process.

The new way of testing: quick and collaborative

Over the past couple of years, there has been a clear trend towards what could be called “agile” usability testing: out go the 20+ page reports – written by a usability test engineer and often just put on the shelves by the developers. Instead, we test with fewer users – 5 is a good number – encourage team members to come and observe, and have a debrief immediately after the tests. These debriefs are working sessions where everybody will contribute their observations, ideally on sticky notes which are then classified into overall issue categories. During these working sessions, it’s important to stay away from too much thinking about technical or design solutions ... they will come in the next step.

Testing as early on as possible

paper prototype user testing
Agile testing in action: one person facilitates while the other one acts as the machine.

Another myth: you can only test once you have a running application on your testing environment. Wrong again. Testing is the most useful and efficient when it is done early on in the process. As soon as you have some kind of screen – and it can even be hand-drawn on paper – you can do usability testing. The timing is important because you have not invested days, weeks and months of development time at that point and you can still make radical changes at an extremely low cost.

One recent example with a client: we convinced the business owners to test the concept we had been working on in order to:

  • trial the new navigation model and
  • verify whether the task flow we had assumed based on the business requirements was correct.

usability test cartoonThe testing was an eye opener to everybody involved. While the navigation was understood and appreciated, it turned out that there was a need to create different profiles even amongst a user group to allow a high degree of personalization. In addition, we found out that some fundamental assumptions turned out to be wrong, and we got comments such as “if you do this, you are going to get hung by the feet!” – a threat no project owner can ignore! How fortunate for the project owner that testing was done this early!

Author's note

Since this article was first published, most of the usability testing we do is remote. We still prefer moderated testing to unmoderated, and when you bring together Zoom (conferencing), Otter (automated transcriptions), and Dovetail (research repository), you have a killer combo for running remote, moderated tests in a very cost-effective way.