January 31, 2007
Pop quiz, hotshot. How do you know if your application works? Sure, maybe your app compiles. Maybe it passes all the unit tests. Maybe it ran the QA gauntlet successfully. Maybe it was successfully deployed to the production server, or packaged into an installer. Maybe your beta testers even signed off on it.
But that doesn't mean it works.
Can users actually understand your application? Can they get their work done in your application? That's what defines a working application. All the other stuff I listed is just noise. You don't know if your application truly works until you've performed usability tests on it with actual users.
And you regularly do usability testing on your application, right?
That's what I thought. One of the central concepts in Steve Krug's book Don't Make Me Think is that usability testing is essential to any software project. Krug calls his simplified approach to usability testing lost our lease, going-out-of-business-sale usability testing:
Usability testing has been around for a long time, and the basic idea is pretty simple: If you want to know whether your software or your Web site or your VCR remote control is easy enough to use, watch some people while they try to use it and note where they run into trouble. Then fix it, and test it again.
In the beginning, though, usability testing was a very expensive proposition. You had to have a usability lab with an observation room behind a one-way mirror, and at least two video cameras so you could record the users' reactions and the thing they were using. You had to recruit a lot of people so you could get results that were statistically significant. It was Science. It cost $20,000 to $50,000 a shot. It didn't happen very often.
But in 1989 Jakob Nielsen wrote a paper titled "Usability Engineering at a Discount" and pointed out that it didn't have to be that way. You didn't need a usability lab, and you could achieve the same results with a lot fewer users. The idea of discount usability testing was a huge step forward. The only problem is that a decade later most people still perceive testing as a big deal, hiring someone to conduct a test still costs $5,000 to $15,000, and as a result it doesn't happen nearly often enough.
What I'm going to commend to you in this chapter is something even more drastic: Lost our lease, going-out-of-business-sale usability testing.
I'm going to try to explain how to do your own testing when you have no money and no time. If you can afford to hire a professional to do your testing, by all means do it -- but don't do it if it means you'll do less testing.
Krug points out that usability testing is only as difficult as you make it. It's possible to get useful results from a usability test with a single user, even:
[Usability] testing always works, and even the worst test with the wrong user will show you things you can do to improve your site. I make a point of always doing a live user test at my workshops so that people can see it's very easy to do and it always produces an abundance of valuable insights. I ask for a volunteer and have him try to perform a task on a site belonging to one of the other attendees. These tests last less than ten minutes, but the person whose site is being tested usually scribbles several pages of notes. And they always ask if they can have the recording of the test to show their team back home. Once person told me that after his team saw the recording, they made one change to their site which they later calculated had resulted in $100,000 in savings.
For more proof that you don't need a lot of users to have an effective usability test, Jakob Neilsen offers the following graph:
Obviously, not doing any usability testing at all is a disaster. But what's not so obvious is that usability testing with just a few users is remarkably effective. And it can be relatively painless if you follow Krug's broad guidelines for low-fidelity usability testing:
- When should I test? Ideally, once per month. You should be running small usability tests continuously throughout the development process. The tests should be short and simple, so you can conduct them almost any time with little advance planning.
- How many users do I need? Three or four max.
- What kind of users? Grab some people. Anyone who can use a computer will do. The best-kept secret of usability testing is that it doesn't much matter who you test. It's a good idea to get representative users, but it's much more important to test early and often. Don't be embarrassed to ask friends and neighbors.
- How much time will it take? 45 minutes to an hour per user. Keep it simple. Keep it small. Although it does take extra time to conduct usability tests, even simple ones, ultimately you will save time. The results of the usability tests will prevent you from wasting time arguing endlessly, or redoing things at the end of a project.
- Where do I conduct the test? Any office or conference room. All you need is a room with a desk, a computer, and two chairs where you won't be interrupted.
- Who should do the testing? Any reasonably patient human being. Choose someone who tends to be patient, calm, emphathetic, and a good listener. With a little practice, most people can get quite good at it.
- What equipment do I need? All you need is some form of screen recording software, such as Camtasia. If you want to get really fancy you can bring in a camcorder to record the person and the screen.
- How do I prepare for the tests? Decide what you want to show. Have a short script (doc) ready to guide the participants through the test.
- How much will it cost? Minus the moderator's time, a $50-$100 stipend per user.
- How do we interpret the results? Debrief the development team and any interested stakeholders over lunch the same day. One of the nicest things about usability testing is that the results tend to be obvious to everyone who's watching. The serious problems are hard to miss.
If you don't already own a copy of Don't Make Me Think, shame on you. In the meantime, I highly recommend downloading Chapter 9 of Steve Krug's Don't Make Me Think (pdf), which has much more detail than the summary I've presented.
Usability testing doesn't have to be complicated. If you really want to know if what you're building works, ask someone to use it while you watch. If nothing else, grab Joe from accounting, Sue from marketing, grab anyone nearby who isn't directly involved with the project, and have them try it. Don't tell them what to do. Give them a task, and remind them to think out loud while they do it. Then quietly sit back and watch what happens. I can tell you from personal experience that the results are often eye-opening.
The benefits of usability testing are clear. You just have to do it to realize any of those benefits.
Posted by Jeff Atwood
The makers of Camtasia, TechSmith, have another product called UserVue. I have found it extremely simple to use, and with a monthly rate it is affordable for when I need it.
Greate article, very clear.
Great article, but you forgot to mention one part of the studies I found extremely enligthening (I don't remember whether it was Krug's or Nielsen's insight, I think it was Krug's cause I remember the graphics but I don't have the book on me right now): *doing two tests with 3 users each finds out more bugs than doing one test with 6 users, or 8*.
Because some bugs or issues may shadow or completely hide other bugs or issues, making users unable to run in them or find them, testing the same application version on 6 users will never be able to find some of the bugs/issues, while 3 users will find most of the bugs the 6 users would find (let's say 75-80% if we use Nielsen's graph), but correcting these issues and then testing again with 3 users will end finding out more bugs/issues than the 6 users test would've uncovered at a marginally higher cost.
We have an app with a very specific business purpose. Somebody off the street would have no idea what we're talking about. Is there still value in testing release n+1 with experienced users? Or just testing the current release with them?
Steve Krug's book is invaluable. I have owned it for 5+ years and still refer to it when it is time to conduct usability tests. It doesn't hurt to subscribe to Nielsen's Alertbox either.
Usability testing is, of course, very useful. But these "usability test is easy and cheap" articles seem to me to implicitly be about testing software aimed at the general population. That is, a program where just about anyone is supposed to be able to just sit down and do something. This makes it fairly easy to get user testers.
But what if your product is a niche product in a specific industry where it is assumed that your users will already understand all the basic concepts of that industry and will already understand the jargon of the industry and let's make it even harder and assume that people generally receive training in your product before using it.
For example, something like Photoshop. A program where you don't really expect the average computer user to be able to just sit down and show you whether your changes to the "inverse gamma dithering defractor" interface make the program easier or harder to use.
Sure, your staff understands the program. But trying to get external usability testers can become more and more expensive and complicated the more specialized your product is. And I get a little frustrated because these "usability testing can be easy and cheap" articles tend to concentrate on the best case scenerio.
Can you explain how Nielsen has managed to define what 100% of usability errors are, and if you only have to use 15 users to attain 100% usability.
I'm always disappointed at how many people involved in a project avoid usability testing as it interferes with their ability to advance their own agenda. I can't count the number of times a BA or PM or whoever has approached my cubicle for design advice (I have a design/usability background) and we have had this kind of conversation:
BA: "What do you think of this idea?"
[translation: 'tell me how great you think my idea is']
Me: "Well, the command buttons and form fields would be more effective if they were grouped logically. I would recommend arranging them like this (draws sketch) and running it by a couple users in a quick usability test to see how it works."
BA: "Oh, I don't think we have time for anything like that. Besides, I'm sure the organization is fine. I like it! Well, thanks for your feedback."
[translation: 'asshole. who asked you anyway, mister "expert?" i'm gonna do it my way 'cos I worked hard to make this page and i don't wanna see you mess with my design.']
People -- /especially/ non-designers -- invest so much of their own ego into their designs, that it becomes a major barrier to even getting any usability testing done, because they're afraid it'll punch holes in what they did. Which of course it shouldn't be viewed as 'punching holes' in the first place, but there you go.
I work on a small web development team of four people and time is definitely scarce so the step of usability testing normally gets skipped or quickly bypassed during the development life cycle so that the next project can come down the pipe but we have had several projects that have had some rough launches because of this.
Thanks for all the great tips to making the process easier so that testing doesn't have to get overlooked and bypassed in order to "save" time.
Great artikel and great site. I come here often, but have noticed one particular behaviour on your site. I am not sure it is intended, but since you're on the subject of usability, I'll mention it. I (and maybe I am not alone in this) have a habbit when reading online, to select 3 or 4 lines of text with the mouse (arround where I am reading at that moment). When I do this on codinghorror however, The selection is always at a different line than the selection I make with the mouse. This is strange and sometimes trying ro select things causes a jump to the top of the page.
stefve - I've noticed a similar behaviour in IE on this and other sites. It works fine in Firefox.
I'll second your recommendation of Krug's book. It is an invaluable source... and it doesn't matter if you write web pages (which is kinda who Krug is targeting) or desktop apps.
Makes me think of those Media Player skins, or the ugly skinned disasters both Roxio (and worse) Ahead put on their coaster-toaster software. Even IE7 got a ridiculous anti-user UI facelift.
Unfortunately, Usability is not something we can automate, otherwise we could start a Usability Driven Design movement to supplant Test Driven Design as a coding methodology.
Joking aside, from a UI perspective, UDD makes absolute sense.
As i'm in the middle of a website usability report for a uni module i'll chime in ... "think aloud" has it's problems: who's used to talking about their cognative processes whilst undertaking a given task and being monitored? As usual, the best results come from a range of techniques.
If I remember well, the referenced chapter is of the first edition of the book. In the second one he removed that chapters that explained how to do a usability test and put them on the web. I think they are the best resource about how to start doing it.
Can you explain how Nielsen has managed to define what 100% of usability errors are
It helps if you read the Neilsen articles I linked in the body of the post:
Why You Only Need to Test With 5 Users
Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier
Different perspective, but same basic themes in the Krug book. Usability testing is only as difficult as you make it.
"the results are often eye-opening" - So true! Yeah, programmers cannot comprehend the stupidity of users.
Yeah, programmers cannot comprehend the stupidity of users.
This statement follows a very common fallacy. Users are *not* stupid: what they are is *not interested* in your program, only in getting something done. The more attention we demand from them just to understand *how* to achieve a goal, the less attention the user is paying to the *goal itself*.
From the programmers' perspective, users might seem stupid, but that's because a user is only going to apply a tiny portion of their attention and cognition to understanding the interface. The rest is spent thinking about *what they want to achieve*. A user interface must be designed to allow the user to devote as little of their attention as possible to the interface, and *still* get the task achieved.
Rock-on, Man! I could not have said that better myself. Can I quote that last paragraph? I know about, oh, 40 or 50 people who should memorize it.
I agree with bignose. Programmers often spend so much time complaining about how stupid users couldn't use their program, when really it was a result of a stupid programmer designing a UI for a website control panel which requires a 1000 page training manual to understand. I didn't really see it mentioned but usability testing is always vital to prevent onset of the feared "developer UI" (http://www.codinghorror.com/blog/archives/000734.html). I know from my experience that it can be easy to end up showing your creation to someone and having them go "great... how do I use it?". Its so easy to fall into the trap of ignoring obvious flaws in your UI design because _you_ know exactly how it works, so of course everyone else should.
[quote=bignose]A user interface must be designed to allow the user to devote as little of their attention as possible to the interface, and *still* get the task achieved.[/quote]
I give up. Bignose is perfectly correct about his perfect interface, but it's only going to be helpful for perfectly stupid people and the programmers who love them.
As a Sergio Aragons fan, I suggest we run the stupid people out of town before they burn it down.
best book ever. lol
I pass my copy around to everyone i know who is "designing" a webpage.
@Jim: "We have an app with a very specific business purpose. Somebody off the street would have no idea what we're talking about. Is there still value in testing release n+1 with experienced users? Or just testing the current release with them?"
I was hired to rewrite a dozen or so DOS applications in Windows. Another programmer had been working on a few of the apps, and the first thing I heard was that the current users were afraid that the apps wouldn't allow data entry as fast as the DOS versions. (The particular app being discussed was for medical claims entry.)
I spent some time with the best data entry clerk, watching her enter claims, where the data was arranged on the forms, the flow of entry across the screen, and so forth. I then designed a data entry screen which mimicked the DOS screen, with the same arrangement of controls and tab order. I even wrote code to allow the Enter key to substitute for the Tab key, so they didn't have to do anything differently there. Data was validated on the fly, so there weren't lookups or listbox controls; if they need to look something up, they leave the control empty and hit Enter and a new form is displayed for the lookup. They can select the record they want and hit Enter, and the lookup form closes and the data is filled in on the main form.
I then got the same data entry clerk to sit at my desk and enter some real-life claims (into a development database, of course). I watched her pretty closely; within about two minutes she seemed to be working at the same speed level as she had with the old system. I did notice that she was having to repetitively type some of the data for each claim, however; after talking about it with her, I made some modifications that allowed the repetitive entries to carry over to the next claim automatically, but also allowed them to easily clear it when it actually needed to change. She tested my changes, and after a few minutes, she decided that it was even *faster* than the old system.
I then brought in a different clerk, and had her enter some claims to see what she thought. She suggested a couple of minor changes, and I added those. I then had both of them test the changes and OK them. I then added two more clerks and repeated the process.
Once all four of them were satisfied, I rolled the app out to the other users for them to test. After they all provided input (and surprisingly no one asked for changes!), I released it into production.
My users all get told the same thing: If you have to repeat a manual task more than once, talk to me and we'll see if it can be automated. If you think something is hard to do, talk to me and we'll see if it can be made simpler. If you have an idea about something that could make things easier for you, talk to me and we'll see if it can be made to happen. When they do talk to me about ideas/changes/whatever, I actually listen and whenever possible implement them. It puts me on really good terms, and I have pretty eager volunteers for the testing on new development; they don't seem to mind doing the extra work when it makes things better for them in the long run.
So when I was forcing my parents to play my games, I was doing usability testing? I just wanted to see if someone older than 30 could grok the experience.
Hey, this has cheered me up a bit! I'm writing a piece of software for the office at my school (full of non-techie form-filling phone-answering-type-guys), and my current method of testing is to load it up, give the laptop to my (non-techie) mother and write down all the times she hits the wrong button or asks me what to do next.
Here I thought I was just being lazy, but it turns out this is the right thing to do!
tomatoqueen, thanks for pointing out another reason why I hate users.
that's: They bitch and moan incessantly.
Alla youse pay attention to the man, because what he does is how it should be done. In 30-odd years of data entry of all kinds, Windows, DOS, and who-know-what-this-ugly-thing-is, I have encountered maybe one or two IT guys who grasped the reason why users should be treated with this courtesy and respect. Those of you complaining about the idiots in the chairs failed Manners 101 and need to learn that you will be a better genius when you pay close attention to the users of your products out there in the marketplace. In turn, we users become more adept and skillful and responsive and responsible users when we see that our needs are being addressed directly.
Any suggestions for how to get usability testing when you can't reach the intended users?
My company writes applications for doctors in a specific medical field, and we are having the hardest time getting any real user to sit down for a usability test. Docs won't waste their time to make a little money, and other people (like lab techs, nurses, residents, etc.) just don't have enough knowledge of the process to give us the feedback we need.
karl: Sounds like I'd really enjoy using some of your software.
But then, I also enjoy cold coffee, burnt cookies, reality TV, political speeches, refusing to ask for directions when I'm lost, and occasionally poking myself in the eye with a sharp stick.
Greg Poole: you're welcome to quote that passage you liked. The wisdom is just restated from _Don't Make Me Think_, _The Humane Interface_, and many others; they're probably more quotable than me.
HugoDataway: oops, I meant to address you with the okay-to-quote. Heck, anyone's allowed to quote what I've said in this discussion if they think it'll help.
same problem as stefve's :)