*
programming and human factors
by Jeff Atwood
*
Pop quiz, hotshot. How do you know if your application works? Sure, maybe your app compiles. Maybe it passes all the unit tests. Maybe it ran the QA gauntlet successfully. Maybe it was successfully deployed to the production server, or packaged into an installer. Maybe your beta testers even signed off on it.
But that doesn't mean it works.
Can users actually understand your application? Can they get their work done in your application? That's what defines a working application. All the other stuff I listed is just noise. You don't know if your application truly works until you've performed usability tests on it with actual users.
And you regularly do usability testing on your application, right?
That's what I thought. One of the central concepts in Steve Krug's book Don't Make Me Think is that usability testing is essential to any software project. Krug calls his simplified approach to usability testing lost our lease, going-out-of-business-sale usability testing:
Usability testing has been around for a long time, and the basic idea is pretty simple: If you want to know whether your software or your Web site or your VCR remote control is easy enough to use, watch some people while they try to use it and note where they run into trouble. Then fix it, and test it again.In the beginning, though, usability testing was a very expensive proposition. You had to have a usability lab with an observation room behind a one-way mirror, and at least two video cameras so you could record the users’ reactions and the thing they were using. You had to recruit a lot of people so you could get results that were statistically significant. It was Science. It cost $20,000 to $50,000 a shot. It didn’t happen very often.
But in 1989 Jakob Nielsen wrote a paper titled "Usability Engineering at a Discount" and pointed out that it didn’t have to be that way. You didn’t need a usability lab, and you could achieve the same results with a lot fewer users. The idea of discount usability testing was a huge step forward. The only problem is that a decade later most people still perceive testing as a big deal, hiring someone to conduct a test still costs $5,000 to $15,000, and as a result it doesn’t happen nearly often enough.
What I’m going to commend to you in this chapter is something even more drastic: Lost our lease, going-out-of-business-sale usability testing. I’m going to try to explain how to do your own testing when you have no money and no time. If you can afford to hire a professional to do your testing, by all means do it— but don’t do it if it means you’ll do less testing.
Krug points out that usability testing is only as difficult as you make it. It's possible to get useful results from a usability test with a single user, even:
[Usability] testing always works, and even the worst test with the wrong user will show you things you can do to improve your site. I make a point of always doing a live user test at my workshops so that people can see it's very easy to do and it always produces an abundance of valuable insights. I ask for a volunteer and have him try to perform a task on a site belonging to one of the other attendees. These tests last less than ten minutes, but the person whose site is being tested usually scribbles several pages of notes. And they always ask if they can have the recording of the test to show their team back home. Once person told me that after his team saw the recording, they made one change to their site which they later calculated had resulted in $100,000 in savings.
For more proof that you don't need a lot of users to have an effective usability test, Jakob Neilsen offers the following graph:
Obviously, not doing any usability testing at all is a disaster. But what's not so obvious is that usability testing with just a few users is remarkably effective. And it can be relatively painless if you follow Krug's broad guidelines for low-fidelity usability testing:
If you don't already own a copy of Don't Make Me Think, shame on you. In the meantime, I highly recommend downloading Chapter 9 of Steve Krug's Don't Make Me Think (pdf), which has much more detail than the summary I've presented.
Usability testing doesn't have to be complicated. If you really want to know if what you're building works, ask someone to use it while you watch. If nothing else, grab Joe from accounting, Sue from marketing, grab anyone nearby who isn't directly involved with the project, and have them try it. Don't tell them what to do. Give them a task, and remind them to think out loud while they do it. Then quietly sit back and watch what happens. I can tell you from personal experience that the results are often eye-opening.
The benefits of usability testing are clear. You just have to do it to realize any of those benefits.
Posted by Jeff Atwood View blog reactions
« Stylesheets for Print and Handheld The Economics of Bandwidth »
Greate article, very clear.
Diego Carrion on February 2, 2007 03:37 AMGreat article, but you forgot to mention one part of the studies I found extremely enligthening (I don't remember whether it was Krug's or Nielsen's insight, I think it was Krug's cause I remember the graphics but I don't have the book on me right now): *doing two tests with 3 users each finds out more bugs than doing one test with 6 users, or 8*.
Because some bugs or issues may shadow or completely hide other bugs or issues, making users unable to run in them or find them, testing the same application version on 6 users will never be able to find some of the bugs/issues, while 3 users will find most of the bugs the 6 users would find (let's say 75-80% if we use Nielsen's graph), but correcting these issues and then testing again with 3 users will end finding out more bugs/issues than the 6 users test would've uncovered at a marginally higher cost.
Masklinn on February 2, 2007 04:32 AMSteve Krug's book is invaluable. I have owned it for 5+ years and still refer to it when it is time to conduct usability tests. It doesn't hurt to subscribe to Nielsen's Alertbox either.
Tod1d on February 2, 2007 06:13 AMUsability testing is, of course, very useful. But these "usability test is easy and cheap" articles seem to me to implicitly be about testing software aimed at the general population. That is, a program where just about anyone is supposed to be able to just sit down and do something. This makes it fairly easy to get user testers.
But what if your product is a niche product in a specific industry where it is assumed that your users will already understand all the basic concepts of that industry and will already understand the jargon of the industry and let's make it even harder and assume that people generally receive training in your product before using it.
For example, something like Photoshop. A program where you don't really expect the average computer user to be able to just sit down and show you whether your changes to the "inverse gamma dithering defractor" interface make the program easier or harder to use.
Sure, your staff understands the program. But trying to get external usability testers can become more and more expensive and complicated the more specialized your product is. And I get a little frustrated because these "usability testing can be easy and cheap" articles tend to concentrate on the best case scenerio.
Bill on February 2, 2007 06:16 AMCan you explain how Nielsen has managed to define what 100% of usability errors are, and if you only have to use 15 users to attain 100% usability.
m1 on February 2, 2007 06:17 AMI'm always disappointed at how many people involved in a project avoid usability testing as it interferes with their ability to advance their own agenda. I can't count the number of times a BA or PM or whoever has approached my cubicle for design advice (I have a design/usability background) and we have had this kind of conversation:
BA: "What do you think of this idea?"
[translation: 'tell me how great you think my idea is']
Me: "Well, the command buttons and form fields would be more effective if they were grouped logically. I would recommend arranging them like this (draws sketch) and running it by a couple users in a quick usability test to see how it works."
BA: "Oh, I don't think we have time for anything like that. Besides, I'm sure the organization is fine. I like it! Well, thanks for your feedback."
[translation: 'asshole. who asked you anyway, mister "expert?" i'm gonna do it my way 'cos I worked hard to make this page and i don't wanna see you mess with my design.']
People -- /especially/ non-designers -- invest so much of their own ego into their designs, that it becomes a major barrier to even getting any usability testing done, because they're afraid it'll punch holes in what they did. Which of course it shouldn't be viewed as 'punching holes' in the first place, but there you go.
The Idiot Developer on February 2, 2007 06:18 AMI work on a small web development team of four people and time is definitely scarce so the step of usability testing normally gets skipped or quickly bypassed during the development life cycle so that the next project can come down the pipe but we have had several projects that have had some rough launches because of this.
Thanks for all the great tips to making the process easier so that testing doesn't have to get overlooked and bypassed in order to "save" time.
Eric on February 2, 2007 06:32 AMGreat artikel and great site. I come here often, but have noticed one particular behaviour on your site. I am not sure it is intended, but since you're on the subject of usability, I'll mention it. I (and maybe I am not alone in this) have a habbit when reading online, to select 3 or 4 lines of text with the mouse (arround where I am reading at that moment). When I do this on codinghorror however, The selection is always at a different line than the selection I make with the mouse. This is strange and sometimes trying ro select things causes a jump to the top of the page.
Great entry. :)
Paula Shuler on February 2, 2007 08:07 AMstefve - I've noticed a similar behaviour in IE on this and other sites. It works fine in Firefox.
[ICR] on February 2, 2007 08:43 AMI'll second your recommendation of Krug's book. It is an invaluable source... and it doesn't matter if you write web pages (which is kinda who Krug is targeting) or desktop apps.
Eric D. Burdo on February 2, 2007 08:46 AMMakes me think of those Media Player skins, or the ugly skinned disasters both Roxio (and worse) Ahead put on their coaster-toaster software. Even IE7 got a ridiculous anti-user UI facelift.
Bob on February 2, 2007 08:53 AMUnfortunately, Usability is not something we can automate, otherwise we could start a Usability Driven Design movement to supplant Test Driven Design as a coding methodology.
Joking aside, from a UI perspective, UDD makes absolute sense.
Haacked on February 2, 2007 09:12 AMAs i'm in the middle of a website usability report for a uni module i'll chime in ... "think aloud" has it's problems: who's used to talking about their cognative processes whilst undertaking a given task and being monitored? As usual, the best results come from a range of techniques.
compstude on February 2, 2007 09:35 AMIf I remember well, the referenced chapter is of the first edition of the book. In the second one he removed that chapters that explained how to do a usability test and put them on the web. I think they are the best resource about how to start doing it.
Paulo Eduardo Neves on February 2, 2007 09:57 AM> Can you explain how Nielsen has managed to define what 100% of usability errors are
It helps if you read the Neilsen articles I linked in the body of the post:
Why You Only Need to Test With 5 Users
http://www.useit.com/alertbox/20000319.html
Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier
http://www.useit.com/papers/guerrilla_hci.html
Different perspective, but same basic themes in the Krug book. Usability testing is only as difficult as you make it.
Jeff Atwood on February 2, 2007 10:34 AM"the results are often eye-opening" - So true! Yeah, programmers cannot comprehend the stupidity of users.
Jason on February 2, 2007 12:25 PMProgrammers aren't Homo Sapiens.. they're Homo Logicus. Users are from a different species.
http://www.codinghorror.com/blog/archives/000091.html
Jeff Atwood on February 2, 2007 12:31 PMThe makers of Camtasia, TechSmith, have another product called UserVue. I have found it extremely simple to use, and with a monthly rate it is affordable for when I need it.
http://www.techsmith.com/uservue.asp
We have an app with a very specific business purpose. Somebody off the street would have no idea what we're talking about. Is there still value in testing release n+1 with experienced users? Or just testing the current release with them?
Jim on February 2, 2007 05:31 PM> Yeah, programmers cannot comprehend the stupidity of users.
This statement follows a very common fallacy. Users are *not* stupid: what they are is *not interested* in your program, only in getting something done. The more attention we demand from them just to understand *how* to achieve a goal, the less attention the user is paying to the *goal itself*.
From the programmers' perspective, users might seem stupid, but that's because a user is only going to apply a tiny portion of their attention and cognition to understanding the interface. The rest is spent thinking about *what they want to achieve*. A user interface must be designed to allow the user to devote as little of their attention as possible to the interface, and *still* get the task achieved.
@bignose
Rock-on, Man! I could not have said that better myself. Can I quote that last paragraph? I know about, oh, 40 or 50 people who should memorize it.
HugoDataway on February 3, 2007 08:33 PMI agree with bignose. Programmers often spend so much time complaining about how stupid users couldn't use their program, when really it was a result of a stupid programmer designing a UI for a website control panel which requires a 1000 page training manual to understand. I didn't really see it mentioned but usability testing is always vital to prevent onset of the feared "developer UI" (http://www.codinghorror.com/blog/archives/000734.html). I know from my experience that it can be easy to end up showing your creation to someone and having them go "great... how do I use it?". Its so easy to fall into the trap of ignoring obvious flaws in your UI design because _you_ know exactly how it works, so of course everyone else should.
Greg Poole on February 3, 2007 09:32 PM[quote=bignose]A user interface must be designed to allow the user to devote as little of their attention as possible to the interface, and *still* get the task achieved.[/quote]
I give up. Bignose is perfectly correct about his perfect interface, but it's only going to be helpful for perfectly stupid people and the programmers who love them.
As a Sergio Aragonés fan, I suggest we run the stupid people out of town before they burn it down.
karl on February 4, 2007 06:44 PMbest book ever. lol
I pass my copy around to everyone i know who is "designing" a webpage.
@Jim: "We have an app with a very specific business purpose. Somebody off the street would have no idea what we're talking about. Is there still value in testing release n+1 with experienced users? Or just testing the current release with them?"
Absolutely. :-)
I was hired to rewrite a dozen or so DOS applications in Windows. Another programmer had been working on a few of the apps, and the first thing I heard was that the current users were afraid that the apps wouldn't allow data entry as fast as the DOS versions. (The particular app being discussed was for medical claims entry.)
I spent some time with the best data entry clerk, watching her enter claims, where the data was arranged on the forms, the flow of entry across the screen, and so forth. I then designed a data entry screen which mimicked the DOS screen, with the same arrangement of controls and tab order. I even wrote code to allow the Enter key to substitute for the Tab key, so they didn't have to do anything differently there. Data was validated on the fly, so there weren't lookups or listbox controls; if they need to look something up, they leave the control empty and hit Enter and a new form is displayed for the lookup. They can select the record they want and hit Enter, and the lookup form closes and the data is filled in on the main form.
I then got the same data entry clerk to sit at my desk and enter some real-life claims (into a development database, of course). I watched her pretty closely; within about two minutes she seemed to be working at the same speed level as she had with the old system. I did notice that she was having to repetitively type some of the data for each claim, however; after talking about it with her, I made some modifications that allowed the repetitive entries to carry over to the next claim automatically, but also allowed them to easily clear it when it actually needed to change. She tested my changes, and after a few minutes, she decided that it was even *faster* than the old system.
I then brought in a different clerk, and had her enter some claims to see what she thought. She suggested a couple of minor changes, and I added those. I then had both of them test the changes and OK them. I then added two more clerks and repeated the process.
Once all four of them were satisfied, I rolled the app out to the other users for them to test. After they all provided input (and surprisingly no one asked for changes!), I released it into production.
My users all get told the same thing: If you have to repeat a manual task more than once, talk to me and we'll see if it can be automated. If you think something is hard to do, talk to me and we'll see if it can be made simpler. If you have an idea about something that could make things easier for you, talk to me and we'll see if it can be made to happen. When they do talk to me about ideas/changes/whatever, I actually listen and whenever possible implement them. It puts me on really good terms, and I have pretty eager volunteers for the testing on new development; they don't seem to mind doing the extra work when it makes things better for them in the long run.
Ken
KenW on February 5, 2007 08:12 AMSo when I was forcing my parents to play my games, I was doing usability testing? I just wanted to see if someone older than 30 could grok the experience.
Jae on February 5, 2007 12:02 PMHey, this has cheered me up a bit! I'm writing a piece of software for the office at my school (full of non-techie form-filling phone-answering-type-guys), and my current method of testing is to load it up, give the laptop to my (non-techie) mother and write down all the times she hits the wrong button or asks me what to do next.
Here I thought I was just being lazy, but it turns out this is the right thing to do!
Rissa on February 6, 2007 03:34 AM{{{KenW}}}
Alla youse pay attention to the man, because what he does is how it should be done. In 30-odd years of data entry of all kinds, Windows, DOS, and who-know-what-this-ugly-thing-is, I have encountered maybe one or two IT guys who grasped the reason why users should be treated with this courtesy and respect. Those of you complaining about the idiots in the chairs failed Manners 101 and need to learn that you will be a better genius when you pay close attention to the users of your products out there in the marketplace. In turn, we users become more adept and skillful and responsive and responsible users when we see that our needs are being addressed directly.
tomatoqueen, thanks for pointing out another reason why I hate users.
that's: They bitch and moan incessantly.
Any suggestions for how to get usability testing when you can't reach the intended users?
My company writes applications for doctors in a specific medical field, and we are having the hardest time getting any real user to sit down for a usability test. Docs won't waste their time to make a little money, and other people (like lab techs, nurses, residents, etc.) just don't have enough knowledge of the process to give us the feedback we need.
karl: Sounds like I'd really enjoy using some of your software.
But then, I also enjoy cold coffee, burnt cookies, reality TV, political speeches, refusing to ask for directions when I'm lost, and occasionally poking myself in the eye with a sharp stick.
Haamu on February 7, 2007 09:09 AMGreg Poole: you're welcome to quote that passage you liked. The wisdom is just restated from _Don't Make Me Think_, _The Humane Interface_, and many others; they're probably more quotable than me.
HugoDataway: oops, I meant to address you with the okay-to-quote. Heck, anyone's allowed to quote what I've said in this discussion if they think it'll help.
same problem as stefve's :)
cmone_ on June 6, 2007 08:43 PM| Content (c) 2008 Jeff Atwood. Logo image used with permission of the author. (c) 1993 Steven C. McConnell. All Rights Reserved. |
Subscribe in a reader
Subscribe via email