My background is in conducting usability tests and designing software user interfaces (typically, I wouldn’t do both of these things for the same product). At the time, I knew intellectually that even subtle decisions about the interface could have a great effect on task completion rates and times.

Working on websites and keeping an eye on analytics, I see the connection between design choices and task completion a lot more clearly. Take the example of deciding whether or not to remove a field from a Contact Us form. Try taking it out and wait for a couple of weeks, then go back and compare the completion rate before your made the change to after you made the change. Maybe it went from a 2.1% completion rate to a 2.2% completion rate. That’s a small movement, but depending on how many people go to the form, it could be a dozen or a few hundred extra completions.

The differences could be even more subtle – changing a color, moving an image, cutting down the text, even changing a label. Come up with a theory and go test it (perhaps using some A/B testing).

After thinking over this matter, I’m left with the feeling that usability testing is a blunt tool in comparison to the precision of analytics. I guess in some ways it is, but there is information you get out of usability testing that you just can’t from looking at analytics data. The thing to keep in mind is the cost effectiveness of being able to run many small, precise experiments, using analytics to find answers. Or perhaps even to raise questions that you want to answer with a usability test.