We’re neck deep in a project right now, schedules are tight (but reasonable). Our general strategy is to get a strong beta done, release it for testing, and get feedback from our testers.
Quite frequently, we’re being hit by small things that spiral into long, time-costing discussions. They all boil down to one thing: While we know what features we need, we are having trouble with the little details, things like ‘where should this message go’ and ‘do they need this feedback immediately, or will it break their flow, so we should hold off’?
These are all things that our testers SHOULD catch, but
a) Each ‘low priority’ bug like this drains time from critical issues
b) We want to have as strong a product as possible
and
c) Even the best testing group will miss things from time to time.
We use our product, and we know how our users use the old version…but we’re all at a loss as to how to think like a user when we try to use the new version (which has significant graphical as well as underlying changes).
edit – a bit more background:
We’re writing a web app used by a widely-distributed base of users. Our app is a big part of their jobs, but not the biggest (and, of course, we only matter to them when it doesn’t work). Getting actual users in to use our product is difficult, as we’re geographically distant from the nearest location that serves as an end user (We’re in Ohio, and I think the nearest location we serve is 3+ hours away).
The closest we can get is our Customer Service team (who have been a big help, really) but they don’t really think like the users either. They also serve as our testers (it really motivates them to find bugs when they know that any they DON’T find may mean a big upswing in number of calls). We’ve had three (of about 12 total) customer service reps back here most of the week doing some preliminary testing…they’ve gotten involved in the discussions as well.
Watching someone using the app is a huge benefit to me. Possibly someone who is not entirely familiar with it.
Seeing how they try to navigate, how they try to enter information or size windows. Things we take for granted after creating/running the app hour after hour, day after day.
Users will always try and do things you never expected and watching them in action might bring to light how you can change something that might have seemed minor, but really makes a big impact on them.