Question

My team is close to deploying our application, and we’re about to go into closed beta with some select customers. I’m wondering what a realistic timeframe would be for producing new beta versions, and how many such cycles we could realistically expect to need before we can call the first version stable enough for release.

The application itself is a medical imaging application, so it absolutely cannot crash or corrupt data. Many users will also be using the software for at least four to eight hours a day continuously, so I expect that normal user errors will be encountered fairly quickly. The application is tied to a specific piece of hardware, and if they have the hardware, they will need this application or the previous version of the application to run their hardware.

Of course, there’s also pressure from above to release it now, now, now! and since they pay my paychecks, I’m obliged to follow their instructions, despite whatever misgivings I may have about speedy releases.

I’m thinking that the following scenarios could play out:

1. Two week cycle time. We have a select group of users, say three to five sites, and as they encounter bugs, we fix them. I think that this cycle time is absurdly fast, but I can already feel that it’s going to be how the Powers That Be will want to deploy. For this approach, we lock the product to a particular build, and any errors that accumulate we fix in the next release (which could be fifty builds later).
2. Six week cycle time. We have the same select group of users, but that group can grow, and as it grows, we act as in step 1. Not as fast as 1, but certainly more cautious. Problem is, users may get the impression that the product is excessively buggy (if they encounter bugs), and won’t have that impression countered until we release another version, at which point they may no longer care. Since there’s that lockin to the hardware I mentioned earlier, that impression of bugginess may just translate into mild grumbling rather than lost sales. However, each newer beta version will be just that much more vetted than the last.
3. As fast as bugs are fixed, get fixed versions into the hands of users. We have a build server, we have several testers, and we’re pretty quick to respond (you might even say, ‘agile’). Are there drawbacks to just giving out bug fixes as fast as we fix a bug, so long as the fix doesn’t break some other behavior that the software needs? If we took this approach, would we be doing cycles, or just a beta ‘period’?

I realize that these questions vary greatly from user to user, and that something like a Blizzard or Gmail beta period is a bit on the long side. I would still like to get a general feel for how I should respond to management’s constant questions about how long it should be in beta.

Answer 1

In our case, we let the customer’s feedback dictate our cycle. After releasing the initial beta, we listen to the feedback. Sometimes there is a catastrophic bug and we halt rollout of the beta, fix it and then resume. Othertimes, we collect bugs based on customer importance and then rollout an update once the complaining dies down. Ultimately, the customer’s perception is what matters most here (in reference to speed) and so balancing their complaints against what management wants is the tricky aspect. In our case, we generally go with somewhere in the range of 2-4 weeks based on how many and how urgent the bugs are.