We’re looking at starting a new, specialized (customer facing) web app; there are a few paths we can take — we can code and host in house, we can code in-house and host externally, we can have someone else do the coding and hosting, COTS, etc.
Let’s assume I’ve reasonable ways of estimating quality of COTS and in-house development efforts.
The part I have difficulty is determining how “good” a dev / hosting shop is. What sorts of questions should I be asking them? What about a Joel Test for dev shops? I assume some of the Joel Test questions apply (since if a dev shop is good to work for, hopefully they will produce good code) but it also needs to involve things like:
1) Server architecture (assuming 99.99% uptime)
2) Customer service / QA
3) Responsiveness to service outages, etc.
4) Contract items
Some questions I can think of:
Do you have a bug database?
How do you handle new change requests / bugs?
Do you guarantee turnaround times?
Are new requests billed differently than bugs?
How do you define bugs?
Do you have testers? How many?
Do you have your own data center? Do you lease rackspace / co-loc? Dedicated NOC staff?
What is the size of your development staff? In-house or outsourced?
How many customers do you have? Can I talk with some of them?
What is your warranty period?
The last question on your list, “How many customers do you have? Can I talk with some of them?” should also translate to the viewing of a portfolio of existing products. Then do research on those particular ‘products’ (sites, applications, what-not).
That alone (but shouldn’t be the only thing) should tell you quite a lot about the quality of their work.