We have an application written in C# .NET that is currently used in production environments. Obviously the release build is used.
Unfortunately sometimes the application misbehaves under certain conditions and we can’t figure out why. We are unable to reproduce the issue in house. Yes, more tracing would help, but often it is after the fact that you realize that more tracing should have been put in place.
What is the best way to debug that installation using the regular, line by line, Visual Studio Attach-to-Process type debugging. Would it be to install the express edition of VS on the customer machine and replace the dlls with the debug versions? Is it enough to just send the PDBs over and the specific source files? Is there a better way? The end goal is to have a development like environment to debug the issue.
Remote debugging is a good feature, but it can seldom be used in production environments because it requires 2-way trust between domains (your own and your customers) that is hard to achieve (administrators of both companies will strongly opposit the idea)
see Remote Debugging Across Domains
However sending PDB files with your application will help. You can ask your customers to use Clr Debugger (DbgCLR.exe) It has some limitations comparing with VS Debugger, but still it is a debugger that can do the work and it is a part of .NET SDK.
If the problem is cannot be debugged with Clr Debugger, customers can try to install a trial version of VS in their production environment and give you a remote desktop connection (if admins will allow you to do this). I suppose that 90-day trial period will be enough to solve the problem
You can also try to build a test environment like your customer have – limit (or increase) number of CPUs, memory etc. to match physical conditions as far as you can. Ask your customer to create a virtual image of their Windows if you think that some additional 3rd party software or non-existing registry record can influence your program.
However, my experience is that in 50% such “non-reproducible” errors in .NET happens because of concurrent access by multiple users (deadlocks, race conditions, logical misbehavior in First Wins/Last Wins scenarios). In most cases these are errors that you will not be able to debug even if you install VS on customer machine (if you run this in non rush hours), because you need to run at least 2 customers on different machines simultaneously.
So, while you are looking for options to provide debugging to customers, keep working on improving tracing and monitoring features. Tracing and performance counters are often your only friends in production environments.