Question

Before refactoring my code to start experimenting, I’m hoping the wisdom of the community can advise me on the correct path.

Problem: I have a WPF program that does a diff on hundreds of ini files. For performance of the diffing I’d like to keep several hundred of the base files that other files are diffed against in memory. I’ve found using custom classes to store this data starts to bring my GUI to a halt once I’ve loaded 10-15 files with approximately 4000 line of data each.

I’m considering several strategies to improve performance:

• Don’t store more than a few files in memory at a time and forget about what I hoped would have been perf improvement in parsing by keeping them in memory
• Experiment with running all the base file data in a BackgroundWorker thread. I’m not doing any work of these files on the GUI thread but maybe all that stored data is affecting it somehow. I’m guessing here.
• Experiment with System.Runtime.Caching class.

The question asked here on SO didn’t, in my mind, answer the question of what’s the best strategy for this type of work. Thanks in advance for any help you can provide!

Answer 1

Assuming 100 character lines of text 15 * 4000 * 100 is only 6MB which is a trivial amount of memory on a modern PC. If your GUI is coming to a halt then to me that is an indication of virtual memory being swapped in and out to disk. That doesn’t make sense for only 6MB so I’d figure out how much it’s really taking up and why. If may well be some trivial mistake that would be easier to fix than re-thinking your whole strategy. The other possibility is that it has nothing to do with memory consumption but rather an algorithm issue.