I have an application that starts by loading a large pickled trie (173M) from disk and then uses it to do some processing. I’m making frequent changes to the processing part, which is inconvenient because loading the trie takes 15 minutes or so. I’m looking for a way to eliminate the repeated loading during testing, since the trie never changes.
One thing I can’t do is use a smaller version of the trie.
Ideas I’ve had so far are memcached and turning the trie into a web service that accepts a query and returns the data I need.
What I’m looking for is the least-effort path to a situation in which I can repeatedly change and reload the processing code while maintaining access to the in-memory trie. A direct reference to the tree would be preferable since this would require minimal code changes, but really I’m looking to minimize overall effort.
You could try using Pythons built-in
reloadmethod or the livecoding project.