Question

I realise the answer is very likely to be ‘no’!

Basically, I have a graph (the nodes and edges kind) which represents a grid of squares; each node object contains references to every other node that this node has an edge to, which seems to mean that when the graph is serialised using cPickle.dump it traverses every node in the graph in a depth-first fashion, meaning that for a well-connected graph representing a 16×16 grid, it’s effectively treating it as a 256-level-deep data structure. This means that larger grids very quickly overrun the default maximum Python recursion depth, particularly since experimentation suggests it seems to take about 4 calls on the stack to go an extra level into the data structure.

The thing is, I also have a dict-of-dicts that refers into this graph in such a way as to allow me to use cartesian coordinates to find particular nodes (e.g. “node = nodes[3][6]”). So conceptually, it’s not a highly-nested data structure at all, it’s a fairly flat one which happens to have a lot of sideways references, but it seems that cPickle works entirely depth-first (which I understand is by far the easiest way to work).

Now, I know about sys.setrecursionlimit(), and I’ve done some experimentation to find out how big I’d need to set the limit for what size of graph, so that’s the ‘easiest’ option. I’m aware that I could just quit the node-to-node links and rely on the dict-of-dicts to maintain the grid and a separate flat structure to maintain edge weightings, but there are various reasons I’d like to avoid that – not least that the node-to-node links allow for a more intuitive use of the data structure. I believe from what I’ve read that I should be able to provide my own implementations of __getstate__ and __setstate__ and override the pickling functionality, but obviously that’s a non-trivial amount of work. If there were a way to get cPickle (or pickle, I’m not fussy!) to use a breadth-first traversal, it should solve the problem pretty simply!

Answer 1

Writing a suitable __getstate__() method does not seem to be that complicated after all. Try something like

class Node(object):
    def __getstate__(self):
        state = self.__dict__.copy()
        state.pop("neighbours")
        return state

This will pickle all attributes of Node instance except for the neighbours attribute, which I assume to contain the links to the neighbours. (You don’t need a __setstate__() method.)

After unpickling the whole graph, you will have to recreate the links to the neighbours on all nodes, but this shouldn’t be that difficult either.