Question

Here’s my situation. I have a graph that has different sets of data being added at different times. For example, set1 might have a few thousand nodes and then set2 comes in later and we apply business logic to create edges from set1 to set2(and disgard any Vertices from set1 that do not have edges to set2). Then at a later point, we get set3, set4, and so on and the same process applies between each set and its previous set.

Question, what’s the best way to organize this? What I did before was name the nodes set1-xx, set2-xx,etc.. The problem I faced was when I was trying to run analytics between the current set and the previous set I would have to run a loop through the entire graph and look for all the nodes that started with ‘setx’. It took a long time as the graph grew, so I thought of another solution which was to create a node called ‘set1’ and have it connected to all nodes for that particular set. I am testing it but I was wondering if there way a more efficient way or a build in way of handling data structures like this? Is there a way to somehow segment data like this?

I think a general solution would be application but if it helps I’m using neo4j(so any specific solution to that database would be good as well).

Answer 1

You have a very special type of a directed graph, called a layered graph.

The choice of the data structure depends primarily on the expected graph density (how many nodes from a previous set/layer are typically connected to a node in the current set/layer) and on the operations that you need to perform on it most of the time. It is definitely a good idea to have each layer directly represented by a numeric index (that is, the outermost structure will be an array of sets/layers), and presumably you can also use one array of vertices per layer. However, the list of edges per vertex (out only, or in and out sets of edges depending on whether you ever traverse the layers backward) may be any of the following:

• Linked list of vertex identifiers; this is good if the graph is very sparse and edges are often added/removed.
• Sorted array of vertex identifiers; this is good if the graph is quite sparse and immutable.
• Array of booleans, indexed by vertex identifiers, determining whether a given vertex is or is not linked by an edge from the current vertex; this is good if the graph is dense.

The “vertex identifier” can take many forms. For example, it can be an index into the array of vertices on the next layer.