Question

I’ve been using Neo4j for a little while now and have an app up and running using Neo4j, its all working really well and Neo4j has been really cool at solving this problem, but I now need to extend the app and having been trying to impl. a Key-Value List of data into Neo4j and I’m not sure the best way to go about it.

I have a List, the list is around 7 million elements in length and so a bit long for just storing the whole list in memory and managing it myself. I tested this and it would consume 3Gb.

My choices are either:

• (a) Neo4j is just the wrong tool for the job and I should use an actual key-value data store. A little adverse to do this as I’d have to introduce another data store just for this list of data.
• (b) Use Neo4j, by creating a node per key-value setting the key and value as properties on the node, but there is no relationship other then having an index to group these nodes together, exposing the key of the key-value as the key on the index.
• (c) Create a single node and hold all key-values as properties, this feels wrong, because when getting the node it will load the whole thing into memory, then I’d have to search the properties for the one I’m interested in, and I might as well manage the List myself.
• (d) The key is a two part key that actually points to two nodes, so create a relationship and set the value as a property on the relationship. I started down this path, but when it came to doing a lookup for a specific key/value it’s not simple and fast, so backed away from this.

Either options ‘a’ or ‘b’ feel the way to go.
Any advice would be appreciated.

Example scenario

We have Node A and Node B which has a relationship between the two Nodes.
The nodes all have a property of ‘foo’, with foo having some value.
In this example node A has foo=X and Node B has foo=Y

We then have this list of K/Vs. One of those K/V is Key:X+Y=Value:Z
So, the original idea was to create another relationship between Node A and Node B and store a property on the relationship holding Z. Then create an index on ‘foo’ and a relationship idx on the new relationship.

When given Key X+Y get the value.
Lookup logic would be get Node A (from X) and Node B (from y), then walk through Node A relationships to Node B lookup for this new relationship type.
While this will work, I do not like the fact I have to lookup through all relationships to/from the nodes looking for a specific type this is inefficient. Especially if there are many relationships of different types.

So the conclusion to go with options ‘A’ or ‘B’, or I’m trying to do something impractical with Neo.

Answer 1

Don’t try to store 7 million items in a Neo4j property — you’re right, that’s wrong.

Redis and Neo4j often make a good pairing, but I don’t quite understand what you’re trying to do or what you mean in “d” — what are the key/value pairs, and how do they relate to the nodes and relationships in the graph? Examples would help.

UPDATE: The most natural way to do this with a graph database is to store it as a property on the edge between the two nodes. Then you can use Gremlin to get its value.

For example, to return a property on an edge that exists between two vertices (nodes) that have some properties:

start = g.idx('vertices')[[key:value]]            // start vertex
edge = start.outE(label).as('e')                  // edge
end = edge.inV.filter{it.someprop == somevalue}   // end vertex
prop = end.back('e').prop                         // edge property
return prop

You could store it in an index like you suggested, but this adds more complexity to your system, and if you need to reference the data as part of the traversal, then you will either have to store duplicate data or look it up in Redis during the traversal, which you can do, see:

Have Gremlin Talk to Redis in Real Time while It’s Walking the Graph
https://groups.google.com/d/msg/gremlin-users/xhqP-0wIg5s/bxkNEh9jSw4J

UPDATE 2:

If the ID of vertex a and b are known ahead of time, then it’s even easier:

g.v(a).outE(label).filter{it.inVertex.id == b}.prop

If vertex a and b are known ahead of time, then it’s:

a.outE(label).filter{it.inVertex == b}.prop