At work we want our next generation product to be based on a graph database. I’m looking for suggestions as to what database engine might be appropriate for our new project:
Out product is intended to keep track of a large number of prices for goods. Here’s a simplistic example of what it does – supposing you wanted to estimate the price of gasoline in the UK – you know that Gasoline is refined from crude-oil. If you new the price of crude oil in the UK you could estimate the price of anything simply by adding the cost of refining, transporting (etc). Actually things are more complex because there are a number of sources of crude-oil and hundreds of refined oil products. The prices of oil products can be affected by the availability of other energy sources (e.g. nuclear, wind, natural gas) and the demand. It’s kind of complex!
The idea is that we want to model the various inter-related goods and their costs of refining, transportation (etc) as an asyclic directed graph. The idea being, when an event causes a price to change then we want to be quickly able to determine what kinds of things are affected and re-calculate those prices ASAP.
Essentially we need a database which can represent the individual commodities as nodes in the graph. Each node will store a number of curves and surfaces of information pertaining to the product.
We want to represent the various costs & transformations (e.g. refining, transportation) as labels on the edges. As with the nodes, the information we want to store could be quite complex – not just single values but curves and surfaces.
The calculations we do are all linear with respect to the size of the objects, however since the graph could be very big we need to be able to traverse the graph very quickly.
We are Java and Python centric – ideally we are after a product that runs on the JVM but has really good APIs for both Python and Java. We don’t care so much about other languages… but .Net would be nice to have (even though it might be years before we get round to doing something with it).
We’d definitely like something which was high-performance – but more importantly the system needs to have a degree of hardware fault tolerance. For example, we’d like to distribute the database across a number of physical servers. In the event that any of the servers go down we’d like to be able to continue without an interruption.
Oh, and we are really lazy. We dont want to spend much time writing infrastructure – so if the database came with tools that allow us to do as much as possible of this kind of thing with very little work that’s fine by us. It would also be a real bonus if there was a grid technology associated with the graph DB, that way we could push a sequence of re-calculate jobs onto a compute grid and have much of our calculation done in paralell.
So, that’s a description of the kind of thing we want to build. What I want to know is whether there are any mature technologies which will help us achieve this. As I mentioned before, we have a preference for Python & JVM, however if the technology is really good and comes with great bindings for Python + Java we’d consider almost anything.
Neo4J is the most mature graphDB I know of – and is java, with bindings for python too, or REST