Question

I’m trying to implement my own triplestore ontop of a SQL database (yes I know there are finished projects out there) and I’m trying to decide on the best way to implement a symbolic “atom”.

In a naive design, we might implement a triplestore in SQL by creating a single “triple” table with three varchar columns called subject, predicate, object. To save space I was going to create an “atom” table, that would store the unique text used in any subject/predicate/object field, and change those fields to foreign keys linking back to the atoms that contain their text.

However, I see a couple ways to implement the Atom table.

1. Store the text as a varchar.

   • Pros: Simple to index and enforce uniqueness of the text.
   • Cons: It could not store arbitrarily large text.

2. Store the text as a text blob, as well as a hash of the text to use when querying and enforcing uniqueness.

   • Pros: Could store arbitrarily large text.
   • Cons: A little more complicated. Possibly, albeit rare, hash collisions depending on the hashing algorithm (md5, sha, etc).

Which is the better approach in terms of performance, long-term reliability, and ability to store any type of data? If I use a hash, is there a valid concern about collisions? Even if collisions are rare, it would only have to happen once to corrupt triplestore.

Answer 1

Don’t waste any time trying to optimize this until you can prove that it’s a bottleneck and is the most important thing to fix.

“To save space…” don’t. Space is almost free. Unless you have over a terabyte of data, you don’t have much to worry about. You can easily waste more time thinking about storage than the storage is worth.

The varchar solution will work and scale fine. The idea of a “string pool” or “atom table” is actually a good one because you’ll have lots of references to the same underlying object. Why repeat the varchar? Why not just repeat an index number?

“Arbitrarily large text” is a strange requirement. Why bother?

A blob will be generally be slower. The hash collision — while little more than a theoretical concern — is something you handle two ways. First, use a hash with more than 32 bits. Second, a collision won’t corrupt anything unless you (foolishly) fail check the actual blobs to see if they’re actually the same. If you want to avoid comparing the entire blob to confirm that there’s no collisions, keep two hashes by different algorithms.