Question

I am developing a web app in Rails that needs to perform computationally intensive data lookup and processing when responding to a request.

I have written a ruby C extension that performs this computation, but to speed things up even greater, I need to create an index.

To keep the index fast, I want to simply use a hash table in C in memory. This hash table will be large. On the order of .5 – 1 GB of memory. I have thought about using a database but I need a very specific format for this index, and ultimately a hash table is fairly simple, and will be very fast.

The index needs to be able to do two things:

1) respond to look up requests
2) respond to requests to add items to the index without having to restart the program

I would prefer to not write a full on C server to build and manage this index.

Is there a way that I can write a ruby C extension with the above two methods, that persists the index in memory in-between calls of the extension?

Is there another way to achieve what I need above without writing a full on server in C?

Worst case, I can write a C server that my rails app can talk to, but I was hoping to find something simpler.

Answer 1

I think hashing has been solved many times. Some flavor of MemcacheD or Redis.
Running it on localhost should make it very fast and persistent (with respect to your app). Redis also has options to back your cache to disk every N updates or N minutes if you really want to keep it.

I would do the heavy lifting in C to calculate said expensive thing. Then stick it in Memcache/Redis when I got it back in Ruby.

If you need to put it in Memcache/Redis while in C, I believe both are written in C so leveraging the libraries shouldn’t be too hard.