Sorry that the title isn’t exactly obvious, but I couldn’t word it better.
We are right now using a conventional DB (oracle) as our job queue, and these “jobs” are consumed by some number of nodes (machines). So the DB server gets hit by these nodes, and we have to pay a lot for the software and hardware for this database server.
Now, it occurred to me the other day that,
1) There are already multiple nodes in the system
2) “Jobs” may not be lost because of node failures, but there is no reason they have to be sitting in a secondary storage (no reason why they couldn’t reside in memory, as long as they are not lost)
Given this, couldn’t one retain these jobs in-memory, making sure that at least n number of copies of this job is present in the entire cluster, thereby getting rid of the DB server?
Are such technologies available?
Did you take a look at Gigaspaces? On an internet scale, you do not need to persist at all. You just have to know sufficient copies are around. If you have low latency connections to places that are not on the same powergrid (or have battery power), pushing out your transactions to the duplicates is enough.