I am currently benchmarking Cassandra with 3 nodes using CassandraSharp.
My main concern is more latency than throughput, so after a bit of GC tuning here are my numbers (on 100 000K inserts, mono-thread):
- Iter/sec: 1600
- Average: 600µs
- 95 cent: 600µs
- 99 cent: 5000µs
- Max: 50 000µs
My problem here is that once in a while I get a “bad” latency (50ms), my goal being to have consistent latency, even at the cost of a higher average.
I believe that this is caused by the GC, and I’m wondering if it could be avoided.
(As a side note, is it a good practice to send a big amount of inserts to one node and let it handle it or should I “load balance” it in the client?)
50ms is within the normal range for a young-generation garbage collection. You can enable GC logging in cassandra-env.sh by uncommenting the appropriate lines towards the bottom to verify that this is the problem.
(Flushes do not block inserts unless your disk is so slow it can’t keep up with insert volume, which is unusual since flushes are sequential i/o.)
If young generation collections are indeed correlated with the higher latencies, you can reduce try making the young generation smaller (also configured in cassandra-env.sh), at the potential cost of trading latency for throughput.