Question

I’m working on Cassandra Hadoop integration (MapReduce). We have used RandomPartitioner to insert data to gain faster write speed. Now we have to read that data from Cassandra in MapReduce and perform some calculations on it.

From the lots of data we have in cassandra we want to fetch data only for particular row keys but we are unable to do it due to RandomPartitioner – there is an assertion in the code.

Can anyone please guide me how should I filter data based on row key on the Cassandra level itself (I know data is distributed across regions using hash of the row key)?

Would using secondary indexes (still trying to understand how they works) solve my problem or is there some other way around it?

I want to use cassandra MR to calculate some KPI’s on the data which is stored in cassandra continuously. So here fetching whole data from cassandra every time seems an overhead to me? The rowkey I’m using is like “(timestamp/60000)_otherid”; this CF contains reference of rowkeys of actual data stored in other CF. so to calculate KPI I will work for a particular minute and fetch data from other CF, and process it.

Answer 1

When using RandomPartitioner, keys are not sorted, so you cannot do a range query on your keys to limit the data. Secondary indexes work on columns not keys, so they won’t help you either. You have two options for filtering the data:

Choose a data model that allows you to specify a thrift SlicePredicate, which will give you a range of columns regardless of key, like this:

SlicePredicate predicate = new SlicePredicate().setSlice_range(new SliceRange(ByteBufferUtil.bytes(start), ByteBufferUtil.bytes(end), false, Integer.MAX_VALUE));
ConfigHelper.setInputSlicePredicate(conf, predicate);

Or use your map stage to do this by simply ignoring input keys that are outside your desired range.