Question

I want to load data with 4 columns and 80 millon rows in MySQL on Redis, so that I can reduce fetching delay.

However, when I try to load all the data, it becomes 5 times larger.

The original data was 3gb (when exported to csv format), but when I load them on Redis, it takes 15GB… it’s too large for our system.

I also tried different datatypes –

1) ‘table_name:row_number:column_name’ -> string
2) ‘table_name:row_number’ -> hash

but all of them takes too much.

am I missing something?

added)

my data have 4 col – (user id(pk), count, created time, and a date)

Answer 1

The most memory efficient way is storing values as a json array, and splitting your keys such that you can store them using a ziplist encoded hash.

1. Encode your data using say json array, so you have key=value pairs like user:1234567 -> [21,'25-05-2012','14-06-2010'].
2. Split your keys into two parts, such that the second part has about 100 possibilities. For example, user:12345 and 67
3. Store this combined key in a hash like this hset user:12345 67 <json>
4. To retrieve user details for user id 9876523, simply do hget user:98765 23 and parse the json array
5. Make sure to adjust the settings hash-max-ziplist-entries and hash-max-ziplist-value

Instagram wrote a great blog post explaining this technique, so I will skip explaining why this is memory efficient.

Instead, I can tell you the disadvantages of this technique.

1. You cannot access or update a single attribute on a user; you have to rewrite the entire record.
2. You’d have to fetch the entire json object always even if you only care about some fields.
3. Finally, you have to write this logic on splitting keys, which is added maintenance.

As always, this is a trade-off. Identify your access patterns and see if such a structure makes sense. If not, you’d have to buy more memory.