Question

I’ve got a huge list of keys and one value (the same for all keys) that should be written to a redis instance. This should only be happen, if an entry with this key already exists.

There is the MSETNX command which is doing the opposite: putting in the entry, when the key does NOT exist.

Sure, i can just check before via EXIST and set it depending on the result, but how do i ensure that this atomic. And as this list can be really huge, how do i do this the pipeline-based?

i’am using redis-py as wrapper and there’s no Lua-Scripting support available on the server.

Answer 1

This is not extremely convenient to do without Lua, but it can still be done using WATCH/MULTI/EXEC commands. The idea is to partition your huge list of keys in blocks of n items (n=20 for instance), and then apply the following function to each block:

def process_block( redis_client, block, value ):
    with redis_client.pipeline() as pipe:
        while True:
            try:
                pipe.watch( block )
                to_set = [ for x in block if pipe.exists(x) ]
                if to_set:
                    pipe.multi()
                    for x in to_set:
                        pipe.set( x, value )
                    pipe.execute()
                break
            except redis.WatchError:
                continue
            finally:
                pipe.reset()

Atomicity is guaranteed by the WATCH/MULTI/EXEC commands, and you will pipeline 1 WATCH, 1 MULTI, up to n SET, and 1 EXEC commands. You will pay for the EXISTS roundtrips though. Choosing the correct value for n is a tradeoff between the pipelining factor and the risk of concurrent interactions. It should not be set too high.