Question

I have just read the ‘dict.c’ source code file of dict implementation. I have got the literal difference between safe/non-safe dict iterator, but don’t yet understand the intent of why introducing the new concept of non-safe iterator.

It’s said by google that ‘the new iterator may perform less useless COW’. But I fail to figure out how it works, so turn to here for help.

Appreciate for any help, an explanation with example would be better.

Answer 1

The purpose of safe/non safe iterators is to clearly establish whether iteration occurs on a mutable or immutable data structure. In the dict implementation, it prevents some operations (rehashing for instance) to be done on the dictionary when a safe iteration is on-going.

Now safe iterators have a downside: they need to increment a reference counter in the iterated object itself, so that the object knows whether safe iteration occurs or not.

Redis leverages the OS copy-on-write (COW) mechanism to deal with background dump. When a dump occurs, fork is called to clone the Redis instance and create a second process. This process will iterate on the whole data to serialize everything into the dump file. Thanks to the COW mechanism, most pages are shared across the two processes, so Redis does not take twice the RAM while it is dumping the data.

Now the pages are only shared if they are accessed in read-only mode. When one of the two processes writes something in memory, the corresponding page is automatically duplicated by the OS.

If a safe iterator was systematically used to iterate on all dictionaries, plenty of pages would be duplicated (because of the reference counter updates). In the cloned process, the data are considered immutable, so using an unsafe iterator makes sense to reduce COW activity. This is mostly relevant when you have a lot of set/hash/zset objects in Redis.