I am considering enabling compression (setting compressed = True) on some big LocalStructuredProperty properties.
However, those are entities that will be often read and written, so I am concerned about the additional CPU required to encode/decode these properties.
I am wondering if ndb’s built-in caching system would alleviate this? Does ndb cache the compressed or uncompressed data for these properties? I realize that compression will always occur on every write, but if I can at least bypass decompression when reading a cached entity, it would probably be worth it.
NDB uses lazy decompression and compression, and if you have an entity that is cached in memory (in the Context object) it will only decompress when you actually access the property for the first time. Also if you read an entity and write it back without ever accessing the compressed property, it won’t be decompressed and compressed at all (regardless of caching).
However the values are written to memcache in compressed form, so if you actually access the compressed properties, the caching won’t help you much.
My recommendation: use compression only if the data would be too big to fit without it (the limit is ~1MB for the entire entity) and then only if it is the kind of data that compresses well (e.g. text, but not images or other media, since those formats already have their own media-specific compression). Log lines compress very well. English text or computer source code compresses pretty well.