Question

I’m matching a bunch of strings using REDIS.hmget. The full store has about ~140k keys. I’ve seen my backend match over 1k strings in less than 1s, but I came accross this particular case where the matching times out before finishing. I get the follow crash trace:

Errno::ETIMEDOUT: Connection timed out
.bundle/gems/ruby/1.8/gems/redis-2.2.2/lib/redis/connection/ruby.rb:58:in `write'
.bundle/gems/ruby/1.8/gems/redis-2.2.2/lib/redis/connection/ruby.rb:58:in `write'
.bundle/gems/ruby/1.8/gems/redis-2.2.2/lib/redis/client.rb:139:in `process'
.bundle/gems/ruby/1.8/gems/redis-2.2.2/lib/redis/client.rb:138:in `each'
.bundle/gems/ruby/1.8/gems/redis-2.2.2/lib/redis/client.rb:138:in `process'
.bundle/gems/ruby/1.8/gems/redis-2.2.2/lib/redis/client.rb:250:in `ensure_connected'
.bundle/gems/ruby/1.8/gems/redis-2.2.2/lib/redis/client.rb:137:in `process'
.bundle/gems/ruby/1.8/gems/redis-2.2.2/lib/redis/client.rb:206:in `logging'
.bundle/gems/ruby/1.8/gems/redis-2.2.2/lib/redis/client.rb:136:in `process'
.bundle/gems/ruby/1.8/gems/redis-2.2.2/lib/redis/client.rb:46:in `call'
.bundle/gems/ruby/1.8/gems/redis-2.2.2/lib/redis.rb:755:in `hmget'
/usr/ruby1.8.7/lib/ruby/1.8/monitor.rb:242:in `synchronize'
.bundle/gems/ruby/1.8/gems/redis-2.2.2/lib/redis.rb:754:in `hmget'

Although it times out, hmget still returns what it has matched so far, which seems to vary between 2 and 2.3k strings. Like I said earlier, I’ve seen the backend match over 1k strings in less than 1 second, so I have a hard time to conceive that 2.3k strings makes Redis timeout, unless it times out only after a couple seconds.

Is there anything I might me missing here? Should I be breaking the match into 2 smaller ones? Is there a way to change the default timeout interval?

Answer 1

I ended up splitting the lookup in small batches and haven’t got the problem since. Something like:

BATCH_MATCHING_COUNT = 1000
values = Array.new
i = 0

while i < keys.length
    values.push(REDIS.hmget("hash", *keys[h, BATCH_MATCHING_COUNT]))
    i+=1
end