Question

Some pieces of our app are written in Ruby and others are written using node.js.

We share data among them using a redis store that stores zlib chunks.
We write to it with the following code using node:

zlib.deflate(xml.toString(), function(error, deflated) {
  ...
  deflated.toString('binary'); // That's the string we write in Redis
  ...
});

Now, we read this data in the redis store using Ruby (1.8.7) and I have to say I’m not sure how to do that.

The typical string we get from the store looks like this:

=> "xuAo \020ÿ\ná.£v½\030dÿCO½±:«¤(\004ƪÿ¾¬®5MÚ\003÷½IÞ q¤°²e°c¼òÈ×\000ó<ùM¸ÐAç\025ÜÈ\r|gê\016Ý/.é\020ãÆî×\003Ôç<Ýù2´F\n¨Å\020!zl \0209\034p|üÀqò\030\036m\020\e`\031¼ÏütÓ=ø¦U/ÔO±\177zB{\037½£-ðBu©ò¢X\000kb­*Ó[V\024Y^½EÎ¥üpúrò­¦\177ÁÃdÈ¢j\0353$a\027²q#¥]*Ýi3J8¤´füd\eså[³öʵ%\fcÇY\037ð¬ÿg§í^¥8£Õ§a¶\001=\r;¡¾\001\020Pí" 

Of course, I tried using Zlib::Inflate.new.inflate(compressed) but that fails with a Zlib::DataError: incorrect header check.

Any idea on what kind of transformation we should do to that string to inflate it from Ruby?

PS: inflating it from node is easy and works, so the problem is not how we compress it.

Answer 1

Any idea on what kind of transformation we should do to that string to
inflate it from Ruby?

UTF-8 to Latin-1

Ideally, there would be no need for any transformation, as long as you work with Buffers directly on the Node side. See pair of Node and Ruby code blocks at the very bottom below; however, the nature of the question is about what can be done on the Ruby side alone to address this.

Ruby-only – Convert from UTF-8 to LATIN-1

require 'zlib'
require 'rubygems'
require 'redis'
require 'iconv'

redis = Redis.new

def inflate(buffer)
    zstream = Zlib::Inflate.new
    buf = zstream.inflate(buffer)
    zstream.finish
    zstream.close
    buf
end


def convert(buffer)
    utf8_to_latin1 = Iconv.new("LATIN1//TRANSLIT//IGNORE", "UTF8")
    utf8_to_latin1.iconv(buffer) 
end

value = redis.get("testkey")
value = convert(value)
puts inflate(value);

Explanation

The above code uses iconv to convert the value retrieved from Redis from UTF-8 back to the intended bytes.

When deflating in Node, the resulting buffer contains the correct zlib generated bytes; the result string from toString('binary'), character for character matches the contents of the deflate result buffer as well; however, by the time the deflate result is stored in Redis, it is UTF-8 encoded. An example:

deflating the string “ABCABC” results in:

<Buffer 78 9c 73 74 72 76 74 72 06 00 05 6c 01 8d>

Yet, Redis returns:

<Buffer 78 c2 9c 73 74 72 76 74 72 06 00 05 6c 01 c2 8d>

Hypothesizing a bit, it would seem that the string resulting from toString('binary') ends up as argument to new Buffer(…) somewhere, perhaps in node-redis. In the absence of a specified encoding argument to new Buffer(), the default UTF-8 encoding is applied. (See first reference). Further hypothesizing, by using only buffers you avoid the need to create a buffer from the string, and as a result, avoid the UTF8 encoding, and so the correct deflate values make it in and out of Redis.

References

• http://nodejs.org/api/buffer.html#buffer_new_buffer_str_encoding
• http://blog.grayproductions.net/articles/encoding_conversion_with_iconv
• http://ruby-doc.org/stdlib-1.8.7/libdoc/zlib/rdoc/Zlib/Inflate.html.

Node

var zlib = require('zlib');
var redis = require("redis").createClient();

var message = new Buffer('your stuff goes here.');
//var message = new Buffer(xml.toString());

redis.on("error", function (err) {
console.log("Error " + err);
});

redis.on("connect", function() {
    console.log(message);
    zlib.deflate(message, function(error, deflated) {
        console.log(deflated);          
        redis.set("testkey",deflated,function (err, reply) {
            console.log(reply.toString());
        });
    });
});

Ruby

require 'zlib'
require 'rubygems'
require 'redis'

redis = Redis.new

def inflate(buffer)
    zstream = Zlib::Inflate.new
    buf = zstream.inflate(buffer)
    zstream.finish
    zstream.close
    buf
end

value = redis.get("testkey")    

puts inflate(value)