Question

I’m currently developing an application in C# that uses Amazon SQS
The size limit for a message is 8kb.

I have a method that is something like:

public void QueueMessage(string message)

Within this method, I’d like to first of all, compress the message (most messages are passed in as json, so are already fairly small)

If the compressed string is still larger than 8kb, I’ll store it in S3.

My question is:

How can I easily test the size of a string, and what’s the best way to compress it?
I’m not looking for massive reductions in size, just something nice and easy – and easy to decompress the other end.

Answer 1

To know the “size” (in kb) of a string we need to know the encoding. If we assume UTF8, then it is (not including BOM etc) like below (but swap the encoding if it isn’t UTF8):

int len = Encoding.UTF8.GetByteCount(longString);

Re packing it; I would suggest GZIP via UTF8, optionally followed by base-64 if it has to be a string:

    using (MemoryStream ms = new MemoryStream())
    {
        using (GZipStream gzip = new GZipStream(ms, CompressionMode.Compress, true))
        {
            byte[] raw = Encoding.UTF8.GetBytes(longString);
            gzip.Write(raw, 0, raw.Length);
            gzip.Close();
        }
        byte[] zipped = ms.ToArray(); // as a BLOB
        string base64 = Convert.ToBase64String(zipped); // as a string
        // store zipped or base64
    }