Question

I’m trying to compress HTML with JavaScript and decompress it with Ruby. Some carachters are not being processed correctly however and I’m looking for a way to fix this.

My compression function first turns the html in to a byte array using this function. It then compresses the array with the js-deflate library. Finally, the output from that is base64 encoded using window.btoa().

var compress = function(htmlString) {
  var compressed, originalBytes;

  originalBytes = Utils.stringToByteArray(htmlString);
  compressed = RawDeflate.deflate(originalBytes.join(''));
  return window.btoa(compressed);
};

On the Ruby end of things I have a Decompression class which first base64 decodes the compressed html. It then uses the Ruby Zlib standard library to decompress the html. This process is described in this Stack Overflow question thread.

require "base64"
require "zlib"

class Decompression
  def self.decompress(string)
    decoded = Base64.decode64(string)
    inflate(decoded)
  end

private

  def self.inflate(string)
    zstream = Zlib::Inflate.new(-Zlib::MAX_WBITS)
    buf = zstream.inflate(string)
    zstream.finish
    zstream.close
    buf
  end
end

I’m using this class to inflate compressed html, which was sent to a local server, and write it to a file.

decompressed_content = Decompression.decompress(params["compressed_content"])
File.write('decompressed.html', decompressed_content)

Then I open the file in the browser to see if it looks correct.

For the most part, this works fine. I can process the Stack Overflow homepage and it comes out like this:

You can see there is some problems though. Some characters are not coming out correctly, most notably the down arrow beside my name in the header

and the multiplication signs in the recent tags list

How can I fix my code so that these parts of the page are processed properly?

I have tried to force the encoding of the inflated html to UTF-8 but it doesn’t change anything.

  def self.decompress(string)
    decoded = Base64.decode64(string)
    # Forcing the encoding of the output doesn't do anything.
    inflate(decoded).force_encoding('UTF-8')
  end

  def self.decompress(string)
    decoded = Base64.decode64(string)
    # Either does forcing the encoding of the inflate input.
    inflate(decoded.force_encoding('UTF-8'))
  end

One key is that the encoding of the string seems to change to ASCII-8BIT after it is Base64 decoded:

def self.decompress(string)
  p "Before decode: #{string.encoding}"
  decoded = Base64.decode64(string)
  p "After decode: #{decoded.encoding}"
  inflated = inflate(decoded)
  p "After inflate: #{inflated.encoding}"
  inflated
end

# Before decode: UTF-8
# After decode: ASCII-8BIT
# After inflate: ASCII-8BIT

Edits

Someone asked for the method I use to get the html in the first place. I simply pull it off the page with jQuery:

$('html')[0].outerHTML

Edit to show effect of adding a Content-Type meta tag to the inflated html

I added <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> to the inflated html. And I now get question mark boxes like this (Chrome browser by the way):

.

If I inspect the source of my inflated html and compare it to the source of the actual Stack Overflow html I can see that there is a different character being used for the upside-down triangle beside my name.

Actual SO Source: <span class="profile-triangle">&#9662;</span>
Inflated Source without meta Content-Type: <span class="profile-triangle">¾</span>
Inflated source with meta Content-Type: <span class="profile-triangle">�</span>

Answer 1

Try adding UTF8 encoding to the HTML you get. Like in

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" charset="UTF-8" />

I suggest this because I can’t actually see it in the source code (ctrl+u on this same page).

EDIT: Turns out you were missing the charset!