Home/ Questions/Q 4052500
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-20T14:19:40+00:00

I want to store web pages in compressed text files (CSV). To achieve the

  • 0

I want to store web pages in compressed text files (CSV). To achieve the optimal compression, I would like to provide a set of 1000 web pages. The library should then spend some time creating the optimal “dictionary” for this content. One obvious “dictionary” entry could be <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">, which could get stored as %1 or something like that because it is present on almost all web pages. By creating a customized dictionary like this, the compression rates should be 99% in my case.

My question is, does a library for doing this exist on Windows with MIT or similar liberal licensing exist? If not, are there any general purpose compression libaries you would recommend. I have tried a bit with zlib, but it outputs binary data. If I would convert this binary data into text, I am worried that the result might be longer than the original text.

EDIT: I need to be able to store the text in CSV files and still be able to import them into a database or even Excel.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    1. “text files (not binary)” is a little too general. If you mean that some
      byte values (00,1A or whatever) can’t be used, then any binary method +
      something like base64 coding can be used. (Although I’d suggest a more efficient method
      from Coroutine demo source).

      To be specific, you can use any general-purpose compressor to compress your
      base file, then base file + target file, then diff these, and you’d get
      a dictionary compression (binary), which can be then converted to “text”
      with base64 or yenc or whatever.

      Alternatively, there’re some coders with build-in support for that, for example
      http://compression.ru/ds/ppmtrain.rar
      http://code.google.com/p/lzham/

    2. If you actually want to have common phrases replaced with references, and
      all other things left untouched (what is kinda implied, but not equals to “text output”),
      you can use text preprocessors like:
      http://xwrt.sourceforge.net/
      http://compression.ru/ds/liptify.rar
      (There were more afair).

    3. Also a hybrid method is possible. You can use a general-purpose LZ compressor like in [1], for example lzma, then replace its entropy coding with something text-based.
      For example, in http://nishi.dreamhosters.com/u/lzmarec_v1_bin.rar
      there’s an utility which removes LZMA’s entropy coding, and its pretty easy to convert
      its output to text.

    • 0