Are there any compression algorithms — lossy or lossless — that have been specifically adapted to deal with real-world (messy and invalid) HTML content?
If not, what characteristics of HTML could we take advantage of to create such an algorithm? What are the potential performance gains?
Also, I’m not asking the question to serve such content (via Apache or any other server), though that’s certainly interesting, but to store and analyze it.
Update: I don’t mean GZIP — that’s obvious — but rather an algorithm specifically designed to take advantage of characteristics of HTML content. For example, the predictible tag and tree structure.
I do not know of “off-the-shelf” compression library explicitly optimized for HTML content.
Yet, HTML text should compress quite nicely with generic algorithms (do read the bottom of this answer for better algorithms). Typically all variations on Lempel–Ziv perform well on HTML-like languages, owing to the highly repeatitive of specific language idioms; GZip, often cited uses such a LZ-based algoritm (LZ77, I think).
An idea to maybe improve upon these generic algorithms would be to prime a LZ-type circular buffer with the most common html tags and patterns at large. In this fashion, we’d reduce the compressed size by using citations from the very first instance of such a pattern. This gain would be particularly sensitive on smaller html documents.
A complementary, similar, idea, is to have the compression and decompression methods imply (i.e. not send) the info for other compression’s algorithm of an LZ-x algorithm (say the Huffman tree in the case of LZH etc.), with statistics specific to typical HTML being careful to exclude from characters count the [statistically weighted] instances of character encoded by citation. Such a filtered character distribution would probably become closer to that of plain English (or targeted web sites’ national languge) than the complete HTML text.
Unrelated to the above [educated, I hope] guesses, I started searching the web for information on this topic.
‘ found this 2008 scholarly paper (pdf format) by Przemysław Skibiński of University of Wrocław. The paper’s abstract indicates a 15% improvement over GZIP, with comparable compression speed.
I may be otherwise looking in the wrong places. There doesn’t seem to be much interest for this. It could just be that the additional gain, relative to a plain or moderately tuned generic algorithm wasn’t deemed sufficient enough to warrant such interest, even in the early days of Web-enabled cell phones (when bandwidth was at quite a premium…).