Question

This question is similar to the Remove html character entities in a string question asked earlier on Stack Overflow. The accepted answer, however, does not address the issue of named HTML entities, e.g. ä for the character ä; It therefore cannot unescape all HTML.

I have some legacy HTML which uses named HTML entities for non-ASCII characters. That is, ö instead of ö, ä instead of ä and so on. A full list of all named HTML entities is available on Wikipedia.

I’d like to unescape these HTML entities into their character equivalents, in a quick and efficient manner.

---

I have the code to do this in Python 3, using regular expressions:

import re
import html.entities

s = re.sub(r'&(\w+?);', lambda m: chr(html.entities.name2codepoint[m.group(1)]), s)

Regular expressions, however, don’t seem very popular, fast or easy to use in Haskell.

---

Text.HTML.TagSoup.Entity (tagsoup) has a useful table and functions for mapping named entities tpo codepoints. Using this, and the regex-tdfa package, I’ve fashioned an extremely slow equivalent in Haskell:

{-# LANGUAGE OverloadedStrings #-}
import Data.ByteString.Lazy.Char8 as L
import Data.ByteString.Lazy.UTF8 as UTF8
import Text.HTML.TagSoup.Entity (lookupEntity)
import Text.Regex.TDFA ((=~~))

unescapeEntites :: L.ByteString -> L.ByteString
unescapeEntites = regexReplaceBy "&#?[[:alnum:]]+;" $ lookupMatch
 where
  lookupMatch m =
    case lookupEntity (L.unpack . L.tail . L.init $ m) of
      Nothing -> m
      Just x -> UTF8.fromString [x]

-- regex replace taken from http://mutelight.org/articles/generating-a-permalink-slug-in-haskell
regexReplaceBy :: L.ByteString -> (L.ByteString -> L.ByteString) -> L.ByteString -> L.ByteString
regexReplaceBy regex f text = go text []
 where
  go str res =
    if L.null str
      then L.concat . reverse $ res
      else
        case (str =~~ regex) :: Maybe (L.ByteString, L.ByteString, L.ByteString) of
          Nothing -> L.concat . reverse $ (str : res)
          Just (bef, match , aft) -> go aft (f match : bef : res)

The unescapeEntities function runs several orders of magnitude slower than the Python version above. The Python code can convert about 130 MB in 7 seconds, whereas my Haskell version runs for several minutes.

I’m looking for a better solution, primarily in terms of speed. But I’d also like to avoid regular expressions, if possible (speed and avoiding regular expressions seem to go hand in hand in Haskell anyway).

Answer 1

Here’s my version. It uses String (instead of ByteString).

import Text.HTML.TagSoup.Entity (lookupEntity)

unescapeEntities :: String -> String
unescapeEntities [] = []
unescapeEntities ('&':xs) = 
  let (b, a) = break (== ';') xs in
  case (lookupEntity b, a) of
    (Just c, ';':as) ->  c  : unescapeEntities as    
    _                -> '&' : unescapeEntities xs
unescapeEntities (x:xs) = x : unescapeEntities xs

I would guess it’s faster because it doesn’t use the expensive regex operations. I haven’t tested it. You could adapt it for ByteString or for Data.Text if you need it faster.