Question

I’m trying to make a html entities encoder/decoder on Python that behaves similar to PHP’s htmlentities and html_entity_decode, it works normally as a standalone script:

My input:

Lorem ÁÉÍÓÚÇÃOÁáéíóúção @#$%*()[]<>+ 0123456789

python decode.py

Output:

Lorem ÁÉÍÓÚÇÃOÁáéíóúção @#$%*()[]<>+ 0123456789

Now if I run it as an Autokey script I get this error:

Script name: 'html_entity_decode'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/autokey/service.py", line 454, in execute
    exec script.code in scope
  File "<string>", line 40, in <module>
  File "/usr/local/lib/python2.7/dist-packages/autokey/scripting.py", line 42, in send_keys
    self.mediator.send_string(keyString.decode("utf-8"))
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-12: ordinal not in range(128)

What am I doing wrong? Here’s the script:

import htmlentitydefs
import re

entity_re = re.compile(r'&(%s|#(\d{1,5}|[xX]([\da-fA-F]{1,4})));' % '|'.join(
    htmlentitydefs.name2codepoint.keys()))

def html_entity_decode(s, encoding='utf-8'):

    if not isinstance(s, basestring):
        raise TypeError('argument 1: expected string, %s found' \
                        % s.__class__.__name__)

    def entity_2_unichr(matchobj):
        g1, g2, g3 = matchobj.groups()
        if g3 is not None:
            codepoint = int(g3, 16)
        elif g2 is not None:
            codepoint = int(g2)
        else:
            codepoint = htmlentitydefs.name2codepoint[g1]
        return unichr(codepoint)

    if isinstance(s, unicode):
        entity_2_chr = entity_2_unichr
    else:
        entity_2_chr = lambda o: entity_2_unichr(o).encode(encoding,
                                                           'xmlcharrefreplace')
    def silent_entity_replace(matchobj):
        try:
            return entity_2_chr(matchobj)
        except ValueError:
            return matchobj.group(0)

    return entity_re.sub(silent_entity_replace, s)

text = clipboard.get_selection()
text = html_entity_decode(text)
keyboard.send_keys("%s" % text)

I found it on a Gist https://gist.github.com/607454, I’m not the author.

Answer 1

Looking at the backtrace the likely problem is that you are passing in a unicode string to keyboard.send_keys, which expects a UTF-8 encoded bytestring. autokey then tries to decode your string which fails because the input is unicode instead of utf-8. This looks like a bug in autokey: it should not try to decode strings unless their are really plain (byte)sstrings.

If this guess is correct you should be able to work around this by making sure you pass a unicode instance to send_keys. Try something like this:

text = clipboard.get_selection()
if isinstance(text, unicode):
    text = text.encode('utf-8')
text = html_entity_decode(text)
assert isinstance(text, str)
keyboard.send_keys(text)

The assert is not needed but is a handy sanity check to make sure html_entity_decode does the right thing.