Question

How do I remove Unicode characters from a bunch of text files in the terminal?

I’ve tried this, but it didn’t work:

sed 'g/\u'U+200E'//' -i *.txt

I need to remove these Unicode characters from the text files:

U+0091 - sort of weird "control" space
U+0092 - same sort of weird "control" space
A0 - non-space break
U+200E - left to right mark

Answer 1

If you want to remove only particular characters and you have Python, you can:

CHARS=$(python -c 'print u"\u0091\u0092\u00a0\u200E".encode("utf8")')
sed 's/['"$CHARS"']//g' < /tmp/utf8_input.txt > /tmp/ascii_output.txt