Question

I have a form (struts 1) that is being validated and during the validation I have been asked to remove MS Word’s curly single and double quote marks. Seems like such a simple request and I am tearing my hair out over it.

My test text is ’ “ ”.

First of all, I found that when I run my code in the debugger and watch what IntelliJ thinks the values are, it displays â\u0080\u0099 â\u0080\u009C â\u0080\u009D and it seems that â\u0080 are nonprinting characters.

I used a piece of code that iterates over a StringBuilder of the text in the field and tests each char in the text. It replaces or deletes some chars, as below:

switch (origCharAsInt) {

    case ((int)'\u00C2'): sbOriginal.deleteCharAt(isb); break; // weird Word A with the caret over it
    case ((int)'\u00C3'): sbOriginal.deleteCharAt(isb); break; // weird Word A with the tilde over it
    case ((int)'\u00E2'): sbOriginal.deleteCharAt(isb); break; // weird Word a with the caret over it

    case ((int)'\u0099'): sbOriginal.setCharAt(isb, '\''); break; // Word single quote
    case ((int)'\u009C'): sbOriginal.setCharAt(isb, '"'); break; // Word left double quote
    case ((int)'\u009D'): sbOriginal.setCharAt(isb, '"'); break; // Word right double quote

    case ((int)'\u2018'):  sbOriginal.setCharAt(isb, '\''); break;  // left single quote
    case ((int)'\u2019'):  sbOriginal.setCharAt(isb, '\''); break;  // right single quote
    case ((int)'\u201A'):  sbOriginal.setCharAt(isb, '\''); break;  // lower quotation mark

    case ((int)'\u201C'):  sbOriginal.setCharAt(isb, '"'); break;  // left double quote
    case ((int)'\u201D'):  sbOriginal.setCharAt(isb, '"'); break;  // right double quote
    case ((int)'\u201E'):  sbOriginal.setCharAt(isb, '"'); break;  // double low quotation mark

    case ((int)'\u2039'):  sbOriginal.setCharAt(isb, '\''); break;  // Single Left-Pointing Quotation Mark
    case ((int)'\u203A'):  sbOriginal.setCharAt(isb, '\''); break;  // Single right-Pointing Quotation Mark

    default: break;

}

This seems to work, in that it replaces some of the more egregious cruft, and the form now appears to contain ' " ". If I then save again, though, the IntelliJ thinks the field contains Â\u0080 Â\u0080\" Â\u0080\".

So I added a few more cases to remove those  characters. But I’m flummoxed by the persistence of the \u0080 characters.

I tried adding in a few more cases to try to remove them, but they did not work.

case ((int)'\u0080'): sbOriginal.deleteCharAt(isb); break; // another weird Word non-printing char
case ((int)'\u0082'): sbOriginal.deleteCharAt(isb); break; // another weird Word non-printing char
case ((int)'\u0083'): sbOriginal.deleteCharAt(isb); break; // another weird Word non-printing char
case ((int)'\u0000'): sbOriginal.deleteCharAt(isb); break; // why are these weird symbols showing up?

Any help/explanation would be greatly appreciated.

Answer 1

You need to escape HTML properly. Unicode isn’t the answer here. This link is your jam.