Question

How do I ensure UTF8 is used for a shared preference menu? I have an android preferences menu that allows a user to set their name, amongst other things.

I need to know how to convert the data stored in shared preferences into UTF8 format

The preference menu is laid out in xml using utf8 encoding in a file called settings in the res/xml folder and looks like this:

<?xml version="1.0" encoding="utf-8"?>
<PreferenceScreen
    xmlns:android="http://schemas.android.com/apk/res/android">
    <EditTextPreference
        android:key="@string/name_key"
        android:title="@string/hof_name_title"
        android:summary="@string/hof_name_summary"
        android:defaultValue="Anonymous" />
    <CheckBoxPreference
        android:key="@string/new_game_preference_key"
        android:title="@string/new_game_preference_title"
        android:summary="@string/new_game_preference_summary"
        android:defaultValue="false" />
</PreferenceScreen>

The class that handles this is

public class PrefsActivity extends PreferenceActivity implements OnSharedPreferenceChangeListener{
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        requestWindowFeature(Window.FEATURE_NO_TITLE);
        getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN,
                WindowManager.LayoutParams.FLAG_FULLSCREEN);
        super.onCreate(savedInstanceState);
        addPreferencesFromResource(R.xml.settings);
    }

    @Override
    protected void onPause() {
        super.onPause();
        // Unregister the listener whenever a key changes
        getPreferenceScreen().getSharedPreferences()
                .unregisterOnSharedPreferenceChangeListener(this);
    }

    @Override
    protected void onResume() {
        super.onResume();
        // Set up a listener whenever a key changes
        getPreferenceScreen().getSharedPreferences()
                .registerOnSharedPreferenceChangeListener(this);
    }

    @Override
    public void onSharedPreferenceChanged(SharedPreferences sharedPreferences,
            String key) {
//      Util.log_debug_message("@@@@ Changed prefrence key = " + key);
        if (key.equalsIgnoreCase(getString(R.string.name_key))){
            update_webserver();
        }else if (key.equalsIgnoreCase(getString(R.string.new_game_preference_key))){
            update_webserver();
        }
    }

    protected void update_webserver(){
        Intent webServerReg = new Intent(this, RegService.class);
        webServerReg.putExtra(Config.C2DM_REG_ID_EXTRA, C2DMessaging.getRegistrationId(this));
        startService(webServerReg);     
    }
}

I assumed that setting <?xml version="1.0" encoding="utf-8"?> would ensure that text would be encoded in UTF8 format but this is not always the case. I am getting values in all sorts of formats. Some examples

“\nFe\xF1a”, “L\xFAcia”, “$’\xE5 \xEE'”,

or

“:-$B-)O:-):-P=-OB-):-D:-Q:-X:-!:-|\n:'(:-*XDo_O:-X:-C:-X:O:-X=-O;-):-):-);-):-D:OX-(o_O:-V:-@:-V:-X:-Do_O::-C
\xBF\xA1\xA1\xAB\xBB\xAE\xA9^\xA5\xA5?[\xA2}?\xA2\xA2\xA5\xA3?$\xBF\xA1\xAE\xA7><[\xA3~?=~~\xB0]\xA3?\xA7\xA1\\\xAB\xBB\xAE^``]]||||}{_|]|||\xB0\xB0?”

Yes that is one loooong set of rubish in the last example.

The problems I have start when trying to send the nmes as a json HTTP POST or PUT request to my webserver using the following code

public void update(String regId) throws AuthenticationException {
    JSONObject mu_to_send = createJsonToPost(regId, false);
    HttpResponse response;
    try {
        response = HttpRequest.httpPutJSONObject(this,
                Config.UPDATE_USER_URL, mu_to_send);
        int responseCode = response.getStatusLine().getStatusCode();
        String responseJson;
        try {
            responseJson = EntityUtils.toString(response.getEntity());
            String msg = "";

            if (responseCode == 201) {
                // mobile_user = new
                // JSONObject(responseJson).getJSONObject("mobile_user");
                // Broadcast.sendBroadcast(this,
                // ResponseReceiver.ACTION_RESP,
                // Intent.CATEGORY_DEFAULT, Config.C2DM_MESSAGE_EXTRA,
                // Config.BROADCAST_WEBSERVER_USER_CREATED);
            } else {
                msg = "Failed to register with server! HTTP response code = "
                        + responseCode;
            }
        } catch (ParseException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    } catch (ClientProtocolException e1) {
        // TODO Auto-generated catch block
        e1.printStackTrace();
    } catch (IOException e1) {
        // TODO Auto-generated catch block
        e1.printStackTrace();
    }

}

public JSONObject createJsonToPost(String regId, boolean set_password) {
    JSONObject holder = new JSONObject();
    JSONObject data = new JSONObject();
    try {
        data.put("auth", regId);
        data.put("name", C2DMessaging.getName(this));
        data.put("receive_new_game_notification",
                C2DMessaging.getGameStartedNotificationPreference(this));
        holder.put("mobile_user", data);
    } catch (JSONException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    return holder;
}

public static HttpResponse httpPutJSONObject(Context context, String url,
        JSONObject data) throws ClientProtocolException, IOException,
        AuthenticationException {
    DefaultHttpClient httpclient = getHttpClient(context);
    HttpPut httpput = new HttpPut(url);
    httpput.addHeader("User-Agent", Config.USER_AGENT);
    httpput.addHeader("Accept", "application/json");
    httpput.addHeader("Content-type", "application/json");
    httpput.addHeader(new BasicScheme().authenticate(
            getCredentials(context), httpput));
    StringEntity se = new StringEntity(data.toString());

    httpput.setEntity(se);
    return httpclient.execute(httpput);
}

This causes all sorts of issues on my web server which expects valid JSON (i.e. UTF8)
as JSON should be sent encoded in UTF8 in the first place.

So

1) Why doesn’t <?xml version="1.0" encoding="utf-8"?> actually set UTF8 encoding? when used in the layout?

2) How best to ensure that I always get valid UTF8 format characters sent to my web server?
Should this be handled by the put request or by the code that saves to the shared preference or or by the code that populates the json object or perhaps a combination of the above
? or something else?

This is a follow on from this RoR question Rails 3 – How to handle PG Error incomplete multibyte character

Answer 1

The key is to understand the difference between UTF-8 and Unicode.

• Java processes characters and strings in memory using Unicode. Each character is stored in two bytes.
• When text is transmitted between processes (eg to a web server) or it is written to/read from disk, the internal representation is converted into an over-the-wire format. This is the encoding or decoding. UTF-8 is the most popular, but other formats include:
  • UTF-16
  • ISO 8859-1

In your question, you mention that the XML files are encoded in utf-8: That is good, and you will be able to put foreign characters in the files, but that specifies the encoding only for that specific XML file.

These XML files will be compiled into Android resources and will contain the correct values (you can check it if you like in the debugger, or by preserving the intermediate Java resource files from the build chain).

The problem is almost certainly where you send data to and receive data from the HTTP server, specifically where that data is converted between the bytes on the network and a Java String. Currently you are not setting it in the request – this can be done as described in the documentation for Apache HTTPClient.

Although the server might already require/assume this, it’s certainly a good thing to state clearly in the request.

You also need to ensure that the server (the one in Rails 3 – How to handle PG Error incomplete multibyte character):

• Is expecting UTF-8
• Decodes the request using a UTF-8 decoder
• Encodes the response using UTF-8 encoding

(Sorry, but I don’t know Ruby on Rails so I don’t know how to specifically help there).

Back in the Android end, you also need to ensure that your HTTP library is decoding the response with the UTF-8 decoder. If you handle this yourself, ensure that the String constructor you use is this one, and the argument is “utf-8”:

• public String (byte[] data, String charsetName)

Once BOTH the client and the server are using UTF-8, your problems will be resolved.

To help debugging here, I suggest:

1. A number of logging statements on server and client that print the relevant strings as close as possible to the HTTP code

2. Running with the client configured to talk through a debugging proxy. Examine the request and response and check that they are indeed UTF-8. Proxies include:

   • Charles
   • WebScarab
   • Fiddler