Home/ Questions/Q 6111637
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-23T14:39:12+00:00

I’m writing code to work with text from Wikipedia and am having issues with

  • 0

I’m writing code to work with text from Wikipedia and am having issues with en dashes being garbled. I haven’t worked with en dashes or other non-standard characters before (non-standard to me being character that don’t appear on my keyboard ;), so I’m not sure where to point the finger at what I’m doing wrong. Here’s what is happening, along with code snippets…..

I send a request to Wikipedia (I’m using the Apache HttpComponents client API for communicating with Wikipedia) for the contents of an article and save it in a String:

DefaultHttpClient client = new DefaultHttpClient();
HttpGet queryRequest = new HttpGet(query);  // query is the URL for retrieving the article contents.
ResponseHandler<String> responseHandler = new BasicResponseHandler();
String responseBody = client.execute(queryRequest, responseHandler);

At this point if I were to send “responseBody” to System.out, en dashes are displayed in my Eclipse console as ‘?’. This might just be an Eclipse console display issue so I’ll move on.

I manipulate the text, ignoring the en dashes, and then send the text back to Wikipedia. 

List<NameValuePair> postParams = new ArrayList<NameValuePair>();
postParams.add(new BasicNameValuePair("text", content);  // content is a String with the article text
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(postParams, "UTF-8");
HttpPost queryRequest = new HttpPost(url);  // url is the basic URL for the Wikipedia api
queryRequest.setEntity(entity);
queryRequest.addHeader("Content-Type", "application/x-www-form-urlencoded");
ResponseHandler<String> responseHandler = new BasicResponseHandler();
String responseBody = client.execute(queryRequest, responseHandler);

When the text, now uploaded to Wikipedia, is displayed in a web browser what was en dashes before are now displayed as ‘?’ in a box (unknown character?). Therefore, somewhere I am inadvertently changing or miscoding the en dashes, but I’m not sure exactly where.

Can someone point me in the right direction?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Now for the real answer. The problem with the non-English characters getting mangled had nothing to do with the Apache HTTPComponents or with an Java string handling/manipulation. The problem was with the Eclipse IDE running on Windows.

    Eclipse in the run configuration defaults to use the system’s default encoding method, Cp1252 for Windows. Since Cp1252 doesn’t support all of the UTF-8 characters, thus problems arise. I found the solution here. In Eclipse you go into the Run Configurations. For the project you are attempting to run, go to the ‘Common’ tab. There is a section for encoding. Change it from “Default” to “Other” and set the encoding to UTF-8.

    All is now well.

    • 0