I am getting JSON data from a web service and would like to display a progress bar while the data is downloading. All the examples I have seen use a StringBuilder like so:
//Set up the initial connection
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
connection.setDoOutput(true);
connection.setReadTimeout(10000);
connection.connect();
InputStream stream = connection.getInputStream();
//read the result from the server
reader = new BufferedReader(new InputStreamReader(stream));
StringBuilder builder = new StringBuilder();
String line = "";
while ((line = reader.readLine()) != null) {
builder.append(line + '\n');
}
result = builder.toString();
I got the ProgressBar to work by downloading the data as a byte array, then converting the byte array to a String, but I’m wondering if there is a ‘more correct’ way to do this. Since I’ve found no other way of doing this, the following class can also serve as a working example, seems a bit of a hack, but it does work well.
package com.royaldigit.newsreader.services;
import android.os.AsyncTask;
import android.util.Log;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import com.royaldigit.newsreader.controller.commands.CommandInterface;
import com.royaldigit.newsreader.model.data.SearchResultDO;
import com.royaldigit.newsreader.model.data.SearchTermDO;
/**
* Gets news results from Feedzilla based on the search term currently stored in model.searchTermDO
*
* Sends progress update and returns results to the CommandInterface command reference:
* * command.onProgressUpdate(progress);
* * command.serviceComplete(results);
*
*
*/
public class FeedzillaSearchService {
private static final String TAG = "FeedzillaSearchService";
private static final String SERVICE_URI = "http://api.feedzilla.com/v1/categories/26/articles/search.json?q=";
private static final int STREAM_DIVISIONS = 10;
private CommandInterface command;
private SearchTermDO currentSearchTermDO;
private Integer maximumResults;
private DownloadTask task;
private ArrayList<SearchResultDO> results;
public Boolean isCanceled = false;
public void getData(CommandInterface cmd, SearchTermDO termDO, Integer maxResults){
command = cmd;
currentSearchTermDO = termDO;
//Feedzilla only allows count to be 100 or less, anything over throws an error
maximumResults = (maxResults > 100)? 100 : maxResults;
results = new ArrayList<SearchResultDO>();
task = new DownloadTask();
task.execute();
}
public void cancel() {
isCanceled = true;
if(task != null) task.cancel(true);
}
/**
* Handle GET request
*
*/
private class DownloadTask extends AsyncTask<Void, Integer, String> {
@Override
protected String doInBackground(Void...voids) {
String result = "";
if(currentSearchTermDO == null || currentSearchTermDO.term.equals("")) return result;
BufferedReader reader = null;
publishProgress(0);
try {
String path = SERVICE_URI + URLEncoder.encode(currentSearchTermDO.term, "UTF-8") + "&count=" + maximumResults;
Log.d(TAG, "path = "+path);
URL url = new URL(path);
//Set up the initial connection
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
connection.setDoOutput(true);
connection.setReadTimeout(10000);
connection.connect();
int length = connection.getContentLength();
InputStream stream = connection.getInputStream();
byte[] data = new byte[length];
int bufferSize = (int) Math.ceil(length / STREAM_DIVISIONS);
int progress = 0;
for(int i = 1; i < STREAM_DIVISIONS; i++){
int read = stream.read(data, progress, bufferSize);
progress += read;
publishProgress(i);
}
stream.read(data, progress, length - progress);
publishProgress(STREAM_DIVISIONS);
result = new String(data);
} catch (Exception e) {
Log.e(TAG, "Exception "+e.toString());
} finally {
if(reader != null){
try {
reader.close();
} catch(IOException ioe) {
ioe.printStackTrace();
}
}
}
return result;
}
protected void onProgressUpdate(Integer... progress) {
int currentProgress = progress[0] * 100/STREAM_DIVISIONS;
if(!this.isCancelled()) command.onProgressUpdate(currentProgress);
}
@Override
protected void onPostExecute(String result){
if(!this.isCancelled()) downloadTaskComplete(result);
}
}
/**
*
* @param data
*/
private void downloadTaskComplete(Object data){
if(!isCanceled){
try {
Log.d(TAG, data.toString());
JSONObject obj = new JSONObject(data.toString());
JSONArray array = obj.getJSONArray("articles");
for(int i = 0; i < array.length(); i++){
SearchResultDO dataObj = new SearchResultDO();
dataObj.title = array.getJSONObject(i).getString("title");
dataObj.url = array.getJSONObject(i).getString("url");
dataObj.snippet = array.getJSONObject(i).getString("summary");
dataObj.source = array.getJSONObject(i).getString("source");
dataObj.date = array.getJSONObject(i).getString("publish_date");
dataObj.termId = currentSearchTermDO.id;
//Reformat date
SimpleDateFormat format1 = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss Z");
try {
Date date = format1.parse(dataObj.date);
SimpleDateFormat format2 = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
dataObj.date = format2.format(date);
} catch(ParseException pe) {
Log.e(TAG, pe.getMessage());
}
results.add(dataObj);
}
command.serviceComplete(results);
} catch(JSONException e){
Log.e(TAG, e.toString());
command.serviceComplete(results);
}
}
}
}
UPDATE: Here is the finished version of the class using the suggestions from Nikolay. I ended up using the StringBuilder after all. The previous version would break because some times connection.getContentLength() returns -1. This version degrades gracefully for that case. Tested this implementation quite a bit and it seems bulletproof.
package com.royaldigit.newsreader.services;
import android.os.AsyncTask;
import android.util.Log;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import com.royaldigit.newsreader.controller.commands.CommandInterface;
import com.royaldigit.newsreader.model.data.SearchResultDO;
import com.royaldigit.newsreader.model.data.SearchTermDO;
/**
* Gets news results from Feedzilla based on the search term currently stored in model.searchTermDO
*
* Sends progress update and returns results to the CommandInterface command reference:
* * command.onProgressUpdate(progress);
* * command.serviceComplete(results);
*
*/
public class FeedzillaSearchService implements SearchServiceInterface {
private static final String TAG = "FeedzillaSearchService";
private static final String SERVICE_URI = "http://api.feedzilla.com/v1/categories/26/articles/search.json?q=";
private CommandInterface command;
private SearchTermDO currentSearchTermDO;
private Integer maximumResults;
private DownloadTask task;
private ArrayList<SearchResultDO> results;
private Boolean isCanceled = false;
public void getData(CommandInterface cmd, SearchTermDO termDO, Integer maxResults){
command = cmd;
currentSearchTermDO = termDO;
//Feedzilla only allows count to be 100 or less, anything over throws an error
maximumResults = (maxResults > 100)? 100 : maxResults;
results = new ArrayList<SearchResultDO>();
task = new DownloadTask();
task.execute();
}
public void cancel() {
isCanceled = true;
if(task != null) task.cancel(true);
}
/**
* Handle GET request
*
*/
private class DownloadTask extends AsyncTask<Void, Integer, String> {
@Override
protected String doInBackground(Void...voids) {
String result = "";
if(currentSearchTermDO == null || currentSearchTermDO.term.equals("")) return result;
BufferedReader reader = null;
publishProgress(0);
try {
String path = SERVICE_URI + URLEncoder.encode(currentSearchTermDO.term, "UTF-8") + "&count=" + maximumResults;
Log.d(TAG, "path = "+path);
URL url = new URL(path);
//Set up the initial connection
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
connection.setDoOutput(true);
connection.setReadTimeout(20000);
connection.connect();
//connection.getContentType() should return something like "application/json; charset=utf-8"
String[] values = connection.getContentType().toString().split(";");
String charset = "";
for (String value : values) {
value = value.trim();
if (value.toLowerCase().startsWith("charset=")) {
charset = value.substring("charset=".length());
break;
}
}
//Set default value if charset not set
if(charset.equals("")) charset = "utf-8";
int contentLength = connection.getContentLength();
InputStream stream = connection.getInputStream();
reader = new BufferedReader(new InputStreamReader(stream));
StringBuilder builder = new StringBuilder();
/**
* connection.getContentLength() can return -1 on some connections.
* If we have the content length calculate progress, else just set progress to 100 and build the string all at once.
*
*/
if(contentLength>-1){
//Odd byte array sizes don't always work, tried 512, 1024, 2048; 1024 is the magic number because it seems to work best.
byte[] data = new byte[1024];
int totalRead = 0;
int bytesRead = 0;
while ((bytesRead = stream.read(data)) > 0) {
try {
builder.append(new String(data, 0, bytesRead, charset));
} catch (UnsupportedEncodingException e) {
Log.e(TAG, "Invalid charset: " + e.getMessage());
//Append without charset (uses system's default charset)
builder.append(new String(data, 0, bytesRead));
}
totalRead += bytesRead;
int progress = (int) (totalRead * (100/(double) contentLength));
//Log.d(TAG, "length = " + contentLength + " bytesRead = " + bytesRead + " totalRead = " + totalRead + " progress = " + progress);
publishProgress(progress);
}
} else {
String line = "";
while ((line = reader.readLine()) != null) {
builder.append(line + '\n');
publishProgress(100);
}
}
result = builder.toString();
} catch (Exception e) {
Log.e(TAG, "Exception "+e.toString());
} finally {
if(reader != null){
try {
reader.close();
} catch(IOException ioe) {
ioe.printStackTrace();
}
}
}
return result;
}
protected void onProgressUpdate(Integer... progress) {
if(!this.isCancelled()) command.onProgressUpdate(progress[0]);
}
@Override
protected void onPostExecute(String result){
if(!this.isCancelled()) downloadTaskComplete(result);
}
}
/**
*
* @param data
*/
private void downloadTaskComplete(Object data){
if(!isCanceled){
try {
Log.d(TAG, data.toString());
JSONObject obj = new JSONObject(data.toString());
JSONArray array = obj.getJSONArray("articles");
for(int i = 0; i < array.length(); i++){
SearchResultDO dataObj = new SearchResultDO();
dataObj.title = array.getJSONObject(i).getString("title");
dataObj.url = array.getJSONObject(i).getString("url");
dataObj.snippet = array.getJSONObject(i).getString("summary");
dataObj.source = array.getJSONObject(i).getString("source");
dataObj.date = array.getJSONObject(i).getString("publish_date");
dataObj.termId = currentSearchTermDO.id;
//Reformat date
SimpleDateFormat format1 = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss Z");
try {
Date date = format1.parse(dataObj.date);
SimpleDateFormat format2 = new SimpleDateFormat(SearchResultDO.DATE_FORMAT_STRING);
dataObj.date = format2.format(date);
} catch(ParseException pe) {
Log.e(TAG, pe.getMessage());
}
results.add(dataObj);
}
} catch(JSONException e){
Log.e(TAG, e.toString());
}
command.serviceComplete(results);
}
}
}
Well, since content length is reported in bytes, there is really no other way. If you want to use a
StringReaderyou could take the length of each line you read and calculate the total bytes read to achieve the same thing. Also, the regular idiom is to check the return value ofread()to check if you have reached the end of the stream. If, for some reason, the content length is wrong, your code may read more/less data then available. Finally, when converting a byte blob to a string, you should explicitly specify the encoding. When dealing with HTTP, you can get that from the ‘charset’ parameter of the ‘Content-Type’ header.