Question

I’m using some client-side JavaScript code to pull a lot of JSON data from a webserver via an HTTP GET. The amount of data can be big, say 50 MB. This is on a LAN so it’s not much of a problem, but it still takes ten seconds or so.

To make my interface more responsive, I would like to process the response in chunks, showing data in the UI as soon as it becomes available (let’s say, per MB or per second). Browser compatibility is not an issue; as long as it works on the latest Chrome and Firefox it’ll be okay. However, I cannot modify the server code.

Is it possible to do this, using XMLHttpRequest or WebSockets or some other technology I haven’t heard of?

XMLHttpRequest.responseText is not explicitly empty while the state is LOADING:

The responseText attribute must return the result of running these steps:

1. If the state is not LOADING or DONE return the empty string and terminate these steps.
2. Return the text response entity body.

But I suppose buffering will happen at various stages along the way, so will it work if I set a timer to periodically poll responseText?

As far as I can tell, WebSockets require a special protocol on the server side as well, so those are out.

Restriction: I cannot modify the server code.

Answer 1

Wanted to post this as a comment but the comment textarea is a bit restrictive

Does the server send the data in chunks at the moment or is it one continuous stream? If you add a handler to the xmlHttpRequestInstance.onreadystatechange how often does it get triggered? If the xmlHttpRequestInstance.readystate has a value of 3 then you can get the current value of the xmlHttpRequestInstance.responseText and keep a track of the length. The next time the readystate changes you can get the most recent data by getting all new data starting at that point.

I’ve not tested the following but hopefully it’s clear enough:

var xhr = new XMLHttpRequest();
var lastPos = 0;
xhr.onreadystatechange = function() {
  if(xhr.readystate === 3) {
    var data = xhr.responseText.substring(lastPos);
    lastPos = xhr.responseText.length;

    process(data);
  } 
};
// then of course do `open` and `send` :)

This of course relies on the onreadystatechange event firing. This will work in IE, Chrome, Safari and Firefox.