Question

I am trying to make a proxy server that gets a page from www.xxx.com for example, cache the response, and then send it to the requesting browser.

To do so, on the server I create an HTTP client that requests the page from xxx.com. The response is returned in the form of chunks (Buffers). Then, since the number of chunks is different according to the webpage, I put the chunks in an array of buffers. Then I send the elements of the array.

My problem is not all the chunks are sent successfully. Is there any other way I can cache the data before sending it? (I know that I can send the data directly, but I need to send the cache instead since I want to send it to more than one browser)

To save the chunks I use:

function getURL(u) {
     u = url.parse(u);
     var client = http.createClient(u.port || 80, u.hostname);
     var request = client.request('GET', '/', {
                            'Host': u.hostname,
                         });
     var cache ={ };
     cache.data = [];

     request.end();
     request.on('response', function(response) {
          cache.statusCode = response.statusCode;
          cache.headers = response.headers;

          response.on('data', function(chunk) {
               cache.data.push(chunk);
     }
}

to send the cache, i use:

function sendCache(response, cache) {
    var writeSuccess = [];
    response.writeHead(cache.statusCose, cache.headers);
    for (var i in cache.data) {
        // don't encode the data, leave it as it is
        writeSuccess[i] = response.write(cache.data[i], "binary");
        console.log("chunk " + i + " is of length " 
            + cache.data[i].length + ". Response success: " + writeSuccess[i]);
    }
}

Here I log the returned value of the response.write to check if it is successful or not. In the node.js API, it is not explained if this function returns something or not, but I just tried it out.

What I noticed, the response.write was sometimes true and then false for other chunks of the cache whereas if I directly send the response without caching, response.write of all chunks is true.

If anyone notices something wrong I am doing or does know a better way to cache the data (preferably in binary so that all non-ASCII characters will be cached to).

Answer 1

In the latest release of Node.js (v 0.4.0), the issue of writing a response from buffer was solved. So by just updating to this version my problem was solved.

However, one has to know that response.write may still give false, but this doesn’t mean that it is not sent, but not sent directly (leaky bucket concept). This is what I was able to conclude from the comment inside the node.js library (I hope I am correct).