Question

I am using nodeJS with some additional modules to do web page scraping and media item identification from a set of websites.

The node server basically throws back a JSON markup of all the items identified on the page and its associated metadata. The JSON data is generated correctly as I can see it in the server logs however when I write it to the client, for some reason the JSON response is terminated.

I tested this with all browsers and using rest clients and it seems to be point to an issue with response.write(response, ‘utf-8’) which may not be sending the whole data or the connection gets closed for some reason.

I verified that there is no chunking involved for my test cases so there is no question of the connection being aggressively closed by the client if its still waiting for the next chunk of data. i.e. response.write in this case returns true which implies that all the data has been written to client.

Any pointers as to what could be causing the connection to be terminated or the response to be truncated? For JSON responses of smaller sizes the response is received correctly by the client.

Code: 
return parseDOM(page, url, function(err, response){
            if(err){
                res.writeHeader(200, {'Content-Type':'application/json'});
                res.end('Error Parsing DOM from ' + url);
                e.message = 'Error Parsing DOM';
                callback(e, req, res, targetUrl);               
                return;
            }
            else {
                if(response){
                    res.writeHeader(200, {'Content-Type':'application/json', 'Content-Length':response.length});
                    console.log(response);
                    res.write(response, 'UTF-8');
                    res.end();
                    callback(null, req, res, targetUrl);
                    return;
                }
            }
        }); 

Answer 1

Sorry. My bad. I see that the content length is wrong. Identified solution via issue:
Node.js cuts off files when serving over HTTPS