Question

I am using node-curl as a HTTPS client to make requests to resources on the web and the code runs on a machine behind a proxy facing the internet.

The code I am using to co:

var curl = require('node-curl');
//Call the curl function. Make a curl call to the url in the first argument.
//Make a mental note that the callback to be invoked when the call is complete 
//the 2nd argument. Then go ahead.
curl('https://encrypted.google.com/', {}, function(err) {
    //I have no idea about the difference between console.info and console.log.
    console.info(this.body);
});
//This will get printed immediately.
console.log('Got here'); 

node-curl detects the proxy settings from the environment and gives back the expected results.

The challenge is: the callback gets fired after the entire https-response gets downloaded, and as far as I can tell there are no parallels for the ‘data’ and ‘end’ events from the http(s) modules.

Further, after going through the source code, I found that indeed the node-curl library receives the data in chunks: reference line 58 in https://github.com/jiangmiao/node-curl/blob/master/lib/CurlBuilder.js . It seems that no events are emitted presently in this case.

I need to forward the possibly-sizable-response back to the another computer on my LAN for processing, so this is a clear concern for me.

Is using node-curl recommended for this purpose in node?

If yes, how can I handle this?

If no, then what would be a suitable alternative?

Answer 1

I would go for the wonderful request module, at least if the proxy settings are no more advanced than what it supports. Just read the proxy settings from the environment yourself:

var request = require('request'),
    proxy = request.defaults({proxy: process.env.HTTP_PROXY});

proxy.get('https://encrypted.google.com/').pipe(somewhere);

Or if you don’t want to pipe it:

var req = proxy.get({uri: 'https://encrypted.google.com/', encoding: 'utf8'});

req.on('data', console.log);
req.on('end', function() { console.log('end') });

Above, I also pass the encoding I expect in the response. You could also specify that in the defaults (the call to request.defaults() above), or you could leave it in which case you will get Buffers in the data event handler.

If all you want to do is to send it to another URL, request is perfect for that:

proxy.get('https://encrypted.google.com/').pipe(request.put(SOME_URL));

Or if you’d rather POST it:

proxy.get('https://encrypted.google.com/').pipe(request.post(SOME_URL));

Or, if you want to proxy the request to the destination server as well:

proxy.get('https://encrypted.google.com/').pipe(proxy.post(SOME_URL));