Question

Node’s HTTP module is not returning all the expected data from this URL: http://itradedata.co.za/pmchart/prices/GetPricesTRI.asp?Shortname=ACE&numprices=5

There are 5 price records when viewing in a browser, but only one in Node’s HTTP GET data. Packet sniffing shows all the data going in, so why is it not in the output?

var http = require('http'),
    host = 'itradedata.co.za',
    records = 5,
    url = '/pmchart/prices/GetPricesTRI.asp?Shortname=ACE&numprices=' + records,
    client = http.createClient(80, host),
    headers = { 'host': host },
    req;

req = client.request('GET', url, headers);
req.on('response', function(res) {
    console.log(res.statusCode);
    res.on('data', function (chunk) {
        console.log(chunk.toString());
    });
});
req.end();

The problem seems to be in the way the server returns data… cURL also does not show the data when running curl {url} in terminal, but it writes everything to file when using curl {url} -o {file}. What is going on here?

Expected result:

# Attempting to get 5 prices for theTicker: ACE<BR>
# Trying JSECODE<BR>
# Sending prices for Entity_ID: 17352 Shortname:  ShareID:  JSECode: <BR>
2011-8-15,46,46,46,46,0,08268
2011-8-12,46,46,46,46,51,0068
2011-8-11,46,46,46,46,51,0068
2011-8-10,46,46,46,46,51,0068
2011-8-8,46,46,46,46,51,00068

Actual result:

# Attempting to get 5 prices for theTicker: ACE<BR>
# Trying JSECODE<BR>
# Sending prices for Entity_ID: 17352 Shortname:  ShareID:  JSECode: <BR>
2011-8-8,46,46,46,46,51,00068

The four missing records are in the middle of the result.

Answer 1

I’m relatively sure that it is because that site sends \r, but no \n.

Buffer contents:

2c 30 30 0d 32 30

0d says “back to the begin of the line”, 0a says “next line”. So it always goes back, but never writes a new line. It just overwrites existing output.

You could try replacing 0d with 0a to see everything.

Edit: Tried it, it works. Code:

req = client.request('GET', url, headers);
req.on('response', function(res) {
    console.log(res.statusCode);
    res.on('data', function (chunk) {
        for (var i=0; i<chunk.length; i++)
            if (chunk[i] === 0xd) chunk[i] = 0xa
        console.log(chunk.toString());
    });
});
req.end();