Question

I am new to Node would appreciate some advice on writing to an array. I have high frequency real-time data coming in, and each tick will be written to a single array.

Would it be beneficial to do this in an asynchronous way? If so, how can this be done?

Here is a trimmed down snippet of my code. The “filter” function is called when a new tick is received. The “storeWindowData” function then saves to the array. I presume this could be blocking under high load?

Appreciate any comments.

Regards, Ben.

var window_data = [];
module.exports = {

filter: function (data) {

    this.storeWindowData(timestamp, ticker, content);

    }
},

storeWindowData: function(timestamp, ticker, content){
    // Check if we have seen this ticker before for this minute
    if(window_data[timestamp] !== undefined && window_data[timestamp][0] !== undefined && window_data[timestamp][0] === ticker){
        window_data[timestamp][1] =  window_data[timestamp][1] + ',' + content;
    } else {
        window_data[timestamp] = [ticker, content];
    }
}     

};

Answer 1

An array is an in memory primitive (well, built in type, not particularly primitive).

How exactly could you, at a JavaScript level, async write to it?

Where would it be stored temporarily while waiting to be written? A second array? (See the problem? Writing to that array would cause two array writes, which would be twice as slow.)

You’re going to have to identify a different bottleneck, or if this really is the problem (which I’m very, very sure that it’s not), you have either too weak of a machine handling this, or you have an insanely high amount of data coming in.