Home/ Questions/Q 210653
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-11T18:01:29+00:00

I have a large set of numbers, probably in the multiple gigabytes range. First

  • 0

I have a large set of numbers, probably in the multiple gigabytes range. First issue is that I can’t store all of these in memory. Second is that any attempt at addition of these will result in an overflow. I was thinking of using more of a rolling average, but it needs to be accurate. Any ideas?

These are all floating point numbers.

This is not read from a database, it is a CSV file collected from multiple sources. It has to be accurate as it is stored as parts of a second (e.g; 0.293482888929) and a rolling average can be the difference between .2 and .3

It is a set of #’s representing how long users took to respond to certain form actions. For example when showing a messagebox, how long did it take them to press OK or Cancel. The data was sent to me stored as seconds.portions of a second; 1.2347 seconds for example. Converting it to milliseconds and I overflow int, long, etc.. rather quickly. Even if I don’t convert it, I still overflow it rather quickly. I guess the one answer below is correct, that maybe I don’t have to be 100% accurate, just look within a certain range inside of a sepcific StdDev and I would be close enough.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You can sample randomly from your set (“population“) to get an average (“mean“). The accuracy will be determined by how much your samples vary (as determined by “standard deviation” or variance).

    The advantage is that you have billions of observations, and you only have to sample a fraction of them to get a decent accuracy or the “confidence range” of your choice. If the conditions are right, this cuts down the amount of work you will be doing.

    Here’s a numerical library for C# that includes a random sequence generator. Just make a random sequence of numbers that reference indices in your array of elements (from 1 to x, the number of elements in your array). Dereference to get the values, and then calculate your mean and standard deviation.

    If you want to test the distribution of your data, consider using the Chi-Squared Fit test or the K-S test, which you’ll find in many spreadsheet and statistical packages (e.g., R). That will help confirm whether this approach is usable or not.

    • 0