Question

I would like to do an algebraic curve fit of 2D data points, but for various reasons – it isn’t really possible to have much of the sample data in memory at once, and iterating through all of it is an expensive process.

(The reason for this is that actually I need to fit thousands of curves simultaneously based on gigabytes of data which I’m reading off disk, and which is therefore sloooooow).

Note that the number of polynomial coefficients will be limited (perhaps 5-10), so an exact fit will be extremely unlikely, but this is ok as I’m trying to find an underlying pattern in data with a lot of random noise.
I understand how one can use a genetic algorithm to fit a curve to a dataset, but this requires many passes through the sample data, and thus isn’t practical for my application.

Is there a way to fit a curve with a single pass of the data, where the state that must be maintained from sample to sample is minimal?

I should add that the nature of the data is that the points may lie anywhere on the X axis between 0.0 and 1.0, but the Y values will always be either 1.0 or 0.0.

So, in Java, I’m looking for a class with the following interface:

public interface CurveFit {
   public void addData(double x, double y);
   public List<Double> getBestFit(); // Returns the polynomial coefficients
}

The class that implements this must not need to keep much data in its instance fields, no more than a kilobyte even for millions of data points. This means that you can’t just store the data as you get it to do multiple passes through it later.

edit: Some have suggested that finding an optimal curve in a single pass may be impossible, however an optimal fit is not required, just as close as we can get it in a single pass.

The bare bones of an approach might be if we have a way to start with a curve, and then a way to modify it to get it slightly closer to new data points as they come in – effectively a form of gradient descent. It is hoped that with sufficient data (and the data will be plentiful), we get a pretty good curve. Perhaps this inspires someone to a solution.

Answer 1

I believe I found the answer to my own question based on a modified version of this code. For those interested, my Java code is here.