Question

I have a very simple problem. I have

x = [1 2 3 4 5];
y = [5.5 43.1 128 290.7 498.4];
p = polyfit(x,y,3);
x2 = 1:.1:5;
y2 = polyval(p,x2);
plot(x,y,'o',x2,y2)
grid on

How can I prove that with polyfit(x,y,4) , I get non-significant parameters, so I should take only polyfit(x,y,3);
regards,

Answer 1

While WebMonster’s advice to use a learning and test set is very valuable in general, for the data set you provided it is not usable.

In general an nth order polynomial is determined by n+1 parameters. So you need at least n+1 data points to be able to fit your data. To find the coefficients (i.e. parameters), we just express that f(x) = y for all x and y, which are n+1 equations in n+1 unknowns and as long as all x-values are unique, this can be solved exactly. When you have more data points (equations) than unknowns, a best solution (most often a least-squares) solution can be calculated. That is a solution that minimizes the distances between the model and the data points.

And that also gives some intuition in what happens with your data set. You have 5 points, so a fourth order polynomial can fit the data perfectly at the points you supplied. That means that any noise present in your measurements will be part of your model, i.e. the model you obtain is only unbiased if your data didn’t contain any disturbances (noise).

However, from this little amount of data, you cannot conclude that you should either use a third or a fourth order model. To do so, you need more information. You either need more data points or you need to have field information. E.g. if you know that the data is generated by a system that can be described by a polynomial of third order, that is (most likely) what you should use.

Using the fourth order model with this little evidence, is clearly ludicrous (since you then assume that your measurements were perfect), it is equally foolish to choose the third order model “just because the fourth order model won’t do”.

I calculated the least-squares cost functions for your data (this is the cost function that is minimized by polyfit), i.e.

for n = 1:4
    p    = polyfit(x,y,n);
    ym   = polyval(p,x);
    e    = y - ym;
    V(n) = sum(e.^2)/2
end

The cost functions V(n) are a measure of how badly your models perform for every order n, the higher the cost, the worse your fit. I calculated V= [6269, 28.885, 28.621, 6.083e-25]. From this you can clearly see that a linear (first order) model has a very bad fit, and the fourth order model has a (near) perfect fit (the cost is practically equal to zero). But both the second and third order model have very similar performance. Introducing the third parameter decreases the cost by only 0.264.

You can get an idea how much all your data points combined deviate from the model: d = sqrt(2*V). For the second order this is 7.601 while for the third order this is only 7.566, so you are looking at a difference of less than 0.04 (in units of your y axis) to judge your models.

If you choose the third order model, this would mean you deem this difference significant. Just looking at your last measurement, that is equivalent to a relative uncertainty of 0.007%. Long story short: I seriously doubt that the third order model is significantly better than your second order model. Personally, I’d go with a second order model, from this data.

I quickly checked those models against adjusted cost functions for the AIC and MDL criterion, and those indicate that a second order model is to be preferred over the third order model as well. These cost functions include a penalty for increased model complexity (i.e. Occam’s Razor), as to prevent over-fitting.

Note that all of this had little to do with how well either model will perform in real life. You really need more data to come to a reasonable conclusion. It might as well be the case that you will need a fourth or higher order model when you include more measurements.