Question

I need to fit data in quite an indirect way. The original data to be recovered in the fit is some linear function with small oscillations and drifts on it, that I would like to identify. Let’s call this f(t). We can not record this parameter in the experiment directly, but only indirectly, let’s say as g(f) = sin(a f(t)). (The real transfer funcion is more complex, but it should not play a role in here)

So if f(t) changes direction towards the turning points of the sin function, it is difficult to identify and I tried an alternative approach to recover f(t) than just the inverse function of g and some data continuing guesses:

I create a model function fm(t) which undergoes the same and known transfer function g() and fit g(fm(t)) to the data. As the dataset is huge, I do this piecewise for successive chunks of data guaranteeing the continuity of fm across the whole set.

A first try was to use linear functions using the optimize.leastsq, where the error estimate is derived from g(fm). It is not completely satisfactory, and I think it would be far better to fit a spline to the data to get fspline(t) as a model for f(t), guaranteeing the continuity of the data and of its derivative.

The problem with it is, that spline fitting from the interpolate package works on the data directly, so I can not wrap the spline using g(fspline) and do the spline interpolation on this. Is there a way this can be done in scipy?

Any other ideas?

I tried quadratic functions and fixing the offset and slope such to match the ones of the preceeding fitted chunk of data, so there is only one fitting parameter, the curvature, which very quickly starts to deviate

Thanks

Answer 1

What you would need is a matrix of spline basis functions, b(t), so you can approximate f(t) as a linear combination of spline basis function

f(t) = np.dot(b(t), coefs)

and then estimate the coefficients, coefs, by optimize.leastsq.

However, spline basis functions are not readily available in python, as far as I know (unless you borrow experimental scripts or search through the code of some packages).

Instead you could also use polynomials, for example

b(t) = np.polynomial.chebvander(t, order) 

and use a polynomial approximation instead of the splines.

The structure of this problem is very similar to generalized linear models where g is your known link function and similar to index problems in econometrics.

It would be possible to use the scipy splines in an indirect way if you create artificial data

y_i = f(t_i) 

where f(t_i) are scipy.interpolate splines, and the y_i are the parameters to be estimated in the least squares optimization. (Loosely based on a script that I saw some time ago that used this for creating a different kind of smoothing splines than the scipy version. I don’t remember where I saw this.)