Question

I’m currently trying to analyze large data sets (40 000+ data points) with matlab. These data sets have a large degree of high frequency noise, luckily however, the noise is of a relatively small amplitude compared to the overall signal.

Basically, what i wish to do is to find the point at which a data set “kicks up”. By kicking up i mean, its gradient changes significantly over a small region. However, these signals that i’m analyzing suffer from “drift” where by, before the “kick up” point the signals slope may be negative, neutral or positive in nature, however, the shape of the graph before and after the “kick up” point is generally the same.

eg: a “kick up” point would look something like this:

[ 1.25 2.5 3.75 5 7.5 12 21 36 49 64 ]

Obviously this is a massively simplified version but the data would follow a straight line like curve (first 5 data points in the above) and then it would follow a more exponential like curve (point 6 onwards).

So i’ve been looking at implementing a machine type learning algorithm whereby i can tell matlab where a “kick up” point is as well as the “shape” of data where a kick up point occurs such that in the future it can accurately locate “kick up” points.

Im relatively new to both Matlab and machine learning(if machine learning is even the best way of going about solving my problem) and i have no idea where i should begin. I was hoping someone could point me in the right direction, be it matlab code examples and/or some beginner theory with coding implementation.

Answer 1

I believe that ML is an overkill in this situation. If the data is on a straight like curve, it means that its derivative is constant, which in turn means that the second is zero. Thus, calculate the approximate second derivative by using diff twice and find where it becomes more than some threshold. This threshold should be close to zero.

x = [ 1.25 2.5 3.75 5 7.5 12 21 36 49 64 ];
plot(diff(diff(x)));