Question

Here is my data:

data1 <- c(726, 718, 699, 737, 743, 734, 726, 722, 715, 714, 752, 750, 
749, 746, 743, 734, 725, 717, 717, 708, 756, 753, 752, 746, 744, 
740, 737, 732, 728, 728, 720, 720, 714, 703, 702, 697, 758, 753, 
753, 746, 743, 734, 720, 706, 697, 761, 744, 749, 741, 738, 738, 
732, 725, 720, 712, 782, 778, 776, 773, 772, 770, 770, 769, 769, 
766, 763, 763, 756, 755, 753, 750, 749, 746, 737, 723, 711, 702, 
685, 782, 779, 778, 778, 776, 776, 775, 772, 770, 770, 766, 763, 
761, 756, 752, 738, 735, 729, 715)

This gives the plot using plot(data1) as:

How can I segregate the eight separate trends using R? I can use identify(data1) and mark them (the shifts) manually and use the indexes to segregate them but it would not be possible in my case since I am dealing with many of these kinds of plots. I want to extract the separate lines programmatically. Please let me know if there is statistical technique (time series and so on) which identifies the change in trends and tags and returns the eight series.

EDIT

I should make it clear that the number of series in the whole data-set need to be identified. The number of series in the sample data happens to be eight. But I will not know this in each case unless I plot the data and identify the breaks manually.

Answer 1

I think Kiril is right on here ( +1 ). This is a substantial area of research/knowledge, sometimes called Change Point or Break Point Analysis. Relevant R packages include strucchange changepoint, as mentioned by Kiril, and bcp.

strucchange‘s breakpoints() uses least squares regression to estimate the
locations of changes when told how many changes there are (which is not helpful here.)

changepoint has several algorithms, as Kiril said, including Scott and Knott’s Binary Segmentation, Auger and Lawrence’s Segment Neighbourhoods, and Killick et al’s Pruned Exact Linear Time (PELT) algorithm. Its a great package, but my very limited experience is that these functions will require a bunch of tuning (which is worth doing).

For important tasks, we would want to leverage our knowledge of the data to make more detailed model(s) and, ultimately, combine several methods into an ensemble classifier.

But for simple click and go, you could use bcp which uses an MCMC-based approximation of Barry and Hartigan’s Bayes procedure:

library(bcp)
changefit <- bcp(data1)
plot(changefit)

One way to segregate the groups would be to pick a posterior probability threshold and split() the data based on it:

data2 <- data.frame(data=data1, prob = changefit$posterior.prob)
threshold <- 0.90
split(data2,c(0, cumsum( ifelse (data2$prob > threshold, 1, 0))))
#split will warn about unequal lengths