Question

I have both time series data and moving average data for the same data-set and the same time interval. I would like to extract the overlapping points. I would like extract those points where the extremum between the current and the last crossing point (intersected point between Moving average and time series) is taken depending on the direction of the trend reveral. I don’t have idea whether perform it based on tables (1-Time series table includes temperature and timestamps 2-Moving average table includes only Moving average results of temperature) or perform it based on plot of them in which I have add moving average to time series plot.

Table of Time Series:

     ambtemp                  dt
1     -1.14 2007-09-29 00:01:57
2     -1.12 2007-09-29 00:03:57
3     -1.33 2007-09-29 00:05:57
4     -1.44 2007-09-29 00:07:57
5     -1.54 2007-09-29 00:09:57
6     -1.29 2007-09-29 00:11:57
7     -1.42 2007-09-29 00:13:57
8     -1.37 2007-09-29 00:15:57
9     -1.32 2007-09-29 00:17:57
10    -1.37 2007-09-29 00:19:57
11    -1.14 2007-09-29 00:21:57
12    -1.16 2007-09-29 00:23:57
13    -1.08 2007-09-29 00:25:57
14    -1.21 2007-09-29 00:27:57
15    -1.26 2007-09-29 00:29:57
16    -1.50 2007-09-29 00:31:57
17    -1.35 2007-09-29 00:33:57
18    -1.56 2007-09-29 00:35:57
19    -1.60 2007-09-29 00:37:57
20    -1.30 2007-09-29 00:39:57
21    -1.24 2007-09-29 00:41:57
22    -1.24 2007-09-29 00:43:57
23    -1.10 2007-09-29 00:45:57
24    -0.99 2007-09-29 00:47:57
25    -1.04 2007-09-29 00:49:57
26    -0.97 2007-09-29 00:51:57
27    -0.92 2007-09-29 00:53:57
28    -0.70 2007-09-29 00:55:57
29    -0.58 2007-09-29 00:57:57
30    -0.49 2007-09-29 00:59:57
31    -0.54 2007-09-29 01:01:57
32    -0.37 2007-09-29 01:03:57
33    -0.24 2007-09-29 01:05:57
34    -0.34 2007-09-29 01:07:57
35    -0.43 2007-09-29 01:09:57
36    -0.52 2007-09-29 01:11:57
37    -0.16 2007-09-29 01:13:57
38    -0.50 2007-09-29 01:15:57
39    -0.15 2007-09-29 01:17:57
40    -0.04 2007-09-29 01:19:57
41     0.00 2007-09-29 01:21:57
42    -0.08 2007-09-29 01:23:57
43    -0.30 2007-09-29 01:25:57
44    -0.14 2007-09-29 01:27:57
45    -0.06 2007-09-29 01:29:57
46    -0.02 2007-09-29 01:31:57
47    -0.25 2007-09-29 01:33:57
48    -0.33 2007-09-29 01:35:57
49     0.01 2007-09-29 01:37:57
50    -0.28 2007-09-29 01:39:57
51    -0.26 2007-09-29 01:41:57
52    -0.34 2007-09-29 01:43:57
53    -0.43 2007-09-29 01:45:57
54    -0.20 2007-09-29 01:47:57
55     0.19 2007-09-29 01:49:57
56     0.28 2007-09-29 01:51:57
57     0.07 2007-09-29 01:53:57
58    -0.32 2007-09-29 01:55:57
59    -0.20 2007-09-29 01:57:57
60    -0.42 2007-09-29 01:59:57
61    -0.40 2007-09-29 02:01:57
62    -0.54 2007-09-29 02:03:57
63    -0.67 2007-09-29 02:05:57
64    -0.42 2007-09-29 02:07:57
65    -0.57 2007-09-29 02:09:57
66    -0.73 2007-09-29 02:11:57
67    -0.54 2007-09-29 02:13:57
68    -0.70 2007-09-29 02:15:57
69    -0.70 2007-09-29 02:17:57
70    -0.58 2007-09-29 02:19:57

Table of Moving average:

  [1]      NA      NA      NA      NA      NA      NA      NA      NA      NA
 [10]      NA      NA      NA      NA      NA      NA      NA      NA      NA
 [19]      NA -1.3250 -1.3300 -1.3360 -1.3245 -1.3020 -1.2770 -1.2610 -1.2360
 [28] -1.2025 -1.1655 -1.1215 -1.0915 -1.0520 -1.0100 -0.9665 -0.9250 -0.8760
 [37] -0.8165 -0.7635 -0.6910 -0.6280 -0.5660 -0.5080 -0.4680 -0.4255 -0.3765
 [46] -0.3290 -0.2955 -0.2770 -0.2475 -0.2370 -0.2230 -0.2215 -0.2310 -0.2240
 [55] -0.1930 -0.1530 -0.1415 -0.1325 -0.1350 -0.1540 -0.1740 -0.1970 -0.2155
 [64] -0.2295 -0.2550 -0.2905 -0.3050 -0.3235 -0.3590 -0.3740 -0.3860 -0.3880
 [73] -0.3695 -0.4045 -0.4450 -0.4900 -0.5125 -0.5215 -0.5515 -0.5825 -0.6140

Overlap of time series and moving average for different window size

Expected Results for window size of 40

Answer 1

Once you have the set-up right, it’s an easy three step process.

Step 1: Use a serious time series package for this sort of stuff. I suggest xts — it’s awesome.

Step 2: Once you have it in this format, it’s just a few lag commands, and a few logical tests.

Step 3: Index your data with the output of the logical tests.

The first problem is getting from your text file into R – i put a copy in my ~/Downloads/ directory, but yours may be elsewhere. The problem with the text file is that you have only two headers, and four columns — this will cause R problems. So let’s skip the header row, and name them ourselves.

require(xts)
require(TTR)

dd <- read.table("~/Downloads/Timeseries.txt", skip=1)
dd <- dd[,-1] # cut off the row numbers

colnames(dd) <- c('ambtemp', 'date', 'time')

dt <- as.POSIXct(paste(dd$date, dd$time))
tempdata_x <- xts(dd$ambtemp, order.by = dt)
colnames(tempdata_x) <- 'ambtemp'

tempdata_x$ma20 <- SMA(tempdata_x$ambtemp, n=20)

max change:

max(diff(tempdata_x$ambtemp), na.rm=TRUE)
[1] 1.92

match(max(diff(tempdata_x$ambtemp), na.rm=TRUE), diff(tempdata_x$ambtemp))
[1] 357

So the row you want is number 357, get it via tempdata_x[357,].

Check for overlaps / crossings etc:

tempdata_x$overlap <- ifelse(tempdata_x$ambtemp == tempdata_x$ma20, 1, 0)
tempdata_x$aboveMA <- ifelse(tempdata_x$ambtemp > tempdata_x$ma20, 1, 0)
tempdata_x$belowMA <- ifelse(tempdata_x$ambtemp < tempdata_x$ma20, 1, 0)
tempdata_x$crossUp <- ifelse(lag(tempdata_x$belowMA) & tempdata_x$aboveMA, 1, 0)
tempdata_x$crossDown <- ifelse(lag(tempdata_x$aboveMA) & tempdata_x$belowMA, 1, 0)

Now your data frame tempdata_x has a few new columns – which we’ll use for indexing later.

A column overlap marked with 1 when ambtemp == ma20; aboveMA marked with 1 when ambtemp > ma20; belowMA marked with 1 when ambtemp < ma20; crossUp marked with a 1 when there is a crossing up from below, and another column crossDown market with a 1 when there is a crossing down from above.

Restrict to cases where there is a moving average (ie remove the leading section)

tempdata_x.trim <- tempdata_x[complete.cases(tempdata_x),]

Use the fact that the columns we created contain 0 and 1 — and so can be coerced to logical to index tempdata_x.trim:

tempdata_x.trim[as.logical(tempdata_x.trim$overlap), 1:2]
tempdata_x.trim[as.logical(tempdata_x.trim$aboveMA), 1:2]
tempdata_x.trim[as.logical(tempdata_x.trim$belowMA), 1:2]
tempdata_x.trim[as.logical(tempdata_x.trim$crossUp), 1:2]
tempdata_x.trim[as.logical(tempdata_x.trim$crossDown), 1:2]

I tested each line on your 720 row text file — it works.