Home/ Questions/Q 6972925
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-27T17:02:15+00:00

I have a large (150,000×7) dataframe that I intend to use for back-testing and

  • 0

I have a large (150,000×7) dataframe that I intend to use for back-testing and real-time analysis of a financial market. The data represents the condition of an investment vehicle at 5 minute intervals (although holes do exist). It looks like this (but much longer): 

        pTime     Time  Price       M1       M2        M3        M4
1  1212108300 20:45:00 1.5518 12.21849 -0.37125   4.50549 -31.00559
2  1212108900 20:55:00 1.5516 11.75350 -0.81792  -1.53846 -32.12291
3  1212109200 21:00:00 1.5512 10.75070 -1.47438  -8.24176 -34.35754
4  1212109500 21:05:00 1.5514 10.23529 -1.06044  -8.46154 -33.24022
5  1212109800 21:10:00 1.5514  9.74790 -1.02759 -10.21978 -33.24022
6  1212110100 21:15:00 1.5513  9.31092 -1.17076 -11.97802 -33.79888
7  1212110400 21:20:00 1.5512  8.84034 -1.28428 -13.62637 -34.35754
8  1212110700 21:25:00 1.5509  8.07843 -1.63715 -18.24176 -36.03352
9  1212111000 21:30:00 1.5509  7.39496 -1.49198 -20.65934 -36.03352
10 1212111300 21:35:00 1.5512  7.65266 -1.03717 -18.57143 -34.35754

The data is pre-loaded into R, but during my back-test I need to subset it by two criteria:

The first criteria is a sliding window to avoid peeking into the future. The window must be such that, each new 5 minute interval on the back-test shifts the whole window into the future by 5 minutes. This part I can do like this:

require(zoo)
zooser <- zoo(x=tser$Close, order.by=as.POSIXct(tser$pTime, origin="1970-01-01"))
window(zooser, start=A, end=B)

The second criteria is another sliding window, but one that slides through time of day and contains only those entries that are within N minutes of the input time on any given day.

Example: If the window’s size is 2 hours, and the input time is 12:00PM then the window must contain all rows with Time between 10:00AM and 2:00PM

This is the part that I am having trouble figuring out.

Edit: My data has holes in it, two consecutive rows could be MORE than 5 minutes apart. The data looks like this (very zoomed in)
enter image description here

As the window moves through these gaps the number of points inside the windows should vary.

The following is my MySQL code that does what I want to do in R (same table structure):

SET @qTime = Time(FROM_UNIXTIME(SAMP_endTime));

SET @inc = -1;
INSERT INTO MetIndListBuys (pTime,ArrayPos,M1,M2,M3,M4)
SELECT pTime,@inc:=@inc+1,M1,M2,M3,M4
 FROM mergebuys USE INDEX (`y`) WHERE  pTime BETWEEN SAMP_startTime AND SAMP_endTime 
AND TIME_TO_SEC(TIMEDIFF(Time,@qTime))/3600 BETWEEN 0-HourSpan AND HourSpan
;
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Say that you have your target time t0 on the same scale as pTime: seconds since epoch. Then t0 – pTime = (difference in the number of days since epoch between the two) + (difference in remaining seconds). Taking t0 – pTime %% (num. seconds per day) will leave us with the difference in seconds in clock arithmetic (wrapped around if the difference is negative). This suggests the following function:

    SecondsPerDay <- 24 * 60 * 60
within <- function(d, t0Sec, wMin) {
  diff <- (d$pTime - t0Sec) %% SecondsPerDay
  wSec <- 60 * wMin
  return(d[diff < wSec | diff > (SecondsPerDay - wSec), ])
}
    • 0