Question

I have a huge table (400k+ rows), where each row describes an event in the FX market. The table’s primary key is an integer named ‘pTime’ – it is the time at which the event occurred in POSIX time.

My database is queried repeatedly by my computer during a simulation that I constantly run. During this simulation, I pass an input pTime (I call it qTime) to a MySQL procedure. qTime is a query point from that same huge table. Using qTime, my procedure filters the table according to the following rule:

Select only those rows whose pTime is a maximum 2 hours away from the input qTime on any day.

ex.

query point: `2001-01-01  07:00`
                                 lower limit: `ANY-ANY-ANY 05:00`
                                 upper limit: `ANY-ANY-ANY 09:00`     

After this query the query point will shift by 1 row (5 minutes), and a new query will be initiated:

query point: `2001-01-01  07:05`
                                 lower limit: `ANY-ANY-ANY 05:05`
                                 upper limit: `ANY-ANY-ANY 09:05`     

This is the way I accomplish that:

SELECT * FROM mergetbl WHERE
TIME_TO_SEC(TIMEDIFF(FROM_UNIXTIME(pTime,"%H:%i"),FROM_UNIXTIME(qTime,"%H:%i")))/3600
BETWEEN -2 AND 2

Although I have an index on pTime, this piece of code significantly slows down my software.

I would like to pre-process this statement for each value of pTime (which will later serve as an input qTime), but I cannot figure out a way to do this.

Answer 1

If you rely only on time – I’d suggest you to add another column of time type with time fraction of pTime and perform queries over it