Question

I have an interesting puzzle. Suppose you have a numpy 2D array, in which each line corresponds to a measurement event and each column corresponds to different measured variable. One additional column in this array specifies the date at which the measurement was taken. The lines are sorted according to the time stamp. There are several (or many) measurements on each day. The goal is to identify the lines that correspond to a new day and subtract the respective values from the subsequent lines in that day.

I approach this problem by a loop that loops over the days, creating a boolean vector that selects the proper lines and then subtracting the first selected line. This approach works, but feels non-elegant. Are there better ways to do this?

Just a small example. The lines below define a matrix in which the first colum
is the day and the remaining two are the measured values

before = array([[ 1,  1,  2],
   [ 1,  3,  4],
   [ 1,  5,  6],
   [ 2,  7,  8],
   [ 3,  9, 10],
   [ 3, 11, 12],
   [ 3, 13, 14]])

at the end of the process I expect to see the following array:

array([[1, 0, 0],
   [1, 2, 2],
   [1, 4, 4],
   [2, 0, 0],
   [3, 0, 0],
   [3, 2, 2],
   [3, 4, 4]])

PS Please help me finding a better and more informative title for this post. I’m out of ideas

Answer 1

numpy.searchsorted is a convenient function for this:

In : before
Out:
array([[ 1,  1,  2],
       [ 1,  3,  4],
       [ 1,  5,  6],
       [ 2,  7,  8],
       [ 3,  9, 10],
       [ 3, 11, 12],
       [ 3, 13, 14]])

In : diff = before[before[:,0].searchsorted(x[:,0])]

In : diff[:,0] = 0

In : before - diff
Out:
array([[1, 0, 0],
       [1, 2, 2],
       [1, 4, 4],
       [2, 0, 0],
       [3, 0, 0],
       [3, 2, 2],
       [3, 4, 4]])

Longer explanation

If you take the first column, and search for itself you get the minimum indices for those particular values:

In : before
Out:
array([[ 1,  1,  2],
       [ 1,  3,  4],
       [ 1,  5,  6],
       [ 2,  7,  8],
       [ 3,  9, 10],
       [ 3, 11, 12],
       [ 3, 13, 14]])

In : before[:,0].searchsorted(x[:,0])
Out: array([0, 0, 0, 3, 4, 4, 4])

You can then use this to construct the matrix that you will subtract by indexing:

In : diff = before[before[:,0].searchsorted(x[:,0])]

In : diff
Out:
array([[ 1,  1,  2],
       [ 1,  1,  2],
       [ 1,  1,  2],
       [ 2,  7,  8],
       [ 3,  9, 10],
       [ 3,  9, 10],
       [ 3,  9, 10]])

You need to make the first column 0 so that they won’t be subtracted.

In : diff[:,0] = 0

In : diff
Out:
array([[ 0,  1,  2],
       [ 0,  1,  2],
       [ 0,  1,  2],
       [ 0,  7,  8],
       [ 0,  9, 10],
       [ 0,  9, 10],
       [ 0,  9, 10]])

Finally, subtract two matrices to get the desired output:

In : before - diff
Out:
array([[1, 0, 0],
       [1, 2, 2],
       [1, 4, 4],
       [2, 0, 0],
       [3, 0, 0],
       [3, 2, 2],
       [3, 4, 4]])