Question

I have a pandas DataFrame containing timestamped events from multiple users. By default, the DataFrame is sorted by timestamp.

uid timestamp other_vars
  1       100        ...
  1       150        ...
  2       150        ...
  2       200        ...
  1       225        ...
  3       300        ...
  3       400        ...

I’d like to get the diff of the timestamp within users. That is, for each event, I want to get the time elapsed since the previous event generated by the same user.

uid timestamp diff other_vars
  1       100   NA        ...
  1       150   50        ...
  2       150   NA        ...
  2       200   50        ...
  1       225   75        ...
  3       300   NA        ...
  3       400  100        ...

Is there a clean way to do this in pandas, ideally without sorting by User? Thanks!

Answer 1

As mentioned in the comments, you can use groupby. I’d groupby and then diff. groupby will (unsurprisingly) group the rows:

>>> df
   uid  timestamp other_vars
0    1        100        ...
1    1        150        ...
2    2        150        ...
3    2        200        ...
4    1        225        ...
5    3        300        ...
6    3        400        ...
>>> for name, gr in df.groupby("uid"):
...     print name
...     print gr
...     
1
   uid  timestamp other_vars
0    1        100        ...
1    1        150        ...
4    1        225        ...
2
   uid  timestamp other_vars
2    2        150        ...
3    2        200        ...
3
   uid  timestamp other_vars
5    3        300        ...
6    3        400        ...

And then we select the column we’re interested in along these groups and diff it:

>>> df["diff"] = df.groupby("uid")["timestamp"].diff()
>>> df
   uid  timestamp other_vars  diff
0    1        100        ...   NaN
1    1        150        ...    50
2    2        150        ...   NaN
3    2        200        ...    50
4    1        225        ...    75
5    3        300        ...   NaN
6    3        400        ...   100

Note that we didn’t sort the timestamps, so if you wanted that you’ve have to do it explicitly.