Question

I have an unbalanced quarterly panel data set with missing values. I want to substract variable A2 from A1 in subsequent quarters. Note that I do not want to get differences of A2, but substract DIFFERENT variables from each other. Differences should be calculated separately for every uid. Besides changing years like Q4 1999 and Q1 2000 are meant to be subsequent.

I am really not sure whether i should concatenate my time index here since packages like zoo only take one index. But that’s not the problem here. Here is a some example data:

structure(list(uid = c(1, 1, 1, 2, 2, 3, 3, 3), tndx = c(1999.4, 
2000.1, 2000.2, 1999.4, 2000.1, 2000.1, 2000.2, 2000.3), A1 = c(2, 
2, 2, 10, 11, 1, 1, 1), A2 = c(3, 3, 3, 14, 14, 2, 100, 2)), .Names = c("uid", 
"tndx", "A1", "A2"), row.names = c(NA, -8L), class = "data.frame")

# which results in
  uid   tndx A1  A2
1   1 1999.4  2   3
2   1 2000.1  2   3
3   1 2000.2  2   3
4   2 1999.4 10  14
5   2 2000.1 11  14
6   3 2000.1  1   2
7   3 2000.2  1 100
8   3 2000.3  1   2  

If you prefer a separated index, use this example:

# Thx Andrie!
x2 <- data.frame(x, colsplit(x$tndx, "\\.", names=c("year", "qtr")))

Is there a good way to solve this with reshape2, plyr or even base or would you rather write a custom function?

Note, it is also possible that some uid occurs only once. Obviously you can’t calculate a lagged difference then. Still I need to check for that and create an NA then.

Answer 1

We split it on the uid using by and within the function that operates on each set of rows for a single uid, we create a zoo object, z, using yearqtr class for the index. Then we merge the time series with an empty series having all the desired quarters including any missing intermediate quarters giving zm and perform the computation giving zz. Finally we convert to the data.frame form on the way out:

library(zoo)
to.yearqtr <- function(x) as.yearqtr(trunc(x) + (10*(x-trunc(x))-1)/4)

DF <- do.call("rbind", by(x, x$uid, function(x) {
    # columns of x are: uid tndx A1 A2
    z <- zoo(x[c("A1", "A2")], to.yearqtr(x$tndx))
    zm <- merge(z, zoo(, seq(start(z), end(z), 1/4)))
    zz <- with(zm, cbind(zm, `A1 - A2 lag` = A1 - lag(A2, -1)))
    if (ncol(zz) <= ncol(z)) zz$`A1 - A2 lag` <- NA # append NA if col not added
    data.frame(uid = x[1, 1], tndx = time(zz), coredata(zz), check.names = FALSE)
}))

which gives this:

> DF
    uid    tndx A1 A2 result A1 - A2 lagged
1.1   1 1999 Q4  2  3     NA             NA
1.2   1 2000 Q1  2  2     NA             -1
1.3   1 2000 Q2  2  3     NA              0
2.1   2 1999 Q4  2  4     NA             NA
2.2   2 2000 Q1 NA NA     NA             NA
2.3   2 2000 Q2 NA NA     NA             NA
2.4   2 2000 Q3 NA NA     NA             NA
2.5   2 2000 Q4 NA NA     NA             NA
2.6   2 2001 Q1  3  4     NA             NA
3.1   3 2000 Q1  1  2     NA             NA
3.2   3 2000 Q2  1 NA     NA             -1
3.3   3 2000 Q3  1  2     NA             NA

EDIT: Completely re-did the solution based on further discussion. Note that this not only adds an extra column but it also converts the index to "yearqtr" class and adds the extra missing rows.

EDIT: Some minor simplifications in the by function.