Question

I have a data.table object similar to this one

library(data.table)

c <- data.table(CO = c(10000,10000,10000,20000,20000,20000,20000),
                SH = c(1427,1333,1333,1000,1000,300,350),
                PRC = c(6.5,6.125,6.2,0.75,0.5,3,3.5),
                DAT = c(0.5,-0.5,0,-0.1,NA_real_,0.2,0.5),
                MM = c("A","A","A","A","A","B","B"))

and I am trying to perform calculations using nested grouping, passing an expression as an argument. Here is a simplified version of what I have:

setkey(c,MM)

mycalc <- quote({nobscc <- length(DAT[complete.cases(DAT)]); 
                 list(MKTCAP = tail(SH,n=1)*tail(PRC,n=1),
                      SQSUM = ifelse(nobscc>=2, sum(DAT^2,na.rm=TRUE), NA_real_),
                      COVCOMP = ifelse(nobscc >= 2, head(DAT,n=1), NA_real_),
                      NOBS = nobscc)}) 


myresults <- c[,.SD[,{setkey=CO; eval(mycalc)},by=CO],by=MM]

which produces

     MM    CO MKTCAP SQSUM COVCOMP NOBS
[1,]  A 10000 8264.6  0.50     0.5    3
[2,]  A 20000  500.0    NA      NA    1
[3,]  B 20000 1225.0  0.29     0.2    2

In the example above I have two elements of the list which use the ifelse construct (in the actual code there are 3), all doing the same test : if the number of observations is greater than 2, then a certain calculation (which is different for each element of the list, and each could be written as a function) is to be performed, otherwise I want the value of the these elements to be NA. Another thing these elements have in common is that they use one and the same column of my data.table: the one called DAT.

So my question is: is there any way I can do the ifelse test only once, and if it is FALSE, pass the value NA to the respective elements of the list, and if TRUE, evaluate a different expression for each of the elements of the list?

NOTE: My goal is to reduce the system.time (system and elapsed). If this modification will not reduce time and calculations, bearing in mind I have 72 million observations, that’s an acceptable answer. I also welcome suggestions to change other parts of the code.

EDIT: Results of summaryRprof()

$by.total
                          total.time total.pct self.time self.pct
"system.time"                  18.94     99.79      0.00     0.00
".Call"                        18.92     99.68      0.10     0.53
"["                            18.92     99.68      0.04     0.21
"[.data.table"                 18.92     99.68      0.02     0.11
"eval"                         18.80     99.05      0.24     1.26
"ifelse"                       18.30     96.42      0.46     2.42
"lm"                           17.70     93.26      0.58     3.06
"sapply"                        8.06     42.47      0.36     1.90
"model.frame"                   7.74     40.78      0.16     0.84
"model.frame.default"           7.58     39.94      0.98     5.16
"lapply"                        6.62     34.88      0.70     3.69
"FUN"                           4.24     22.34      1.10     5.80
"model.matrix"                  4.04     21.29      0.02     0.11
"model.matrix.default"          4.02     21.18      0.26     1.37
"match"                         3.66     19.28      0.86     4.53
".getXlevels"                   3.12     16.44      0.12     0.63
"na.omit"                       2.40     12.64      0.24     1.26
"%in%"                          2.30     12.12      0.34     1.79
"simplify2array"                2.24     11.80      0.12     0.63
"na.omit.data.frame"            2.16     11.38      0.14     0.74
"[.data.frame"                  2.12     11.17      1.18     6.22
"deparse"                       1.80      9.48      0.66     3.48
"unique"                        1.80      9.48      0.54     2.85
"[["                            1.52      8.01      0.12     0.63
"[[.data.frame"                 1.40      7.38      0.54     2.85
".deparseOpts"                  1.34      7.06      0.96     5.06
"paste"                         1.32      6.95      0.16     0.84
"lm.fit"                        1.20      6.32      0.64     3.37
"mode"                          1.14      6.01      0.14     0.74
"unlist"                        1.12      5.90      0.56     2.95

Answer 1

Instead of forming and operating on data subsets like this:

setkey(c,MM)
myresults <- c[, .SD[,{setkey=CO; eval(mycalc)},by=CO], by=MM]

You could try doing this:

setkeyv(c, c("MM", "CO"))
myresults <- c[, eval(mycalc), by=key(c)]

This should speed up your code, since it avoids all of the nested subsetting of .SD objects, each of which requires its own call to [.data.table.

---

On your original question, I doubt the ifelse evaluations are taking much time, but if you want to avoid them, you could take them out of mycalc and use := to overwrite the desired values with NA:

mycalc <- quote(list(MKTCAP = tail(SH,n=1)*tail(PRC,n=1),
                      SQSUM = sum(DAT^2,na.rm=TRUE),
                      COVCOMP = head(DAT,n=1),
                      NOBS = length(DAT[complete.cases(DAT)]))) 
setkeyv(c, c("MM", "CO"))
myresults <- c[, eval(mycalc), by=key(c)]


myresults[NOBS<2, c("SQSUM", "COVCOMP"):=NA_real_]
## Or, alternatively
# myresults[NOBS<2, SQSUM:=NA_real_]
# myresults[NOBS<2, COVCOMP:=NA_real_]