Question

Say I have a data frame like this:

Df <- data.frame(
    V1 = c(1,2,3,NA,5),
    V2 = c(1,2,NA,4,5),
    V3 = c(NA,2,NA,4,NA)
)

Now I want to count the number of valid observations for every combination of two variables. For that, I wrote a function sharedcount:

sharedcount <- function(x,...){
    nx <- names(x)
    alln <- combn(nx,2)
    out <- apply(alln,2,
      function(y)sum(complete.cases(x[y]))
    )
    data.frame(t(alln),out)
}

This gives the output:

> sharedcount(Df)
  X1 X2 out
1 V1 V2   3
2 V1 V3   1
3 V2 V3   2

All fine, but the function itself takes pretty long on big dataframes (600 variables and about 10000 observations). I have the feeling I’m overseeing an easier approach, especially since cor(…,use=’pairwise’) is running still a whole lot faster while it has to do something similar :

> require(rbenchmark)    
> benchmark(sharedcount(TestDf),cor(TestDf,use='pairwise'),
+     columns=c('test','elapsed','relative'),
+     replications=1
+ )
                           test elapsed relative
2 cor(TestDf, use = "pairwise")    0.25     1.0
1           sharedcount(TestDf)    1.90     7.6

Any tips are appreciated.

---

Note : Using Vincent’s trick, I wrote a function that returns the same data frame. Code in my answer below.

Answer 1

The following is slightly faster:

x <- !is.na(Df)
t(x) %*% x

#       test elapsed relative
#    cor(Df)  12.345 1.000000
# t(x) %*% x  20.736 1.679708