Question

I am using data.table quite a lot. It works well but I am finding it is taking me a long time to transition my syntax so that it takes advantage of the binary searching.

In the following data table how would 1 select all the rows, including where the CPT value is NA but exclude rows where the CPT value is 23456 or 10000.

cpt <- c(23456,23456,10000,44555,44555,NA)
description <- c("tonsillectomy","tonsillectomy in >12 year old","brain transplant","castration","orchidectomy","miscellaneous procedure")
cpt.desc <- data.table(cpt,description)

setkey(cpt.desc,cpt)

The following line works but I think it uses the vector scan method instead of a binary search (or binary exclusion). Is there a way to to drop rows by binary methods?

cpt.desc[!cpt %in% c(23456,10000),]

Answer 1

Only a partial answer, because I am new to data.table. A self-join works for number, but the same fails for strings. I am sure one of the professional data tablers knows what to do.

library(data.table)

n <- 1000000
cpt.desc <- data.table(
  cpt=rep(c(23456,23456,10000,44555,44555,NA),n),
  description=rep(c("tonsillectomy","tonsillectomy in >12 year old","brain transplant","castration","orchidectomy","miscellaneous procedure"),n))

# Added on revision. Not very elegant, though. Faster by factor of 3
# but probably better scaling 
setkey(cpt.desc,cpt)
system.time(a<-cpt.desc[-cpt.desc[J(23456,45555),which=TRUE]])
system.time(b<-cpt.desc[!(cpt %in% c(23456,45555))] )
str(a)
str(b)

identical(as.data.frame(a),as.data.frame(b))

# A self-join works Ok with numbers
setkey(cpt.desc,cpt)
system.time(a<-cpt.desc[cpt %in% c(23456,45555),])
system.time(b<-cpt.desc[J(23456,45555)])
str(a)
str(b)

identical(as.data.frame(a),as.data.frame(b)[,-3])

# But the same failes with characters
setkey(cpt.desc,description)
system.time(a<-cpt.desc[description %in% c("castration","orchidectomy"),])
system.time(b<-cpt.desc[J("castration","orchidectomy"),])
identical(as.data.frame(a),as.data.frame(b)[,-3])

str(a)
str(b)