Question

I wish to filter a dataframe, original.data, in R. The dataframe could have around 1-2 million observations. The dataframe has several fields and the names may vary. The user can select which fields to filter by. These field names are stored in names(all.filters), where all.filters is a list of variable length. The user may then choose the levels for each of the fields in names(all.filters). For example, this list may look something like:

> all.filters
$Period
[1] "2010-12-31" "2011-03-31" "2011-06-30" "2011-09-30" "2011-12-31"
[6] "2012-03-31" "2012-06-30" "2012-09-30"

$Size
[1] "L"  "VL"

$Number
[1] "11" "21" "35" "42" "45" "47" "49" "52" "57"

I am using the following code to apply the chosen filters:

attach(original.data)    
filter.names <- names(all.filters)
flag <- 1
for(filter in filter.names){
   flag <- flag*(is.element(get(filter),all.filters[[filter]]))
}
filtered.data <- original.data[flag==1,]

This works, but it feels a little slow. Note that get(filter) retrieves the column of original.data with column name equal to filter. I’m not sure if this is a good way to filter the data, but the variable nature of all.filters limits my choices a bit – I wanted to use subset, but I’m not sure what to put as the select argument. I would like to make this filtering step as fast as possible so that when the user updates a filter selection, the data can be plotted quickly.

Once the data is filtered, I use reshape2 to summarise the data before plotting it with ggplot2. I am thinking that it may be more efficient to apply the filters at one of these steps if possible.

Any suggestions would be greatly appreciated.

Answer 1

You can use a data.table with appropriately set keys. This will be memory efficient.

Then you can pass your list of filters to the i component of [.data.table

.period <- seq(from = as.Date("2010/1/1", "%Y/%m/%d"), to = as.Date("2012/1/1", 
    "%Y/%m/%d"), by = "3 months")
.size <- c("XS", "S", "M", "L", "XL")
.number <- as.character(1:100)
DF <- expand.grid(Period = .period, Size = .size, Number = .number, stringsAsFactors = F)

DF$other <- rnorm(nrow(DF))

library(data.table)

DT <- as.data.table(DF)

DT[, `:=`(Period, as.IDate(.period))]


##           Period Size Number    other
##    1: 2010-01-01   XS      1  0.17947
##    2: 2010-04-01   XS      1  1.43252
##    3: 2010-07-01   XS      1 -0.97142
##    4: 2010-10-01   XS      1 -0.98021
##    5: 2011-01-01   XS      1 -0.62964
##   ---                                
## 4496: 2011-01-01   XL    100  0.65831
## 4497: 2011-04-01   XL    100 -0.45277
## 4498: 2011-07-01   XL    100 -0.14236
## 4499: 2011-10-01   XL    100 -0.02376
## 4500: 2012-01-01   XL    100 -0.11525

all_filters <- list(Period = as.IDate(as.Date("2010/1/1", format = "%Y/%m/%d")), 
    Size = "L", Number = c("11", "21", "35", "42", "45", "47", "49", "52", "57"))


setkeyv(DT, names(all_filters))

DT[all_filters]

##        Period Size Number   other
## 1: 2010-01-01    L     11  1.4122
## 2: 2010-01-01    L     21 -0.4923
## 3: 2010-01-01    L     35  1.1262
## 4: 2010-01-01    L     42  1.3527
## 5: 2010-01-01    L     45 -0.3758
## 6: 2010-01-01    L     47 -0.1847
## 7: 2010-01-01    L     49 -0.8503
## 8: 2010-01-01    L     52 -1.0645
## 9: 2010-01-01    L     57 -0.6092

The only issue I can see is that you will have to reset the key each time to ensure you are referencing the correct columns. Also, you will need to ensure that the filter identifiers are the same class as the columns in the data.frame — it may be easier to work with character not factor columns

EDIT

To filter with more than level on more than one column, use CJ. CJ is a cross join, (the data.table equivalent of expand.grid, with keys set)

all_filters <- list(Period = as.IDate(as.Date("2010/1/1", format = "%Y/%m/%d")), 
  Size = c("L",'XL'), Number = c("11", "21", "35", "42", "45", "47", "49", "52", "57"))




cj_filter <- do.call(CJ, all_filters)

# note you could avoid this `do.call` line by
# cj_filter <- CJ(Period = as.IDate(as.Date("2010/1/1", format = "%Y/%m/%d")), 
  Size = c("L",'XL'), Number = c("11", "21", "35", "42", "45", "47", "49", "52", "57"))

setkeyv(DT, names(cj_filter))

DT[cj_filter]
       Period Size Number       other
 1: 2010-01-01    L     11  0.36289104
 2: 2010-01-01    L     21  1.26356767
 3: 2010-01-01    L     35 -0.18629723
 4: 2010-01-01    L     42  0.92267902
 5: 2010-01-01    L     45  1.68796072
 6: 2010-01-01    L     47  1.75107447
 7: 2010-01-01    L     49  0.24048407
 8: 2010-01-01    L     52  0.06675221
 9: 2010-01-01    L     57  0.49665392
10: 2010-01-01   XL     11  0.33682495
11: 2010-01-01   XL     21  0.67642271
12: 2010-01-01   XL     35 -0.16412768
13: 2010-01-01   XL     42  0.72863394
14: 2010-01-01   XL     45 -0.55527588
15: 2010-01-01   XL     47  1.30850591
16: 2010-01-01   XL     49  1.08688166
17: 2010-01-01   XL     52 -0.31157250
18: 2010-01-01   XL     57  0.43626422

You could also do

 setkeyv(DT, names(all_filters))

 DT[do.call(CJ,all_filters)]