I’m an R user, and I frequently find that I need to write functions that require subsetting large datasets (10s of millions of rows). When I apply such functions over a large number of observations, it can get very time consuming if I’m not careful about how I implement it.
To do this, I have sometimes used the data.table package, and this provides much faster speeds than subsetting using data frames. Recently, I’ve started experimenting with packages like RMySQL, pushing some tables to mysql, and using the package to run sql queries and return results.
I have found mixed performance improvements. For smaller datasets (millions), it seems that loading up the data into a data.table and setting the right keys makes for faster subsetting. For larger datasets (10s to 100s of millions), it appears the sending out a query to mysql moves faster.
Was wondering if anyone has any insight into which technique should return simple subsetting or aggregation queries faster, and whether or not this should depend on the size of the data? I understand that setting keys in data.table is somewhat analogous to creating an index, but I don’t have much more intuition beyond that.
If the data fits in RAM, data.table is faster. If you provide an example it will probably become evident, quickly, that you’re using data.table badly. Have you read the “do’s and don’ts” on the data.table wiki?
SQL has a lower bound because it is a row store. If the data fits in RAM (and 64bit is quite a bit) then data.table is faster not just because it is in RAM but because columns are contiguous in memory (minimising page fetches from RAM to L2 for column operations). Use data.table correctly and it should be faster than SQL’s lower bound. This is explained in FAQ 3.1. If you’re seeing slower with data.table, then chances are very high that you’re using data.table incorrectly (or there’s a performance bug that we need to fix). So, please post some tests, after reading the data.table wiki.