Question

I have two data.tables in R:

> tables()
     NAME          NROW   MB COLS                                 KEY       
[1,] dtb      2,536,206   68 dte,permno,capm_beta,mkt_beta_bucket permno,dte
[2,] idx_dtb        573    1 dte                                  dte       
[3,] ssd_dtb 58,808,208 1571 dte,permno,xs_ret,mkt_cap            permno,dte
Total: 1,640MB

I would like to do a right outer join: select * from dtb right join ssd_dtb using (permno, dte)

The equivalent command in data.table would be:

mdtb <- dtb[ssd_dtb]

So far this has been running for about 20 minutes. This seems a bit long as it would normally take less time on an SQL server to run. Am I misusing the package?

EDIT AFTER QUESTION ANSWERED:
I thought it might be helpful to understand why I had to do dtb[ssd_dtb] and not ssd_dtb[dtb]. I devised a quick example:

> x <- data.table(id=1:5, t=1:15, v=1:15)
> x
    id  t  v
 1:  1  1  1
 2:  2  2  2
 3:  3  3  3
 4:  4  4  4
 5:  5  5  5
 6:  1  6  6
 7:  2  7  7
 8:  3  8  8
 9:  4  9  9
10:  5 10 10
11:  1 11 11
12:  2 12 12
13:  3 13 13
14:  4 14 14
15:  5 15 15
> y <- data.table(id=c(1,1,2,3), t=c(1,2,6,13), v2=41:44)
> y
   id  t v2
1:  1  1 41
2:  1  2 42
3:  2  6 43
4:  3 13 44

> x[y]
   id  t  v v2
1:  1  1  1 41
2:  1  2 NA 42
3:  2  6 NA 43
4:  3 13 13 44
> x[y, nomatch=0]
   id  t  v v2
1:  1  1  1 41
2:  3 13 13 44
> y[x]
    id  t v2  v
 1:  1  1 41  1
 2:  1  6 NA  6
 3:  1 11 NA 11
 4:  2  2 NA  2
 5:  2  7 NA  7
 6:  2 12 NA 12
 7:  3  3 NA  3
 8:  3  8 NA  8
 9:  3 13 44 13
10:  4  4 NA  4
11:  4  9 NA  9
12:  4 14 NA 14
13:  5  5 NA  5
14:  5 10 NA 10
15:  5 15 NA 15
> y[x, nomatch=0]
   id  t v2  v
1:  1  1 41  1
2:  3 13 44 13 

EDIT 2: The ouput of the above request for solution 1 is below:

> Rprof()
> mdtb  Rprof(NULL)
> summaryRprof()
$by.self
               self.time self.pct total.time total.pct
"[.data.table"     15.36    62.39      24.62    100.00
".Call"             7.96    32.33       7.96     32.33
"pmin"              0.92     3.74       1.16      4.71
"list"              0.24     0.97       0.24      0.97
"vector"            0.14     0.57       0.14      0.57

$by.total
               total.time total.pct self.time self.pct
"[.data.table"      24.62    100.00     15.36    62.39
"["                 24.62    100.00      0.00     0.00
".Call"              7.96     32.33      7.96    32.33
"pmin"               1.16      4.71      0.92     3.74
"list"               0.24      0.97      0.24     0.97
"vector"             0.14      0.57      0.14     0.57
"integer"            0.14      0.57      0.00     0.00

$sample.interval
[1] 0.02

$sampling.time
[1] 24.62

Answer 1

If I can guess without a reproducible example, I’ll try. In this case a few things do spring to mind. Good question.

1. There is one TODO in the source which could cause a large slowdown in this case. I wasn’t aware it was as bad as that, though. If you know that dtb‘s key is unique (i.e. no duplicated dte within each permno), as is very likely here, then set mult="first" to work around it. Or mult="last" would be the same. What it’s doing is finding the start and end point of groups, but when a key is unique all those groups are just 1 row.

2. A smaller effect may be that the i table is large compared to x in your case. If possible, try to arrange the query with i smaller than x; i.e. ssd_dtb[dtb] rather than dtb[ssd_dtb]. They would be the same when nomatch=0.

If the workaround in 1 helps, it would be great if you could run the following and send me the results. Also, please raise a bug.report(package="data.table") linking back to this question. That way you’ll get automatic updates as the status changes.

Rprof()
dtb[ssd_dtb]  # reduce size so that this takes 30 seconds or something manageable
Rprof(NULL)
summaryRprof()

Rprof()
dtb[ssd_dtb,mult="first"]  # should be faster than above if my guess is right
Rprof(NULL)
summaryRprof()