Question

I’m currently trying to get used to data.table package in R. I want to get the index for the last 1 ocurring in each row of a data table, say a and add that new column to a. My code for this is the following:

  a = data.table(matrix(sample(c(0,1),500,rep=T),50,10))
  a[,ind:=apply(a==1,1,function(x) max(which(x)))]

Nevertheless, I think this can be written in a short way using more data.table syntax. Therefore, my question is: how to do this without the apply function within the j component of [?

Answer 1

Great question. Yes, the apply by row isn’t page efficient, the which will be allocating for each and every row, and the a==1 creates a new logical matrix as large as a.

In data.table we do things by column. Sometimes, it’s data.table-ish to use a for loop through columns (never for loop through rows) :

a[,ans:=0L]
for (i in 1:10) set(a,which(a[[i]]==1),"ans",i)

identical(a$ind, a$ans)
# [1] TRUE

As you can see it’s a completely different style. But, I think, this should be :

• page efficient; i.e., it runs by columns which are contiguous in memory
• needs working memory as large as (just) one column rather than whole of a
• calls which() (a non primitive, vectorized function) 10 rather than nrow(a) times

I didn’t do any speed tests, though, so I might have to eat my words.

See ?set.

In response to comment, to inspect how it works, set happens to return a pointer to the data.table, so we can look at the first few rows as it progresses.

a[,ans:=0L]   # add column by reference, initialized with 0L

> head(a)
   V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ans
1:  0  0  1  0  1  1  0  1  1   1   0
2:  0  0  1  1  0  0  1  1  1   1   0
3:  0  1  0  0  0  1  1  1  1   0   0
4:  0  0  0  1  1  0  1  1  1   1   0
5:  1  1  1  1  0  0  0  0  0   1   0
6:  1  1  0  1  1  0  1  0  1   1   0

Now hopefully the following reveals how it’s working :

> for (i in 1:10) print(head(set(a,which(a[[i]]==1),"ans",i)))
   V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ans
1:  0  0  1  0  1  1  0  1  1   1   0
2:  0  0  1  1  0  0  1  1  1   1   0
3:  0  1  0  0  0  1  1  1  1   0   0
4:  0  0  0  1  1  0  1  1  1   1   0
5:  1  1  1  1  0  0  0  0  0   1   1
6:  1  1  0  1  1  0  1  0  1   1   1
   V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ans
1:  0  0  1  0  1  1  0  1  1   1   0
2:  0  0  1  1  0  0  1  1  1   1   0
3:  0  1  0  0  0  1  1  1  1   0   2
4:  0  0  0  1  1  0  1  1  1   1   0
5:  1  1  1  1  0  0  0  0  0   1   2
6:  1  1  0  1  1  0  1  0  1   1   2
   V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ans
1:  0  0  1  0  1  1  0  1  1   1   3
2:  0  0  1  1  0  0  1  1  1   1   3
3:  0  1  0  0  0  1  1  1  1   0   2
4:  0  0  0  1  1  0  1  1  1   1   0
5:  1  1  1  1  0  0  0  0  0   1   3
6:  1  1  0  1  1  0  1  0  1   1   2
   V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ans
1:  0  0  1  0  1  1  0  1  1   1   3
2:  0  0  1  1  0  0  1  1  1   1   4
3:  0  1  0  0  0  1  1  1  1   0   2
4:  0  0  0  1  1  0  1  1  1   1   4
5:  1  1  1  1  0  0  0  0  0   1   4
6:  1  1  0  1  1  0  1  0  1   1   4
   V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ans
1:  0  0  1  0  1  1  0  1  1   1   5
2:  0  0  1  1  0  0  1  1  1   1   4
3:  0  1  0  0  0  1  1  1  1   0   2
4:  0  0  0  1  1  0  1  1  1   1   5
5:  1  1  1  1  0  0  0  0  0   1   4
6:  1  1  0  1  1  0  1  0  1   1   5
   V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ans
1:  0  0  1  0  1  1  0  1  1   1   6
2:  0  0  1  1  0  0  1  1  1   1   4
3:  0  1  0  0  0  1  1  1  1   0   6
4:  0  0  0  1  1  0  1  1  1   1   5
5:  1  1  1  1  0  0  0  0  0   1   4
6:  1  1  0  1  1  0  1  0  1   1   5
   V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ans
1:  0  0  1  0  1  1  0  1  1   1   6
2:  0  0  1  1  0  0  1  1  1   1   7
3:  0  1  0  0  0  1  1  1  1   0   7
4:  0  0  0  1  1  0  1  1  1   1   7
5:  1  1  1  1  0  0  0  0  0   1   4
6:  1  1  0  1  1  0  1  0  1   1   7
   V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ans
1:  0  0  1  0  1  1  0  1  1   1   8
2:  0  0  1  1  0  0  1  1  1   1   8
3:  0  1  0  0  0  1  1  1  1   0   8
4:  0  0  0  1  1  0  1  1  1   1   8
5:  1  1  1  1  0  0  0  0  0   1   4
6:  1  1  0  1  1  0  1  0  1   1   7
   V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ans
1:  0  0  1  0  1  1  0  1  1   1   9
2:  0  0  1  1  0  0  1  1  1   1   9
3:  0  1  0  0  0  1  1  1  1   0   9
4:  0  0  0  1  1  0  1  1  1   1   9
5:  1  1  1  1  0  0  0  0  0   1   4
6:  1  1  0  1  1  0  1  0  1   1   9
   V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ans
1:  0  0  1  0  1  1  0  1  1   1  10
2:  0  0  1  1  0  0  1  1  1   1  10
3:  0  1  0  0  0  1  1  1  1   0   9
4:  0  0  0  1  1  0  1  1  1   1  10
5:  1  1  1  1  0  0  0  0  0   1  10
6:  1  1  0  1  1  0  1  0  1   1  10
>