Question

I am trying to create a huge matrix in ff, and I know that ff is good for this sort of thing.

But, there is a major problem. The dimensions of the matrix exceed .Machine$max_integer! I am running on a 64 bit machine, using 64bit R and 64bit ff.

Is there any way to get around this problem?

It’s been suggested that R is using the MAXINT value from stdint.h. Is there any way to fix this without changing that file and possibly breaking build?

> ffMatrix <- ff(vmode="boolean", dim=c(300000,300000))
Error in if (length < 0 || length > .Machine$integer.max) stop("length must be between 1 and .Machine$integer.max") : 
  missing value where TRUE/FALSE needed
In addition: Warning message:
In ff(vmode = "boolean", dim = c(300000, 300000)) :
  NAs introduced by coercion

> 300000**2 > .Machine$integer.max
[1] TRUE

Answer 1

That is still a very large matrix, you would need a parallel computer :-). 300000×300000 > 2^32 (int is likely to be 32 bits, even on 64-bit machine) .

See also this link