Question

I’m trying to plot a large matrix (8,000 x 8,000) as a raster image in R. Unfortunately this requires too much memory for my 32-bit OS, so I’m plotting the data as two (4000 x 8000) images before recombining them.

I’ve looked at a lot of packages, and I’ve found no suitable function. I know images are imported as S4 objects with the colours stored in an array, implying that there should be a way to combine them, but I can’t figure it out. Does anyone know how to do this in R?
Thanks

Edit:

The data is stored in 8000 csv files, with file1 corresponding to the 1st row of the matrix, file2 to the second…

example code

# get the name of each matrix-row file
# each file is a vector of length 8000, each filei corresponding to matrix row i 

a <- list.files()

for(i in 1:4000){

     # read the data into R, and combine it with the other rows

     matrixRow <- read.table(a[i])
     matrixToPlot <- rbind(matrixToPlot, matrixRow)   

 }


 png("test", 4000, 4000)
     rasterImage(as.raster(matrixToPlot))
 graphics.off()
 ## identical code for matrix-row 4001, 4002, ...8000

Answer 1

I tried the following while eyeing the memory on my system monitor, and it stays below 3 Gb the whole time:

    A <- matrix(rnorm(8000*8000),ncol = 8000)
    print(object.size(A),units = "Mb") # 488.3 Mb
    gc() # remove wasted mem (something I don't fully understand)

    # open a plot of appropriate dimensions- scale to your case
    plot(NULL, type = "n", xlim = c(0, 8001), ylim = c(0, 8000))
    # add chunks of the image a bit at a time, removing waste memory at each chunk
    for (m in 1:4){
      for (n in 1:4){
        image(x = (m * 2000 - 2000):(m * 2000 - 1) + .5, # pixels are centered..
              y = (n * 2000 - 2000):(n * 2000 - 1) + .5, 
              t(A[(m * 2000 - 1999):(m * 2000), (n * 2000 - 1999):(n * 2000)]), 
              useRaster = TRUE, # this is key
              add = TRUE)     
        gc() # my naive guess is that this keeps wasted mem from building up
      }
    }

this gets the thing plotted in window about as economically as I can think of. You could do the same thing by only having parts of A at a time in memory.