Question

I have nested for loops as below. I need p:q=1:300, and n=20. Function “mark” is the model of my interest(Package RMark). I know rbind can be slow but I have no idea what should be used to replace it. Otherwise what else I can do to make this function faster? Thanks.

foo<-function(data, p, q, n){
results.frame <- data.frame()
for (i in 1:n){
    for (i in p:q) {
        run.model<-mark(data[sample(nrow(data), i),], model="Occupancy")       
        results<-data.frame(summary(run.model)$real$p, Occupancy=summary(run.model)$real$Psi, se.p=t(as.matrix(summary(run.model, se=T)$real$p$se)), se.Psi=summary(run.model, se=T)$real$Psi$se, stations=i)
        results.frame<-rbind(results.frame, results)
        } 
    }
write.table(results.frame, "C:\\RWorkspace\\simulation_results.txt")
return(results.frame)
}

Answer 1

Yes, rbind can be slow; the faster thing is usually to make the matrix the right size to start with and fill it in appropriately. It’s also usually faster to fill in a matrix instead of a data frame.

However, with the size you indicate, I would suspect that mark is what is slowing the function down and you won’t get much noticeable speedup by doing that. It would be easy to test that by storing a single result in run.model and then commenting that line out of your loop; that will tell you how much time it’s spending just storing the results. (You could also “profile” the function, but this would be simpler.)

EDIT: I’m actually wrong; the size you indicate is big enough that the rbind is quite possible causing problems. On my system, which is fairly fast and has a decent amount of memory, it takes 7.73 sec to rbind using data frames with n=20 and only 0.09 sec with n=1, so clearly some memory churning is happening. As for speedup, with n=20 it takes only 1.00 sec to rbind matrices and 0.033 sec to fill it in.

foo <- function(data, p, q, n){
  # make a single results line; remove this line when you put in your code 
  results <- c(1, Occupancy=2, se.p=3, se.Psi=4, stations=5)
  # make the matrix the right size to start with
  results.frame <- matrix(ncol=5, nrow=(q-p+1)*n)
  for (i in 1:n){
    for (j in p:q) {
      # get results here; commented out to show loop speed only
      # put in your actual code here instead
      results.frame[ 1+(i-1)*(q-p+1)+(j-p), ] <- results
    } 
  }
  # get the names right by taking the names from the last time through the loop
  colnames(results.frame) <- names(results)
  results.frame
}