Question

Good afternoon,

After computing a rather large vector (a bit shorter than 2^20 elements), I have to store the result in a database.

The script takes about 4 hours to execute with a simple code such as :

#Do the processing
myVector<-processData(myData)
#Sends every thing to the database
lapply(myVector,sendToDB)

What do you think is the most efficient way to do this?

I thought about using the same query to insert multiple records (multiple inserts) but it simply comes back to “chucking” the data.

Is there any vectorized function do send that into a database?

Interestingly, the code takes a huge amount of time before starting to process the first element of the vector. That is, if I place a browser() call inside sendToDB, it takes 20 minutes before it is reached for the first time (and I mean 20 minutes without taking into account the previous line processing the data). So I was wondering what R was doing during this time?

Is there another way to do such operation in R that I might have missed (parallel processing maybe?)

Thanks!

PS: here is a skelleton of the sendToDB function:

sendToDB<-function(id,data) {
  channel<-odbcChannel(...)
  query<-paste("INSERT INTO history VALUE(",id,",\"",data,"\")",sep="")
  sqlQuery(channel,query)
  odbcClose(channel)
}

That’s the idea.

UPDATE

I am at the moment trying out the LOAD DATA INFILE command.

I still have no idea why it takes so long to reach the internal function of the lapply for the first time.

SOLUTION

LOAD DATA INFILE is indeed much quicker. Writing into a file line by line using write is affordable and write.table is even quicker.

The overhead I was experiencing for lapply was coming from the fact that I was looping over POSIXct objects. It is much quicker to use seq(along.with=myVector) and then process the data from within the loop.

Answer 1

What about writing it to some file and call LOAD DATA INFILE? This should at least give a benchmark. BTW: What kind of DBMS do you use?