I have an R function which calls a webserver database through it’s API using the RCurl library and getURLContent() function. I often loop over repeated calls to getURLContent(), which can be taxing to the server. Is there anything I can pass through the R/curl interface to avoid being too demanding on the server?
Here’s an example, where url[i] is the i’th url address in a vector corresponding to different queries to some API.
curl = getCurlHandle()
sapply(url, function(u) getURLContent(u, curl=curl))
The only “generic” suggestion I have would be to ensure that curl keeps the connection to the server alive between requests. This feature of the HTTP protocol is called keep-alive. I read that some curl implementations do this by default. But I don’t know which version you are using. This will not reduce the load on the server which is generated by PHP, it will however reduce the overhead associated with each request and could therefore also speed up your R code.