Question

I just started playing around with the Twitter Streaming API and using the command line, redirect the raw JSON reponses to a file using the command below:

curl https://stream.twitter.com/1/statuses/sample.json -u USER:PASSWORD -o "somefile.txt"

Is it possible to stay completely within R and leverage RCurl to do the same thing? Instead of just saving the output to a file, I would like to parse each response that is returned. I have parsed twitter search results in the past, but I would like to do this as each response is received. Essentially, apply a function to each JSON response.

Thanks in advance.

EDIT: Here is the code that I have tried in R (I am on Windows, unfortunately). I need to include the reference to the .pem file to avoid the error. However, the code just “runs” and I can not seem to see what is returned. I have tried print, cat, etc.

download.file(url="http://curl.haxx.se/ca/cacert.pem", destfile="cacert.pem")
getURL("https://stream.twitter.com/1/statuses/sample.json", 
       userpwd="USER:PWD",
       cainfo = "cacert.pem")

Answer 1

I was able to figure out the basics, hopefully this helps.

#==============================================================================
# Streaming twitter using RCURL
#==============================================================================
library(RCurl)
library(rjson)

# set the directory
setwd("C:\\")

#### redirects output to a file
WRITE_TO_FILE <- function(x) {

     if (nchar(x) >0 ) {
          write.table(x, file="Twitter Stream Capture.txt", append=T, row.names=F, col.names=F)
     }

}

### windows users will need to get this certificate to authenticate
download.file(url="http://curl.haxx.se/ca/cacert.pem", destfile="cacert.pem")

### write the raw JSON data from the Twitter Firehouse to a text file
getURL("https://stream.twitter.com/1/statuses/sample.json", 
       userpwd=USER:PASSWORD,
       cainfo = "cacert.pem", 
       write=WRITE_TO_FILE)