Question

How do I download through R the source code for a webpage that has tags that were created by JavaScript?

When I use the FireFox ‘Inspect Element’ function, tags are sometimes not shown in the HTML source file. In other words, information I need is in the JavaScript code. Is there a way to read this information into R?

Related question: How to view webpage source code using R?

Answer 1

You can use getURL from RCurl to get the HTTP response.

library(RCurl)
address <- "https://discussions.apple.com/thread/4356115?tstart=0"
txt <- getURL(address)

Now you can spit the string on the opening tag, then split that on the closing tag

ss <- strsplit(txt, "<strong class=\"jive-thread-reply-message-correct-label\">")[[1]]
strsplit(ss[2], "</strong>")[[1]][1]

Which gives:

[1] "This solved my question"

---

It turns out that there is more than one of the div tag you wanted, and the above gets the wrong one. I don’t know how to do it purely in R, but I followed the post you referenced by VitoshKa and I got it to work.

First, in Firefox go to Tools -> Add-ons. Search for and install mozrepl. Then, in Firefox click Tools -> MozRepl -> Start.

Now, in R:

mz <- socketConnection("localhost", "4242")
writeLines("var w=window.open(\"https://discussions.apple.com/thread/4356115?tstart=0\")\n",mz)
out <- readLines(mz) #empty the buffer
writeLines("w.document.getElementsByTagName('html')[0].innerHTML\n", mz)
out <- readLines(mz)

(loc <- grep("jive-thread-reply-message-correct-label", out))
#[1] 1150 2845

Now, out is a vectorloc holds the positions of the strings that contain your tag. It appears twice. The first one is the one you’re interested in.

out[loc[1]]

You can extract the information from this the same way I showed above with strsplit, or with a regular expression and gsub

---

You can close the window that opens with writeLines("w.window.close()", mz)