Home/ Questions/Q 6530655
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-25T09:46:56+00:00

I’ve confirmed that R calls of XML functions such as htmlParse and readHTML send

  • 0

I’ve confirmed that R calls of XML functions such as htmlParse and readHTML send a blank user agent string to the server.

?XML::htmlParse tells me under isURL that “The libxml parser handles the connection to servers, not the R facilities”. Does that mean there is no way to set user agent?

(I did try options(HTTPUserAgent="test") but that is not being applied.)

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    XML::htmlParse uses the libxml facilities (i.e. NanoHTTP) to fetch HTTP content using the GET method. By default, NanoHTTP does not send a User-Agent header. There is no libxml API to pass a User-Agent string to NanoHTTP, although one can pass arbitrary header strings to lower-level NanoHTTP functions, like xmlNanoHTTPMethod. Hence, it would require significant source code modification in order to make this possible in the XML package.

    Alternatively, options(HTTPUserAgent="test") sets the User-Agent header for functions that use the R facility for for HTTP requests. For example, one could use download.file like so:

    options(HTTPUserAgent='BioStatMatt-via-R')
download.file('http://biostatmatt.com/', destfile='biostatmatt.html')
XML::htmlParse('biostatmatt.html')

    The (Apache style) access log entry looks something like this:

    160.129.***.*** - - [01/Sep/2011:20:16:40 +0000] "GET / HTTP/1.0" 200 4479 "-" "BioStatMatt-via-R"
    • 0