Question

I am wanting to run a web scraping script on a server.

The current script collects the html on the specified page.

$url = "http://websms"
 [net.httpWebRequest] $request = [net.webRequest]::create($url)
 [net.httpWebResponse] $response = $request.getResponse()
 $responseStream = $response.getResponseStream()
 $sr = new-object IO.StreamReader($responseStream)
 $result = $sr.ReadToEnd()

 $result

This works fine on a typical web page. However I am wanting to run it on a servers admin page which of course requires a login.

I thought before I attempt to login I will try and scrape the login page of the server. Running the above script I get the following result.

   Exception calling "GetResponse" with "0" argument(s): "The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel."
At C:\temp\web3.ps1:3 char:56
+  [net.httpWebResponse] $response = $request.getResponse <<<< ()
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : DotNetMethodException

Any idea of how to get around this issue or perhaps if you could point me a different direction so I can scrape elements from the admin html page of the sever.

Thanks Guys!

Answer 1

This one liner will ignore SSL certificate errors:

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}

Errors regarding self-signed untrusted certificates, mismatching names or expiration will be ignored after this is executed.