Question

Lately I’m having ‘fun’ with web page scraping. Web site I want to work with has no API so I have no choice, I have to do that this way.

The one of the problems I have is reading HTML tree’s elements (I mean tags, inner texts and stuff like that). I use HttpWebRequest and HttpWebResponse to send GET/POST requests to server.

Having webResponse I can read HTML source code this way:

StreamReader sr = new StreamReader(webResponse.GetResponseStream(), Encoding.UTF8);
string sourceCode = sr.ReadToEnd();

What I need is value of this input tag:

<form action="/file.php" method="post">
    <input name="abc" id="abc" type="hidden" value="some_random_value" />
</form>

How can I do that?

Answer 1

I would use HtmlAgilityPack

string html = @"<form action=""/file.php"" method=""post"">
                <input name=""abc"" id=""abc"" type=""hidden"" value=""some_random_value"" />
                </form>";
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(html);

//Xpath
var value1 = doc.DocumentNode.SelectSingleNode("//input[@id='abc']")
                             .Attributes["value"].Value;

//Linq
var value2 = doc.DocumentNode.Descendants("input")
                .First(i => i.Attributes["id"] != null && 
                            i.Attributes["id"].Value == "abc")
                .Attributes["value"].Value;