Question

I am trying to obtain values from a table but am a bit confused on how to properly do it. I want to retrieve the Strike/Symbol/Bid/Ask from the following page: http://finance.yahoo.com/q/op?s=MSFT&m=2012-09

For my code, I have tried number of things but maybe I am not undrestanding correctly how to utilize Xpath.

private void optionchainButton_Click(object sender, EventArgs e)
        {
            string URL = "http://finance.yahoo.com/q/op?s=" + tickerEditBox.Text;
            string HtmlFile = @".\localfile.html";

            using (WebClient client = new WebClient ()) // WebClient class inherits IDisposable
            {
                client.Proxy = null;
                //client.DownloadFile(URL, @".\localfile.html");
                HtmlWeb hw = new HtmlWeb();
                HtmlAgilityPack.HtmlDocument htmlDoc = hw.Load(URL);
                if (htmlDoc.DocumentNode != null)
                {
                    foreach (HtmlNode text in htmlDoc.DocumentNode.SelectNodes("//table/tbody/tr/td/text()"))
                    {
                        Console.WriteLine(text.InnerText);
                    }
                }

            }

        }

Answer 1

For an alternative to HtmlAgilityPack try CsQuery (on nuget as “CsQuery”), you can use CSS selectors and the jQuery API which are may be more familiar and make it easy to do this kind of parsing. Here is how I would do it with CsQuery:

string URL = "http://finance.yahoo.com/q/op?s=MSFT&m=2012-09";

CQ doc = CQ.CreateFromUrl(URL);

// The two tables have a class "yfnc_datamodoutline1", but wrap an inner table 
// too.
// This selector gets the rows of the child table where the actual data lies

var rows = doc.Select(".yfnc_datamodoutline1 table tr");

// Each th header has the class ".yfnc_tablehead1" - figure out which column 
// to use for the four parts you are interested in by finding the appropriate
// header column based on the title, and grabbing it's index

var headers = rows.First().Find(".yfnc_tablehead1");

int strikeIndex = headers.Filter(":contains('Strike')").Index();
int symbolIndex = headers.Filter(":contains('Symbol')").Index();
int bidIndex = headers.Filter(":contains('Bid')").Index();
int askIndex = headers.Filter(":contains('Ask')").Index();

// iterate over all rows, except the header one (the "has" excludes the header 
// row)

foreach (var row in rows.Has("td")) {
    CQ cells = row.Cq().Find("td");

    string output = String.Format("Strike: {0} Symbol: {1} Bid: {2} Ask: {3}",
        cells[strikeIndex].Cq().Text(),
        cells[symbolIndex].Cq().Text(),
        cells[bidIndex].Cq().Text(),
        cells[askIndex].Cq().Text());

     Console.WriteLine(output);
}

If you are familiar with CSS & jQuery the methods & selectors should make sense except for the Cq() method in the last loop. This just wraps an element as a CQ object so you can use the jQuery API against it. This is exactly the same as you would do in jQuery with $(row) to wrap a DOM element. That is, when you iterate over a jQuery object, you get actual DOM elements, not more jQuery objects, so if you want to use the jQuery API against each element in a loop, you need to wrap them in jQuery again. This is how you do it in CsQuery. The same loop in jQuery would be coded like:

rows.Has("td").each(function(i,row) {
    var cells = $(row).find("td"); 
    ..
});

I have tested this code and it works and returns output like this:

Strike: 20.00 Symbol: MSFT120922C00020000 Bid: N/A Ask: N/A
Strike: 21.00 Symbol: MSFT120922C00021000 Bid: N/A Ask: N/A
Strike: 22.00 Symbol: MSFT120922C00022000 Bid: N/A Ask: N/A
Strike: 23.00 Symbol: MSFT120922C00023000 Bid: N/A Ask: N/A
Strike: 24.00 Symbol: MSFT120922C00024000 Bid: N/A Ask: N/A
Strike: 25.00 Symbol: MSFT120922C00025000 Bid: N/A Ask: N/A

…