Question

I am using the HtmlAgilityPack to retrieve information from a web page and, at present, am using it to show values on the page itself using a ListView control in a button click method.

 protected void Button1_Click(object sender, EventArgs e)
 {
     string url = TextBox1.Text.ToString();
     var webGet = new HtmlWeb();
     var document = webGet.Load(url);
     // Below code crawl the data and store in generic IEnumerable<T> fashion //
     var TheWeb = 
           from info in document.DocumentNode.SelectNodes("//div[@class='article-listing']//div[@class='media-data']")
           from link in info.SelectNodes("h4//a").Where(x => x.Attributes.Contains("href"))
           from content in info.SelectNodes("p").Where(y => y.HasAttributes != true)
           from author in info.SelectNodes("p[@class='article-meta']//a[@rel='author']").Where(z => z.Attributes.Contains("href"))
           from date in info.SelectNodes("p[@class='article-meta']//span")
           select new
           {
                 LinkURL = link.Attributes["href"].Value,
                 Text = content.InnerText,
                 Author = author.InnerText,
                 Date = date.InnerText
           };
     lvLinks.DataSource = TheWeb;
     lvLinks.DataBind(); 
}

But now I want to store the data in SQL Server and want to run the code using some function (instead of button click).

For that I want to store the data in some other form instead of an IEnumerable<> style which uses LINQ to extract values.

Please suggest.

Answer 1

You can have a custom class with the structure

public class ParseData
{
    public string LinkURL { get; set; }
    public string Text { get; set; } 
    public string Author { get; set; }
    public string Date { get; set; }
}

Populate it with your query

var TheWeb = 
       from info in document.DocumentNode.SelectNodes("//div[@class='article-listing']//div[@class='media-data']")
       from link in info.SelectNodes("h4//a").Where(x => x.Attributes.Contains("href"))
       from content in info.SelectNodes("p").Where(y => y.HasAttributes != true)
       from author in info.SelectNodes("p[@class='article-meta']//a[@rel='author']").Where(z => z.Attributes.Contains("href"))
       from date in info.SelectNodes("p[@class='article-meta']//span")
       select new ParseData
       {
             LinkURL = link.Attributes["href"].Value,
             Text = content.InnerText,
             Author = author.InnerText,
             Date = date.InnerText
       };
var parseData = TheWeb.ToList();

And now using System.Xml.Serialization.XmlSerializer serialize this data in XML. Store this XML in your DB and retrieve when ever required.

Tutorial explaining how to Serialize object in XML and Deserialize back to object.

Hope this works for you.