Question

I use the following code to display RSS news topic and summary, it works fine but I want to display the whole news in my own site, now my users can click on news link and whole news is visible in the main site, but I want it to be displayed in my site. is is possible? can I use RSS to display the whole content in my own site or I should parse the main news page and display it in my site? I think it is very difficult to parse this page.

my server code:

//Create a WebRequest
        WebRequest rssReq =
        WebRequest.Create("myRSSsite");
        //Create a Proxy
        WebProxy px = new WebProxy("myRSSsite", true);
        //Assign the proxy to the WebRequest
        rssReq.Proxy = px;
        //Set the timeout in Seconds for the WebRequest
        rssReq.Timeout = 5000;
        try
        {
            //Get the WebResponse
            WebResponse rep = rssReq.GetResponse();
            //Read the Response in a XMLTextReader
            XmlTextReader xtr = new XmlTextReader(rep.GetResponseStream());
            //Create a new DataSet
            DataSet ds = new DataSet();
            //Read the Response into the DataSet
            ds.ReadXml(xtr);
            //Bind the Results to the Repeater
            rssRepeater.DataSource = ds.Tables[2];
            rssRepeater.DataBind();

also I use a repeater on my ASPX page to display RSS, but I have a list containing news title (linking to the main site) and a small summary, nothing more! is there any way I can get the whole news HTML so that I can store it in my database?

Answer 1

Site scrape the news article using Html Agility Pack:

HttpWebRequest oReq = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse resp = (HttpWebResponse)oReq.GetResponse();

var doc = new HtmlAgilityPack.HtmlDocument();

doc.Load(resp.GetResponseStream());
var element = doc.GetElementbyId("Id-of-the-article-body-element");
string content = element.OuterHtml;

If the article body element doesn’t have an ID, you have to use an XPath query to find it. Take a look at this tutorial on HTML Agility Pack.