Home/ Questions/Q 570351
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-13T13:23:31+00:00

I am looking to develop a Web scraper in C# window forms. What I

  • 0

I am looking to develop a Web scraper in C# window forms. What I am trying to accomplish is as follows:

  1. Get the URL from the user.
  2. Load the Web page in the IE UI control(embedded browser) in WINForms.
  3. Allow the User to select a text (contiguous , small(not exceeding 50 chars)). from the loaded web page.
  4. When the User wishes to persist the location (the HTML DOM location) it has to be persisted into the DB, so that the user may use that location to fetch the data in that location during his subsequent visits.

Assume that the loaded website is a pricelisting site and the quoted rate keeps on changing, the idea is to persist the DOM hierarchy so that I can traverse it next time.

I would be able to do this if all the HTML elements had their id attributes. In the case where the id is null , i am not able to accomplish this .

Could someone suggest a valid idea on this (a bare minimum code snippet if possible).?

It would be helpful , even if you can share some online resources.

thanks,

vijay

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    One approach is to build a stack of tags/styles/id down to the element which you want to select.

    From the element you want, traverse up to the nearest id element. This way you will get rid of most of the top header etc. Then build a sequence to look for.

    Example:

    <html>
  <body>
    <!-- lots of html -->
    <div id="main">
       <div>
          <span>
             <div class="pricearea">
                <table> <!-- with price data -->

    For the exmaple you would store in your db a sequence of: [id=main],div,span,div,table or perhaps div[class=pricearea],table.

    Using styles/classes might also be used to create your path. It’s your choice to look for either a tag, an attribute of a tag or a combination. You want it as accurate as possible with as few elements as possible to make it robust.

    If the layout seldom changes, this would let you navigate to the same location each time.

    I would also suggest you perhaps use HTML Agility Pack or something similar for the DOM parsing, as the IE control is slow.

    Screen scraping is fun, but it’s difficult to get it 100% for all pages. Good luck!

    • 0