Question

I have a system that basically copies data from other databases to my own SQL Server so I can format the data to suit my needs. That data is used to populate dropdowns to end up with a variable can return the rows I want to show to the user.

All of the databases I’ve encountered up to now had a logical way the data of the dropdowns was stored. But now I have one that is different and not logical.

What I want to do is use the existing website that provides a somewhat logical dropdown itself to get the data I need. That dropdown fills a multiple selection dropdown after a selection.

I want to select each option in the first dropdown and then read the data it puts in the second multiple selection dropdown. Is there a way to do this?

Answer 1

What you essentially want to do is web scraping. In a similar project, this is what I did:

1. Use the available objects in .NET that can send http requests and can receive http responses. In my case, I used WebClient:

       byte[] response;
       using (var client = new WebClient())
       {
           try
           {
               response = client.UploadValues(URL, "POST", postData);
           }
           catch (Exception ex)
           {
               // Log error here                  
           }
       }
   

2. See the postData? This is a NameValueCollection that contains the data that is sent by the page when you do a POST. Use Fiddler or a similar tool to find out what data needs to be included. If the page uses .NET web forms, you may have to do something special for the __VIEWSTATE, like do a sample request and get the __VIEWSTATE value from there. Here’s a sample for postData:

       var postData = new NameValueCollection();
       postData.Add("__EVENTTARGET",  "");
       postData.Add("__EVENTARGUMENT", "");
       postData.Add("__VIEWSTATE", "");
   

3. Use HtmlAgilityPack to easily “parse” the response that you get when you post. Using Xpath, you will be able to get the new items in the second dropdown.

       var doc = new HtmlDocument();
       doc.Load(new MemoryStream(requestBytes));
       return doc;