Question

I am looking for a way to automate a task which is very time consuming. We have manufacturer name and product number which we search on web to find the site that has both the information or rather search for manufacture or the vendor sites and use the information on the link to populate relevant fields in our database.

Searching process manually consumes lots of time, what i am looking for is for a tool which can take name and product # as input go on web and search for sites which matches the input and dump the links for those.

Can anyone help me here!

Answer 1

What you’re trying to achieve requires quite a bit of work and I’m afraid you didn’t provide enough information for me to dive into details. However, in broadlines, a simple crawling algorithm would require the following steps:

• create an instance of HttpWebRequest for the required url. If you need to perform a POST request, use GetRequestStream() method to populate the body.
• call GetResponse() on the request to get the response from server. Use GetResponseStream() to read the HTML markup from the response;
• use HtmlAgilityPack to iterate through DOM nodes and select required data.
• use Fiddler to see what is the difference between what a broswer sends and what your application sends.
• last but not least, should you encounter a problem that you cannot solve or don’t know the answer to – you already know how to use StackOverflow 🙂