Question

I am creating a website in asp.net but i have some issue..

I have coded a program which can crawl a give web page i.e. thenextweb.com for its links, and content and images.

Now i want to store these crawled data inside my table *Crawlr_Data*.

I want that the crawler runs after every 30 minutes and updated the table with new links if available.

{ON the Home page of my website i am showing the information stored in the database}

How can i run the crawler on back end and update the database ?
What technology like (web services, WCF) should i use or any other thing in visual studio which i can use so that i if host website online its crawler keeps on running and updating table}

Please suggest
Thanks

Answer 1

There are two ways to do this with the Microsoft stack.

1. Create a service to run on the server. Have the service itself manage when it wakes up and crawls.

2. Create a console app to do the crawl. Run the console app as a scheduled task using windows task scheduler as often as you like.

I guess there are other ways to do it — so saying there are just two is not totally accurate — there are 3rd party programs that will do it for you also… I expect most if not all of them are implemented as a service. You could also write a program that runs on the server not as a console app or as a service. But this is generally a bad idea.