I’m writing a web scraping app in .NET and would like to have it hosted by some party such as arvixe.
The problem is that I do not understand web hosting solutions well and in addition, I’m coming from a desktop background.
What’s the best way (worker threads to scrape on a timed loop? some kind of process) for me to write a app that runs on this host and collects data (web scraping) and then saves that to a sql server database?
I would really appreciate a little detail, as I am fairly new to the web space.
Here is the application I created some time ago that crawles web and gets some data:
https://github.com/alexanderbeletsky/TddDemand
It implements web crawler + EF4 storage.
It is actually class libraly + console application, but class library could be used as a part of ASP.NET Web Application – no problems here.
But, I don’t really understand why you need a web site for that. For this scenario simple application (service) is more suitable.