I have a webcrawler application. It successfully crawled most common and simple sites. Now i ran into some types of websites wherein HTML documents are dynamically generated through FORMS or javascripts. I believe they can be crawled and I just don’t know how. Now, these websites do not show the actual HTML page. I mean if I browse that page in IE or firefox, the HTML code does not match what’s actually in the IE or firefox. These sites contain textboxes, checkboxes, etc… so I believe they are what they call “Web Forms”. Actually I am not much familiar with web development so correct me if I’m wrong.
My question is, does anyone in similar situation as I am now and have successfully solved these types of “challenges”? Does anyone know of a certain book or article regarding web crawling? Those that pertains to these advanced type of websites?
Thanks.
I found an article which tackles deep web and its very interesting and I think this answers my questions above.
http://www.trycatchfail.com/2008/11/10/creating-a-deep-web-crawler-with-net-background/
Gotta love this.