A website may be accessed not only by a user on a browser, but also programs, bots and crawlers. I have a website running on Google App Engine with python which has non-static HTML pages that are generated by a python program by combining, merging and looping strings. However, they are also not dynamic pages in the sense that no user input is required to generate these pages. The content generation by python is solely for convenience, brevity and ease of maintenance, and is set completely by the url.
Some search engines cannot index dynamic pages. I would like to know if these pages qualify as ‘dynamic’, i.e. whether they can be crawled or indexed for the usual metadata and content by such bots, and in general would like a way to check how any url appears to a bot or crawler like the ones used by search engines, so that I can see when a certain url is uncrawlable.
If anyone knows of any resources or techniques available, it’d be really helpful.
Lynx is a text-based browser that gives you a pretty good idea of how a searchbot would see your page. Ancient, tried and true.