I am working on a website which loads its data via AJAX. I also want that the whole website can be crawled by search engines like google and yahoo.
I want to make 2 versions of the site…
[1] When a user comes the hyperlinks should work just like GMAIL (#’ed hyperlinks)
[2] When a crawler comes the hyperlinks should work normally (AJAX mode off)
How can i identify a Crawler??
You should not present a different form of your website to your users and a crawler. If Google discovers you doing that, they may reduce your search ranking because of it. Also, if you have a version that’s only for a crawler, it may break without you noticing, thus giving search engines bad data.
What I’d recommend is building a version of your site that doesn’t require AJAX, and having prominent links on each page to the non-AJAX version. This will also help users who may not like the AJAX version, or who have browser which aren’t capable of handling it properly.