I’ve created my website so
site.com/#!/page/var1/ans1/var2/ans2
maps onto
site.com/pages/page.php?var1=ans1&var2=ans2
Using Javascript. I have also made it so
site.com?_escaped_fragment_=/page/var1/ans1/var2/ans2
maps onto
site.com/pages/page.php?var1=ans1&var2=ans2
Using PHP so my website will be apparently crawlable with Google. I have tested it with Google Fetch and it seems to work. However all the other non Google spiders I have tried to test my website don’t work with the google techniques of using _escaped_fragment_. I want to make sure when the Google spider goes to site.com, it will see x, go to site.com?_escaped_fragment_=/page and grab it properly. Are there any ways I can make sure it’s grabbing all the links on my website and crawling them? The test spider sites I’ve tested only list the links with the full path of the domain in them – but does that mean for ALL the links on my website I have to put the entire domain before them? That will ruin my ajax.
For the ‘non google spiders’; In Bing Webmaster Tools, on the Crawl tab – sub tab Crawl Settings – on the bottom of the page is a checkbox asking ‘SITE SUPPORTS AJAX CRAWLABLE URLs’ – URL’s that include the #! . If you haven’t checked it, try setting it and see if Bing will crawl your pretty URLs. This will probably help Slurp crawl the site as well(??)
As for ‘any ways I can make sure it’s grabbing all the links on my website and crawling them’ just google site:nameOfYoursite.com and all of the URLs in the Google index for your site will come up on the SERP.