Question

The web app I’m building has a JavaScript powered interface and fetches data from the server using AJAX. Everyone is on the same “page” but the data after the hashtag in the URL is used to determine which data to load, which is then displayed on the page. An example of a URL in my web app might be http://www.myapp.com/#/user/stackmaster. JavaScript sees the data after the hashtag (in this case “user” and “stackmaster”) and uses AJAX to load the user whose username is stackmaster, and then displays it on the screen. The reason I structure my URLs like this because I want search engines to be able to index individual pages.

Is it possible to have an AJAX based web application like this be able to find and index individual pages? It is my understanding that web crawlers such as Google can’t index dynamic content loaded with AJAX, right? Are there any alternate techniques to help the search engines find this data?

Thanks in advance for your help.

Answer 1

If you’re running an AJAX application with content that you’d like to
appear in search results, Google has a new process that, when
implemented, can help Google (and potentially other search engines)
crawl and index your content. Historically, AJAX applications have
been difficult for search engines to process because AJAX content is
produced dynamically by the browser and thus not visible to crawlers.
While there are existing methods for dealing with this problem, they
involve regular manual maintenance to keep the content up-to-date.

GETTING STARTED

https://developers.google.com/webmasters/ajax-crawling/

An agreement between crawler and server

In order to make your AJAX application crawlable, your site needs to abide by a new agreement. This agreement rests on the following:

The site adopts the AJAX crawling scheme. For each URL that has
dymanically produced content, your server provides an HTML snapshot,
which is the content a user (with a browser) sees. Often, such URLs
will be AJAX URLs, that is, URLs containing a hash fragment, for
example http://www.example.com/index.html#key=value, where #key=value is the
hash fragment. An HTML snapshot is all the content that appears on the
page after the JavaScript has been executed. The search engine indexes
the HTML snapshot and serves your original AJAX URLs in search
results.

https://developers.google.com/webmasters/ajax-crawling/docs/learn-more

http://googlewebmastercentral.blogspot.in/2007/11/spiders-view-of-web-20.html