Question

Since Google Can Now Execute AJAX & JavaScript For Indexing, what does that mean for seo for google?

What do we have to do differently, and what can we change so google bots don’t crawl through everything.

Before we had the luxury that did not happen, but now it seems that it is. And that could add to increased bandwidth due to the google crawlers.

How can we prevent google (and any other search engine in the future) from indexing our ajax code?

Answer 1

First, just because Google does it doesn’t mean anybody else does. It also doesn’t mean that they’re officially doing it for anybody other than the sites listed in the article or that they’re doing it well yet.

Second, you can include a robots.txt file that will tell Google not to index certain pages.

Here are some links for more information about robots.txt files and Google’s web crawlers:

http://www.google.com/support/webmasters/bin/answer.py?answer=156449
http://code.google.com/web/controlcrawlindex/docs/getting_started.html
http://code.google.com/web/controlcrawlindex/docs/robots_txt.html

http://code.google.com/web/controlcrawlindex/docs/crawlers.html