Question

I have a page that has a form using this ajaxForm jQuery plugin. The form submits, and when it’s complete, there is a call using $.get() to load some new content to the page.

My problem is, the Googlebot “appears” to be indexing the url in the $.get() method.

My first question is, is that even possible? I was under the impression the Googlebot didn’t evaluate javascript for the most part (I read something about it being able to index content on urls with !#).

My second question is, if Google is indexing this call to that url, is there a way to prevent it?

Thanks in advance.

Answer 1

You could robots.txt the file specifically, googlebot will should honor it.

From robotstxt.org:

User-agent: *
Disallow: /~joe/junk.html
Disallow: /~joe/foo.html
Disallow: /~joe/bar.html

You can also look at Google’s Webmaster Central to remove the file from the listing.