Question

On my site I have a directory of things which is generated through jquery ajax calls, which subsequently creates the html.
To my knwoledge goole and other bots aren’t aware of dom changes after the page load, and won’t index the directory.

What I’d like to achieve, is to serve the search bots a dedicated page which only contains the links to the things.

Would adding a noscript tag to the directory page be a solution? (in the noscript section, I would link to a page which merely serves the links to the things.)

I’ve looked at both the robots.txt and the meta tag, but neither seem to do what I want.

Answer 1

It looks like you stumbled on the answer to this yourself, but I’ll post the answer to this question anyway for posterity:

Implement Google’s AJAX crawling specification. If links to your page contain #! (a URL fragment starting with an exclamation point), Googlebot will send everything after the ! to the server in the special query string parameter _escaped_fragment_.

You then look for the _escaped_fragment_ parameter in your server code, and if present, return static HTML.

(I went into a little more detail in this answer.)