Question

If you go to wordpress admin and then settings->privacy, there are two options asking you whether you want to allow your blog to be searched though by seach engines and this option:

I would like to block search engines,
but allow normal visitors

How does wordpress actually block search bots/crawlers from searching through this site when the site is live?

Answer 1

According to the codex, it’s just robots meta tags, robots.txt and suppression of pingbacks:

Causes <meta name='robots' content='noindex,nofollow' /> to be generated into the section (if wp_head is used) of your site’s source, causing search engine spiders to ignore your site.

Causes hits to robots.txt to send back:

User-agent: *

Disallow: /

Note: The above only works if WordPress is installed in the site root and no robots.txt exists.

These are "guidelines" that all friendly bots will follow. A malicious spider searching for E-Mail addresses or forms to spam into will not be affected by these settings.