Let’s say I’ve published a few WordPress posts with the following URLs:
http://example.com/2011/08/show_this/
http://example.com/2011/08/hide_this/
http://example.com/2009/04/hide_this_too/
If I want to hide all the posts that are named with the prefix “hide_” from search engines but show all others, how can this be achieved with robots.txt?
Adding this to your robots.txt should do it.