I have hosting setup on hostgator which creates addon domains inside a subdirectory of the main site. So the structure is
public_html
- main_site
- main_site_folders
- addon_site1
- addon_site2
and so on. If I disallow addon_site1 and addon_site2 from robots.txt in main site will it stop search engines from indexing the addon sites? I am not able to find a direct answer to this, if somebody can point to me a document that explains this concept without ambiguity, it’ll be appreciated.
If you block
/main_site/addon_site, then robots would not go towww.main_site.com/addon_site. However, that wouldn’t prevent them from going towww.addon_site.comdirectly.If you want to prevent them from indexing
www.addon_site.com, you’ll need to put a robots.txt file in theaddon_sitefolder:You need to clarify in your question what you’re trying to do.