Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
We have an Umbraco site in a load balanced environment and we need to make sure only the actual URL gets crawled and not the different production URLs.
We only want example.com to be indexed while load balancers at production1.example.com and production2.example.com are not.
Do I add a disallow rule for those URLs to the robots.txt, or add a meta nofollow tag to the head? Or is there another way to have the load balancing URLs not indexed by crawlers?
Best solution: Don’t make node-specific URLs publicly available (we usually use local ip/port to check a site on a specific node).
Since you have those domains, you may serve a different robots.txt depending on the domain (using URL rewriting).