I have a website which has two domains added. Both domains point to the root of the website. Is it possible to alter the robots.txt so that one of the domains doesn’t get crawled, while the other still does?
Share
Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I have a website which has two domains added. Both domains point to the root of the website. Is it possible to alter the robots.txt so that one of the domains doesn’t get crawled, while the other still does?
You ought to perform a 301 redirect to your preferred domain, which tells robots what URL you want crawled.
For example
Should 301 redirect to
Similarly
Should 301 redirect to