I have disallowed certain pages using robots.txt for all crawlers. Do I have to write metatags for those files or web crawlers will just skip them so there is no need to do so?
Share
Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I have disallowed certain pages using robots.txt for all crawlers. Do I have to write metatags for those files or web crawlers will just skip them so there is no need to do so?
You are good to go. All of the big search engines (Google, really) obey any entries you make in
robots.txt. http://www.robotstxt.org/robotstxt.htmlAlso, be aware that the
robots.txtfile itself is viewable, so don’t use this as a security measure. http://www.cre8asiteforums.com/forums/index.php?showtopic=55546