Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
The Robot Exclusion document says:
It is not enforced by anybody, and there no guarantee that all current and future robots will use it. Consider it a common facility the majority of robot authors offer the WWW community to protect WWW server against unwanted accesses by their robots.
Is that mean a BAD robot, not implementing Robot Exclusion Protocal (robots.txt) , can read any content of a website?
Yes. This protocol is a means to mark up portions of a site which it makes no sense to crawl for robots. It is not a security facility.
Which is to say, any sensitive or private information served up by a web site should be available only within an authenticated session. It is not enough to use a URL-based-on-GUID-with-no-links-to-it or robots.txt to protect such data.
There were some embarrassing incidents, e.g. when a search engine unintentionally indexed (and cached) Send SMS confirmation pages of a cellphone provider, altogether with SMS texts and phone numbers of recipients.