Is this a good idea??
What does abusive crawling mean? How is that bad for my site?
Is this a good idea?? http://browsers.garykeith.com/stream.asp?RobotsTXT What does abusive crawling mean? How is that
Share
Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
Is this a good idea??
What does abusive crawling mean? How is that bad for my site?
Not really. Most “bad bots” ignore the robots.txt file anyway.
Abuse crawling usually means scraping. These bots are showing up to harvest email addresses or more commonly, content.
As to how you can stop them? That’s really tricky and often not wise. Anti-crawl techniques have a tendency to be less than perfect and cause problems for regular humans.
Sadly, like “shrinkage” in retail, it’s a cost of doing business on the web.