Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
Our site is receiving requests which try to access non existing pages. Usually it is only case of misspelled address. Like the right link is /Info/SiteMap.aspx, but the bad request is /Inf/SiteMap.aspx with missing ‘o’. There are usually many request, like 20 in row in one minute. All throw 404.
Is it problem of some indexing robot? Or does someone test gaps in our site?
Have any experiences or tips?
Yes, Consider the following things.
The IP address of the host sending the requests – If they are distinct but several IPs we can suspect it as a distributed attack done using Zombies.
. If the requests are coming from same source, check the delay between requests. Generally crawlers do not use extremely short periods between the requests.
Indexing Robots (Crawlers) do not perform “Brute force” type indexing. They just retrieve the links from one page and recursively traverse page by page. So the reason for this should be not a indexing robot.
Check for any patterns. I mean a sequential naming pattern etc.