Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
Is there any fast (maybe multi-threaded) way to crawl my site (clicking on all local links) to look for 404/500 errors (i.e. ensure 200 response)?
I also want to be able to set it to only click into 1 of each type of link. So if I have 1000 category pages, it only clicks into one.
Is http://code.google.com/p/crawler4j/ a good option?
I’d like something that is super easy to set up, and I’d prefer PHP over Java (though if Java is significantly faster, that would be ok).
You can use the old and stable Xenu tool to crawl your site.
You can configure him to use 100 threads and sort the results by status code[500\404\200\403]