Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
Currently I’m using Mechanize and the get() method to get each site, and check with content() method each mainpage for something.
I have a very fast computer + 10Mbit connection, and still, it took 9 hours to check 11K sites, which is not acceptable, the problem is, the speed of the get() function, which , obviously, needs to get the page,is there any way to make it faster,maybe to disable something,as I only need to main page html to be checked.
Thanks,
Make queries in parallel instead of serially. If I needed to do this, I’d fork off a process to grab the page. Something like Parallel::ForkManager, LWP::Parallel::UserAgent or WWW:Curl may help. I tend to favor Mojo::UserAgent.