Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
One website stores information in ids, like that http://website.com/index.php?id=1. It has about 500,000 records.
I need to retrieve them and put it in database (MySQL). Now I use curl and it gets that information, but it’s very slow. I got only ~4,000 records in one hour.
My question, is there any way to do this faster?
If you’re not against using perl, the CPAN module Web::Scraper is the conventional way for scraping pages efficiently. See http://metacpan.org/pod/Web::Scraper .