Question

I’m currently building a user panel which will scrape daily information using curl. For each URL it will INSERT a new row to the database. Every user can add multiple URLs to scrape. For example: the database might contain 1,000 users, and every user might have 5 URLs to scrape on average.

How do I to run the curl scraping – by a cron job once a day at a specific time? Will a single dedicated server stand this without lags? Are there any techniques to reduce the server load? And about MySQL databases: with 5,000 new rows a day the database will be huge after a single month.

If you wonder I’m building a statistics service which will show the daily growth of their pages (not talking about traffic), so as i understand i need to insert a new value per user per day.

Any suggestions will be appreciated.

Answer 1

1. I understand that each customer’s pages need to be checked at the same time each day to make the growth stats accurate. But, do all customers need to be checked at the same time? I would divide my customers into chunks based on their ids. In this way, you could update each customer at the same time every day, but not have to do them all at once.

2. For the database size problem I would do two things. First, use partitions to break up the data into manageable pieces. Second, if the value did not change from one day to the next, I would not insert a new row for the page. In my processing of the data, I would then extrapolate for presentation the values of the data. UNLESS all you are storing is small bits of text. Then, I’m not sure the number of rows is going to be all that big a problem if you use proper indexing and pagination for queries.

Edit: adding a bit of an example

function do_curl($start_index,$stop_index){

    // Do query here to get all pages with ids between start index and stop index

    $query = "select * from db_table where id >= $start_index and id<=$stop_index";

    for($i=$start_index; $i<= $stop_index; $i++;){

         // do curl here
    }

}

urls would look roughly like

http://xxx.example.com/do_curl?start_index=1&stop_index=10;
http://xxx.example.com/do_curl?start_index=11&stop_index=20;

The best way to deal with the growing database size is to perhaps write a single cron script that would generate the start_index and stop_index based on the number of pages you need to fetch and how often you intend to run the script.