Question

I can’t get into too many specifics as this is a project for work, but anyways..

I’m in the process of writing a SOAP client in PHP that pushes all responses to a MySQL database. My main script makes an initial soap request that retrieves a large set of items (approximately ~4000 at the moment, but the list is expected to grow into hundreds of thousands at some point).

Once this list of 4000 items is returned, I use exec(“/usr/bin/php path/to/my/historyScript.php &”) that sends a history request for each item. The web service api supports up to 30 requests / sec. Below is some pseudo code for what I am currently doing:

$count = 0;

foreach( $items as $item )
{
    if ( $count == 30 )
    {
      sleep(1); // Sleep for one second before calling the next 30 requests
      $count = 0;
    }

    exec('/usr/bin/php path/to/history/script.php &');
    $count++;
}

The problem I’m running into is that I am unsure when the processes finish and my development server is starting to crash. Since data is expected to grow, I know this is a very poor solution to my problem.

Might there be a better approach I should consider using for a task like this? I just feel that this is more of a ‘hack’

Answer 1

I am not sure, but i feel that the reason for your application crash, you are keeping large set of data in PHP variable. Look into this, based on RAM size this(data size) will leads to system crash. And my suggestion is try to limit incoming data from external service per request, instead number of request to the service.