I’m working on an application that utilizes Amazon’s MWS API written mostly in PHP. This application allows Amazon sellers to sign up, provide some Amazon seller credentials and the application then begins to download this user’s orders from Amazon placing them into a MySQL database.
I’ve built the methods to sync data accurately to the database using a script with multiple functions but noticed this takes way too long. This script simply loops through all users in the database and iterates all orders one at a time. Right now, with only 5 test users, the time is doable but I’m looking for a more extensible method. Think about 500 users all running synchronously one at a time. WAY too long!
I’m pretty new to PHP and especially running asynchronous processes from it. The only way I’ve found that I can do this is to have a starter script that finds all users in the database and spawns a sync script for each user then releases it. I don’t like this idea because if I did had 500+ users, my nightly sync would consist of 500 spawned instances of this PHP script.
Has anyone done something similar to this before? If so, I’d love to hear how best to make this sync more efficient.
Agree w/DaveRandom, check out HTTP Request Pool for implementation, rather than rolling your own http://www.php.net/manual/en/class.httprequestpool.php