I am running calls onto PayPal’s TransactionSearch API through PHP cURL.
Unfortunately, the API is very slow to respond, sometimes taking anywhere from 30 seconds to more than 5 minutes (depending on the number of records returned from the API) for a single customer.
At the moment, the script is running off a cron job, and looping through each customer one by one. However, if the number of customers scale up, the entire process would take a very long time (few hours) to complete. This is not good enough.
Essentially, I need to run (and process) multiple API calls simultaneously. What’s the best way to achieve this?
Since the bottleneck is the remote server, I suggest using curl_multi_exec. You’ll be processing a big number of HTTP connections at once and then process their results in one thread.
This is not the fastest solution, which would be to process responses as soon as they’re available in multiple threads, but this approach can make processing 50+ times faster without significant changes.