Question

I have a web portal where you can add new users to a hosted platform, when you add a user via the form it saves the user information into a mysql table (pdo), actually probably 3 or 4 tables will be inserted into.

Also a SOAP request is sent to an external site and the response read, depending on the user options further soap requests may be sent.

Also a connection is made to an smpp provider to send out an SMS.

So just adding the 1 user can take a fair amount of execution time. I want to make a bulk import csv option, upload the file and process each row as a new user to be added.

How would be the best way to say import 100 users without errors and time outs left right and centre?

I was thinking about an ajax request per user, as seen here how to put each php foreach value into each jquery ajax

But the guy who answered also says “However, I would highly recommend passing the values as an array and having only one ajax call”. But wouldn’t this be prone to timeouts?

Ideally i feel the best way would be to add them one at a time, and get a result per user.

Answer 1

You could upload the CSV file and have the server respond immediately if the upload was successful. The server could stick the CSV in a queue and have a scheduler pull the file from the queue and do all operations on a separate process. The client would periodically send an ajax request to check if the server has completed the bulk operations. If completed the server would respond to the ajax call with a summary of all the operations on all items in the CSV and any errors. I wouldn’t send an ajax call per item in the CSV that defeats the whole purpose of batching the items.

Are you running on Linux? If so an example would be to use cron to execute your batch processing php script every minute. Eg in you crontab file:

* * * * * php /path/batch_processor.php

The queue of files could be stored in a DB table which records upload time, processing start and finish etc and the php script just pulls the first unprocessed file and goes to work.

Alternatively you could start a background process immediately on uploading the CSV file using exec instead of using the scheduler. PHP manual says:

If a program is started with this function, in order for it to
continue running in the background, the output of the program must be
redirected to a file or another output stream. Failing to do so will
cause PHP to hang until the execution of the program ends.

So you could use something like so in your upload handler.

exec("nohup php /path/batch_processor.php 1 > batch_output.txt  2 > batch_error.txt &");

The 1 > and 2 > redirect the script output and error; & indicates to run in background; nohup ignores hangup signals eg if process owner session ends.

If you’re on windows then search around, there’s plenty of info out there on running background processes.

When your batch script has completed it could flag the csv file as completed in a db table or another shared resource so when you handle your ajax call to check on the status of the csv batch you can simply lookup and check the completed flag and return the batch output to the client if it is finished.

If you wanted to improve the functionality and give incremental status updates for each ajax call regardless of whether the batch has finished then your batch process would write to a shared resource (db or file etc) for each item it processed and your ajax handler would read from this and return to the client.