Question

I’m testing a php script to create csv file which containing large amount of data.
This is the way I do this task:

$csvdata = "ID,FN,LN,ADDR,TEL,PRO\n
            1,fn1,ln1,addr1,tel1,pro1\n...etc,";
$fname = "./temp/filename.csv";
$fp = fopen($fname,'w');
fwrite($fp,$csvdata);
fclose($fp);

I have notice that when the string ($csvdata) contain like 100,000 data rows the script work fine. But when it gets about more that 10,00,000 data rows it stop in the middle where I build the string $csvdata (I’m building $csvdata string by concatenating data in a for loop, data from database).

Could someone let me know what’s went wrong when we use large string value?

Thank you
Kind regards

Answer 1

I’m building $csvdata string by concatenating data in a for loop, data from database.

From this I gather that you are preparing the entire file to be written as a string and finally writing it to the file.

If so, then this approach is not good. Remember that the string is present in the main memory and every time you append, you are consuming more memory. The fix for this is to write the file line by line that is read a record from the database, transform/format if you want to, prepare the CSV row and write it to the file.