Question

I have a csv file looks like this

col1, col2, col3
1 ,  John, ABC

I need to parse ABC in column 3 into A,B,C because the new relation needs col1, col3 (each letter).

$j is the row, and $i is the column.

$j = 0;  
while (($row = fgetcsv($handle, 1000, ',')) !== false) {
    $i = 0;
    $id = 0;
    foreach ($row as $field) {
        # save the id of each row
        if($j >0 && $i == 0){
            $id = $field;
        }

        # parse the third column for non-header rows)
        if($j > 0 && $i >= 3){
            $char_array = str_split($field);
            foreach($char_array as $value){
                //echo $value;
                //echo $id.$value."<br/>";
                mysql_query("INSERT INTO sample VALUES('".$id."', '".$value."')");
            }
        }
        $i = $i + 1;  # increment the column
    }
    $j = $j +1;   # move to the next row
}

1. This is slow. I have more than 720 rows, and most of them have more than 5 characters in the third column. ABCDE, so on average we have 720 x 5. That’s a huge number.
   I get timeout. I can change the max time for execution locally, but I need to run this on my student linux account. I don’t have that kind of privilege.

2. I suspect because of timeout (it didn’t say it this time), I have up id = 502. I am missing the rest (I have up to 720 rows).

What can I do? I am only interested in the first and third column.

Answer 1

IO disk read/write is slow too. But it doesn’t timeout and with cache you are not going to perform that bad actually. In fact, I’ve done this all the time, with data of size larger than 20MB in PHP. Not that bad…. If I really want to do big data, I’d use Python, C++ algorithms because I have them ready anyway.

Example, based on your code (might contain errors):

$big = array(); 
$j = 0;
$k = 0;

while(....) {

    ....
    foreach($char_array as $value){
        #mysql_query("INSERT INTO sample VALUES('".$id."', '".$value."')");
        $big[$k][0] = $id;
        $big[$k][1] = $value;
        $k = $k+1;
    }

  ....
}

Here you make a nested array. $big is the outer array, and inside you assign the column and row.

Use fputcsv to write into a new csv. The new csv, according to your specification, should looks like this:

1, A

1, B

and here is the code to write into csv (from php doc)

$fp = fopen('new.csv', 'w');

foreach ($big as $fields) {
    fputcsv($fp, $fields);
}
fclose($fp);

}

now you can load csv into db.

---

EDIT.
I just tried with 720 rows with data like this and copy and paste 720 times.

1, ABCDEFGHIJK

Didn’t take that long… maybe 1-2 seconds.