Question

I am using PHP to import data from a CSV file using fgetcsv(), which yields an array for each row. Initially, I had the character limit set at 1024, like so:

while ($data = fgetcsv($fp, 1024)) {
  // do stuff with the row
}

However, a CSV with 200+ columns surpassed the 1024 limit on many rows. This caused the line read to stop in the middle of a row, and then the next call to fgetcsv() would start where the previous one left off and so on until an EOL was reached.

I have since upped this limit to 4096, which should take care of the majority of cases, but I would like put a check in to be sure that the entire line was read after each line is fetched. How do I go about this?

I was thinking to check the end of the last element of the array for end of line characters (\n, \r, \r\n), but wouldn’t these be parsed out by the fgetcsv() call?

Answer 1

Thank you for the suggestions, but these solutions really didn’t solve the issue of knowing that we account for the longest line while still providing a limit. I was able to accomplish this by using the wc -L UNIX command via shell_exec() to determine the longest line in the file prior to beginning the line fetching. The code is below:

// open the CSV file to read lines
$fp = fopen($sListFullPath, 'r');

// use wc to figure out the longest line in the file
$longestArray = explode(" ", shell_exec('wc -L ' . $sListFullPath));
$longest_line = (int)$longestArray[0] + 4; // add a little padding for EOL chars

// check against a user-defined maximum length
if ($longest_line > $line_length_max) {
    // alert user that the length of at least one line in the CSV is too long
}

// read in the data
while ($data = fgetcsv($fp, $longest_line)) {
    // do stuff with the row
}

This approach ensures that every line is read in its entirety and still provides a safety net for really long lines without stepping through the entire file with PHP line by line.