Question

Is it possible to delete all lines that have the same first 30 characters of the line and then only to left the first line that has these characters?

Example:

xx2 Lorem ipsum dolor sit amet, fdsfdsfs
xx2 Lorem ipsum dolor sit amet, 43434343

The second should be deleted… Hope it’s possible… Thanks

Answer 1

If you need to deal with really large files, reading only one line at a time and writing to a temp file will consume less memory. Using a temp file and renaming it to the input file when done will do the operation atomically without losing your original file. Checking for array keys instead of values will offer fast lookup since keys are indexed. You also need to handle the edge case of a blank line returning false on substr.

<?php
$infile_name = "infile.txt";

$seen = array();
$infile = fopen($infile_name, "r");
if ( $infile !== false ) {
    // Temporary file to write results to
    $outfile_name = tempnam(sys_get_temp_dir(), 'tmp');
    $outfile = fopen($outfile_name, "w");

    while (!feof($infile)) {
        $line = fgets($infile);
        if ( $line == '' ) {
            // blank line, just write it
            fwrite($outfile, $line);
        }
        else {
            $prefix = substr( $line, 0, 30 );

            if ( !array_key_exists($prefix, $seen) ) {
               fwrite($outfile, $line);

               // Store the prefix as a key for fast indexed lookup
               $seen[$prefix] = true;
            }
        }
    }

    fclose($infile);
    fclose($outfile);

    // Remove the old file and put the new file in its place
    unlink($infile_name);
    rename($outfile_name, $infile_name);
}
?>