Question

I’m working with an unstructured plain text file. In addition to a lot of clutter, the file includes blocks of texts that are separated from the rest of the text by empty lines.

How can I use PHP to extract all blocks of texts with more than 100 words?

Answer 1

Depending on how large the file is or could be gives different approaches.

1. The simplest approach would be if you were dealing with small enough files that handling it all in memory was a feasible option. Then you could simply use a regular expression to split up all the chunks of text, then loop through and get all the chunks larger than 100 words.

2. The safest I think would be to open the file and fetch lines one at a time until you reach an empty line. If the total words in that block are more than 100 then store the block. Then continue with the next block.

Here’s an example:

// Option 1
$contents = file_get_contents($filename);
$blocks = array();
// Split the contents by 2 line breaks in a row, plus any extra ones.
// i.e. 3 blank lines in a row will be treated the same as 1 blank line.
foreach(preg_split('/\n\n\n*/m', $contents) as $block) {
    if (str_word_count($block, 0) > 100)
        $blocks[] = $block;
}

// Option 2 - longer but does not store the contents in memory.
$blocks = array();

$fp = fopen($filename, 'r');

$block = '';
while($line = fgets($fp)) {
    if (!ctype_space($line)) { // depends on your meaning of an empty line
        $block .= $line;
    }
    elseif ($block != '') {
        if (str_word_count($block, 0) > 100)
            $blocks[] = $block;
        $block = '';
    }
}
if (str_word_count($block, 0) > 100)
    $blocks[] = $block;
$block = '';