Question

I’m trying to parse a text file word by word and I need to be able to to store the offset of each word from the beginning of the file so that I can then find the exact instance of the word in the text file. Now I’m dealing with some very large text files so I was wondering what is the most efficient way of doing this?

EDIT: Some more details.

I will have two tables in a database. Lets call them WORDS which stores the words and REFERENCES which a reference for each word to a file.

The REFERENCES table will be something like this:

id INT PRIMARY
file_id INT /* ID of the text file */
offset INT /* Offset from the start of the file to get to the start of the word */

Then each record of WORDS has a referenced_id which related to a record in REFERENCES.

This is a simplified example. In reality I’ll have another table that relates each record from WORDS to one or more records in REFERENCES as there may be many instances of each word in many files.

The purpose of all this is to be able to show exactly where the instances of the word have been found when searching. So when I find the word I can get it’s reference and then take an excerpt from the text file.

Answer 1

most effective bu time or by memory used?

1. By time – load all file in memory and parse it.
   And then popup each word, storing it (word) length

$page = file_get_contents( $file );
$arr_w = explode( ” “, $page );

$arr_c = array(); count of words
$arr_o = array(); offset of words 🙂

$c = 0;
foreach ($arr_w as $w)
{
$c++;
str_replace( $page, $w, “”, 1);
$arr_c[$w]++;
$arr_o[$c] = strlen($w) + 1;
}

This is just idea …

1. By memory usage, – your will need continuously read file and analise. It is much differs from this example …