Question

I have a database of thousands (about 10,000) keywords. When a user posts a blog on my site, I would like to automatically search for the keywords in the text, and tag the post with any direct matches.

So far, all I can think of is to pull the ENTIRE list of keywords, loop through it, and check for the presence of each tag in the post…which seems very inefficient (that’s 10,000 loops).

Is there a more common way to do this? Should I maybe use a MySQL query to limit it down?

I imagine this is not a totally rare task.

Answer 1

No, just don’t do that.

Instead of looping through 10000 elements, it is better to extract the words from the sentence or text, then add it to the SQL query and that way you will have all the needed records. This is surely more efficient than the solution you proposed.

You can do this in the following way using PHP:

$possible_keywords = preg_split('/\b/', $your_text, PREG_SPLIT_NO_EMPTY);

The above will split the text on the words’ boundaries and will return no empty elements in the array.

Then you just can create the SQL query in a fashion similar to the following:

SELECT * FROM `keywords` WHERE `keywords`.`keyword` IN (...)

(just put the comma-separated list of extracted words in the bracket)

You should probably filter the $possible_keywords array before making the query (to include only the keywords with appropriate length and to exclude duplicates) plus make keyword column indexed.