Question

My system stores products from many different e-shops and I need to pair products according to their names. For example:

INPUT: MySQL table products

id | name                       | id_seller
1    porsche 911 red edition      1
2    red porsche 911 gt           2
3    icecream                     1

DESIRED OUTPUT: Suggestion that product 1 is similar to product 2.

In the first step it would be sufficient to make suggestions just on the number of common words – 3 out of 4 in this Porsche example.

More sophisticated solution would involve comparing the order of words not just their occurrences, but I guess it wouldn’t be trivial.

Can it be done using just MySQL query and its built-in functions or any sophisticated library/add-on has to be used?

Answer 1

Here is a SQLFiddle example to find pairs of products with at least one common word in the name column:

select t.id id1, t.name name1, t1.id id2, t1.name  name2 from t
join t t1
where t.id<t1.id
      AND
      t.name regexp 
        CONCAT('([[:<:]]',
               REPLACE(
                   TRIM(t1.name),
                   ' ',
                   '[[:>:]]|[[:<:]]'
               ),
               '[[:>:]])');

If you need to find lines with at least N common words you should create tmp table splitting each row to words. Here is an example and stored procedure to do it. For your example this table looks like:

id | name                       
1    porsche
1    911
1    red
1    edition      
2    red
2    porsche
2    911
2    gt           
3    icecream                     

in this case you can use following query to find id’s with at least N common words (in this case N=3):

select t1.id,t2.id,count(*)
from tmp t1,tmp t2
where t1.id<t2.id and t1.name=t2.name
group by t1.id,t2.id having count(*)>=3