I’m working with an API that maps my GTIN/EAN queries to product data.
Since the data returned originates from merchant product feeds, the following is almost universally the case:
- Multiple results per GTIN
- Products’ titles are pretty much unstructured
- Products’ titles are “polluted” with
- SEO-related stuff,
- information about the quantity contained,
- “buy two, get one free” offers,
- etc.
I’m looking for a programmatic way to either
- choose the “cleanest”/most canonical version available
- or generate a new one that represents the “lowest common denominator”.
Consider the following example results for a single EAN query:
- Nivea Deo Roll-On Dry Impact for Men
- NIVEA DEO Roll on Dry/blau
- Nivea Deo Roll-On Dry Impact for Men, 50 ml, 3er Pack (3 x 50 ml)
- Nivea Deo Roll on Dry/blau 50 ml
- Nivea Deoroller 50ml dry for Men blau Mindestabnahme: 6 Stück (1 VE)
- NIVEA Deoroller, Dry Impact for Men
- NIVEA DEO Roll on Dry/blau_50 ml
My homebrew approach looks like this:
- Basic cleanup:
- Lowercase the titles,
- strip excessive whitespace,
- throw out apparent stopwords such as “buy” and “click”
- Build an array for
word => global occurence"Nivea" => 7"Deo" => 5"Deoroller" => 2…"VE" => 1
- Calculate the “cumulative word value” for each of the titles
"Nivea Deo" => 12"Nivea Deoroller VE" => 10
- Divide the cumulative value by the length of the title, resulting in a score
"Nivea Deo" => 6"Nivea Deoroller VE" => 3.34
Obviously, my approach is pretty basic, error-prone and biased towards short sentences with frequently used words – yielding more or less satisfactory results.
- Would you choose a different approach?
- Is there some NLP magic way to take care of the problem that I don’t know of?
Since your existing metric seems to bias towards shorter phrases, you should consider factoring in bigrams into the mix. So instead of considering scores for just individual words, additionally consider the score for consecutive pairs of words as well (e.g. ‘nivea deo’, deo roll-on’, ‘roll-on dry’, etc). When computing the score for each title, factor in the scores for every unigram and bigram you can generate out of the title together, but maybe give the bigrams more weight, and this should encourage your algorithm to prefer longer phrases.
If you have large existing corpus of lots of names like these at your disposal, consider using something like TF-IDF
What you are doing right can be likened to just using TF. Using your global corpus, you can compute the idf of each unigram and bigram, which is basically a measure of unique or rare a word or phrase is across the entire corpus.
tf = the number of times you have seen an ngram within these results
idf = a global measure of how unique an ngram might be across all results (or atleast a very large number of them)
So when computing the score for a title, instead of simply adding up the tf’s of each ngram in it, you add up the tf*idf of each ngram instead. Rarer ngrams (which possibly do a better job at distinguishing this item from all other items) have a higher idf, so your algorithm should give higher weight to them. A lot of junk terms (like Mindestabnahme) would have really high idf, but they would have a really small tf, so they might not make a big difference. Alternatively prune off tokens you see fewer than k times, to get rid of noise.
Another NLP trick to know about is Levenshtein distance .. which is a way to quantify how similar two strings are. You can compute the levenshtein distance between every pair of strings within your results, and then try preferring the result which has the lowest average distance from all the other strings. This might not work well by itself… but factoring this score in with your existing approach might help you navigate some tricky cases.