I am trying to do some pattern ‘mining’ in piece of multi word on each line. I have done the N-gram analysis using the Text::Ngrams module in perl which give me the frequency of each word . I am however quite confused about the finding patterns in this text.
The tf-idf finds frequency also I presume but how does this differ from the Ngram analysis that I did and how does the similarity measure also help.
Please are there any perl modules or snippets of code I could get to understand some of this concepts .
Please I am from a physics background but have to do some pattern recognizing so I am a little new to some of these , a good reference on this topics will be appreciated.
Assuming you have a bunch of N documents and you:
Want to find out if Document X (containing an article on how to be a bodybuilder) is similar to another Document Y whose contents you do not know. If Document Y would be “similar” to Document X, it might contain the usual terms one associates with bodybuilding – eg: weight-lifting, barbells, dumbells and maybe Arnold.
So, the similarity of Document X, Document Y would be pretty high. One way to measure this similarity, is using the Cosine Angle between these two documents.
Cosine Similarity Reference: http://www.miislita.com/information-retrieval-tutorial/cosine-similarity-tutorial.html
Use CPAN to search for Perl modules. For eg, to compute cosine similarity you could try the Text::Document module