How are the term frequencies (TF), and inverse document frequency (IDF), affected by stop-word removal and stemming?
Thanks!
How are the term frequencies (TF), and inverse document frequency (IDF), affected by stop-word
Share
Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
How are the term frequencies (TF), and inverse document frequency (IDF), affected by stop-word removal and stemming?
Thanks!
tf is term frequency
idf is inverse document frequency which is obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient.
stemming effect is grouping all words which are derived from the same stem (ex: played, play,..), this grouping will increase the occurrence of this stem because frequencies are calculated using stem not words,
For example, if you have 2 documents:
the first one contains ‘play’ 2 times and ‘played’ 5 times,
and the second document contains ‘play’ 3 times and ‘played’ 1 time
if you do a search for ‘play’ without stemming the second document will be first because it has more occurrence of the word ‘play’, while if you do stemming, both words will be ‘play’ after stemming and the first document will be first cause it contains the stem play 7 times and the second document contains the stem play 4 times.
Concerning stopwords removal, it is found frequently in all document and isn’t consider as a keyword for any of them, it will have high freq without any scene.