I am started working on resume retrieval(document) component based on lucene.net engine. It works great, and it fetches the document and score it based on the
the idea behind the VSM is the more
times a query term appears in a
document relative to the number of
times the term appears in all the
documents in the collection, the more
relevant that document is to the
query.
Lucene’s Practical Scoring Function is derived from the below.
score(q,d)=coord(q,d)·queryNorm(q)· ∑( tf(t in d) ·idf(t)2 · t.getBoost() · norm(t,d) )
t in q
in this
- tf(t in d) correlates to the term’s frequency, defined as the number of times term t appears in the currently scored document d. Documents that have more occurrences of a given term receive a higher score
- idf(t) stands for Inverse Document Frequency. This value correlates to the inverse of docFreq (the number of documents in which the term t appears). This means rarer terms give higher contribution to the total score.
This is very great indeed in most of the situation, but due to the fieldnorm calculation the result is not accurate
fieldnorm aka “field length norm” value represents the length of that field in that doc (so shorter fields are automatically boosted up).
Due to this we didn’t get the accurate results.
Say for an example i got 10000 documents in which 3000 documents got java and oracle keyword. And the no of times it appears vary on each document.
- assume doc A got 10 java 20 oracle among 1000 words and doc B got 2 java 2 oracle among 50 words
- if am searching for a query “java and oracle”, lucene returns doc B with high score
due to the length normalization.
Due to the nature of the business we need to retrieve the documents got more search keyword occurrence should come first, we don’t really care about the length of the document.
Because of this a Guy with a big resume with lot of keywords is been moved below in the result and some small resumes came up.
To avoid that i need to disable length normalization. Can some one help me with this??
I have attached the Luke result image for your reference.
In this image, document with java 50 times and oracle 6 times moved down to 11 th position.
But this document with java 24 times and oracle 5 times is a top scorer due to the fieldnorm.
Hope i conveyed the info clear… If not please ask me, i ll give more info
You can disable length normalization with Field.setOmitNorms(true)