Question

Can anybody please explain to me how to calculate ‘avgLengthPath’ variable in the BM25 implementation for Lucene. What I understand is that I have to calculate it during the indexing. But still it was not clear how to do so.

the example provided :

IndexSearcher searcher = new IndexSearcher("IndexPath");

//Load average length
BM25Parameters.load(avgLengthPath);
BM25BooleanQuery query = new BM25BooleanQuery("This is my Query", 
    "Search-Field",
    new StandardAnalyzer());

TopDocs top = searcher.search(query, null, 10);
ScoreDoc[] docs = top.scoreDocs;

//Print results
for (int i = 0; i $<$ top.scoreDocs.length; i++) {
      System.out.println(docs[i].doc + ":"+docs[i].score);
}

suggest that there are a method or class to load average length from.

Would appreciate any help…

Thanks

Answer 1

I have solved the problem and I would like to share me answer to get any corrections or comments..

The problem was how to calculate the avgLengthPath arguments. When I looked at the method that takes this argument:load() it can be seen that it require a String which is the path to a file which contain the average length. So avgLengthPath would be something like:

/Users/admib/Study/avgLength

load() method is as follow:

    public static void load(String path) throws NumberFormatException,
        IOException {
    BufferedReader in = new BufferedReader(new FileReader(path));
    String line;
    while (null != (line = in.readLine())) {
        String field = line;
        Float avg = new Float(in.readLine());
        BM25Parameters.setAverageLength(field, avg);
    }
    in.close();
}

Now, lest see how create such file. We can see that the above method read the file line by line and send each two lines to another method called BM25Parameters.setAverageLength(). the formate of the avgLengthPath file should be something like this:

CONTENT 
459.2903f
ANCHOR
84.55523f

Where the first line is the filed name and the second line is the average length for this field.
Also, the third line is another filed and the forth line is the average length for that filed.

The problem bout such file is that we cannot get the documents length from Lucene in its default sitting. To overcome this, I re-indexed my collection and added the document length as one of the fields to be indexed by Lucene.

First I created a method that takes a file and return the document length as a string. I call it getDocLength(File f):

    public static String getDocLength(File f) throws IOException {
    FileInputStream stream = new FileInputStream(f);
    try {
        FileChannel fc = stream.getChannel();
        MappedByteBuffer bb = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());

        String doc = Charset.defaultCharset().decode(bb).toString();
        int length  = doc.length();
        return Integer.toString(length);
    } finally {
        stream.close();
    }
}

This method is called during indexing process to add the document length field as follow:

 protected Document getDocument(File f) throws Exception {
    Document doc = new Document();
    String docLength = Integer.toString(io.getDocLength(f));
    doc.add(new Field("contents", new FileReader(f), Field.TermVector.YES));
    doc.add(new Field("docLength", i, Field.Store.YES, Field.Index.NOT_ANALYZED));
    doc.add(new Field("filename", f.getName(), Field.Store.YES, Field.Index.NOT_ANALYZED));
    doc.add(new Field("fullpath", f.getCanonicalPath(), Field.Store.YES, Field.Index.NOT_ANALYZED));         
    return doc;
}

Finally I created a method that loop through all docs in the index and calculate the average document length and finally save the result into the avgLengthPath file with the correct formate. I called this method generateAvgLengthPathFile():

    public static void generateAvgLengthPathFile(String luceneIndexPath, String outputFilePath) {
    try {
        Directory dir = FSDirectory.open(new File(luceneIndexPath));
        IndexReader reader = IndexReader.open(dir);
        int totalLength = 0;
        //here we loop through all the docs in the index 
        for (int i = 0; i < reader.maxDoc(); i++) {
            if (reader.isDeleted(i)) {
                continue;
            }
            Document doc = reader.document(i);
            totalLength += Integer.parseInt(doc.get("docLength"));
        }
        //calculate the avarage length
        float avarageLength = totalLength * 1.0f / reader.maxDoc() * 1.0f;
        //create the a String varibale with the correct formate
        String avgLengthPathFile = "contents" + "\n" + avarageLength;

        //finally, save the file 
        Writer output = null;
        String text = "contents" + "\n" + avarageLength;
        File file = new File(outputFilePath);
        output = new BufferedWriter(new FileWriter(file));
        output.write(text);
        output.close();

    } catch (Exception e) {
System.err.println(e);
    }
}