Question

Hello guys I have a problem with my application involving Lucene java library and I dont know what is exactly the error

here are sample of the error console

ERROR/AndroidRuntime(25909): java.lang.OutOfMemoryError: (Heap Size=32775KB, Allocated=30112KB, Bitmap Size=0KB)
ERROR/AndroidRuntime(25909): at org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.(FreqProxTermsWriterPerField.java:193)
ERROR/AndroidRuntime(25909): at org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.newInstance(FreqProxTermsWriterPerField.java:204)
ERROR/AndroidRuntime(25909): at org.apache.lucene.index.ParallelPostingsArray.grow(ParallelPostingsArray.java:48)
ERROR/AndroidRuntime(25909): at org.apache.lucene.index.TermsHashPerField.growParallelPostingsArray(TermsHashPerField.java:137)
ERROR/AndroidRuntime(25909): at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:440)
ERROR/AndroidRuntime(25909): at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:172)
ERROR/AndroidRuntime(25909): at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:278)
ERROR/AndroidRuntime(25909): at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:766)
ERROR/AndroidRuntime(25909): at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2067)
ERROR/AndroidRuntime(25909): at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2041)

and it says that the field of the indexwriter and i dont know what it is

so can you guys help me ? Ill appreciate for your reply

and here are my code

public class CalculateWeightPage {

protected static Crawlers crawlers;
protected static StopWordsAndStemmer stemmer;
protected static CountWords countWords;
protected static StringSplitter splitter;
protected static ShortingStringArray shortingStringArray;

public static String[][] calulateRelevancePage(String[][] wkt,String urlPage) {

    // 1.1.Defining parameters
    int p = 0;
    int count = 0;
    int count2 = 0;
    String title = "";
    String body = "";
    int titleFreq = 0;
    int bodyFreq = 0;
    String[][] wkp = null ;
    int newTf = 0;
    int y = 0;
    int counter = 0;
    try {



        // 1.2.Extracting the text body and title from webPage
        Map bodyTitle = crawlers.extractBodyAndTitle(urlPage);

        if(bodyTitle.containsKey("title")){

            title = stemmer.removeStopWordsAndStem(((String) bodyTitle.get("title")).toLowerCase());
            body = stemmer.removeStopWordsAndStem(((String) bodyTitle.get("body")).toLowerCase());

            // 1.4.Making a list containing unique words from text title and body
            List bodyTitleUnique = splitter.StringUnique(body);

            int sizeList = bodyTitleUnique.size();
            wkp =  new String[sizeList][2];

            // 1.5.Calculating each tf 
            for (int r = 0; r < sizeList; r++) {
                titleFreq = 0;
                bodyFreq = 0;
                // 1.5.1.Calculating tf in title
                titleFreq = countWords.calculate(title, bodyTitleUnique.get(r).toString());

                // 1.5.2.Calculating tf in body
                bodyFreq = countWords.calculate(body, bodyTitleUnique.get(r).toString());

                if (!(titleFreq == 0)) {
                    newTf = (titleFreq * 2) + (bodyFreq - titleFreq);
                } else {
                    newTf = titleFreq + bodyFreq;
                }

                // 1.6.Inserting the result into string array
                if(!(newTf == 0)){
                    wkp[r][0] = bodyTitleUnique.get(r).toString();
                    wkp[r][1] = String.valueOf(newTf);
                }
            }

        }else{
            return wkp;
        }

    } catch (Exception e) {
        // TODO: handle exception
    }
    return wkp;

}

}

and this is for the second code

public class CountWords {
CountWords() {

}

protected static StopWordsAndStemmer stemmer;

public static int calculate(String txt, String keyword) {

    StopAnalyzer analyzer = new StopAnalyzer(Version.LUCENE_CURRENT);
    RAMDirectory idx = new RAMDirectory();
    int counts = 0;
    int count = 0;
    try {
        IndexWriter writer = new IndexWriter(idx, analyzer, true,
                IndexWriter.MaxFieldLength.UNLIMITED);

        Document doc = new Document();

        //String text1 = stemmer.removeStopWordsAndStem(txt.toLowerCase());

        writer.addDocument(createDocument("", txt));

        writer.optimize();
        writer.close();

        Searcher searcher = new IndexSearcher(idx);

        IndexReader ir = IndexReader.open(idx);
        TermDocs termDocs = ir.termDocs(new Term("content", keyword.toLowerCase()));

        while (termDocs.next()) {
            count = count + termDocs.freq();
        }
        //counts = count(count);

        searcher.close();

    } catch (IOException ioe) {
        ioe.printStackTrace();
    }
    return count;

}

private static Document createDocument(String title, String content) {
    Document doc = new Document();
    doc.add(new Field("content", new StringReader(content)));
    return doc;
}

private static int search(Searcher searcher, String queryString)throws ParseException, IOException {

    StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
    QueryParser parser = new QueryParser(Version.LUCENE_CURRENT, "content",analyzer);
    Query query = parser.parse(queryString);

    TopScoreDocCollector collector = TopScoreDocCollector.create(10, true);
    searcher.search(query, collector);

    return collector.getTotalHits();
}

public static Integer count(int count) {
    if (count == 0) {
        count = 1;
    } else {
        count = count;
    }
    return count;
}

}

Answer 1

It is likely that you get this error because all the data you need cannot fit in memory.

However, your solution looks a little over-engineered: you don’t need Lucene to compute term frequencies in memory (method calculate of CountWords), you just need to analyze your input and store frequencies in a HashMap<String, Integer> map.

Moreoever, although your code might work, there are some things that look incorrect in your code:

• you should call commit before optimize so that segments are created before being optimized (although the optimization is certain to be pointless because you will only have a single segment),
• you are opening one index searcher and one index reader on the same directory, but opening an index searcher opens an index reader under the hood, so two equivalent readers will actually be opened although only one is required,
• you close your index searcher, but not the reader.