Question

I have a python script to simply index unicode sentences into a lucene index. And it works fine on the 100 sentences and my 1000 sentences trial. However when i needed to index 200,000 sentences i get a merge error at the 4514th sentence, what is the problem and how can it be solved?

The Error:

Exception in thread "Thread-4543" org.apache.lucene.index.MergePolicy$MergeException: java.io.FileNotFoundException: /home/alvas/europarl/index/_70g.tii (Too many open files)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:271)
Traceback (most recent call last):
Caused by: java.io.FileNotFoundException: /home/alvas/europarl/index/_70g.tii (Too many open files)
    at java.io.RandomAccessFile.open(Native Method)  File "indexer.py", line 183, in <module>

    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
    at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:593)
    at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:435)
    at org.apache.lucene.index.TermInfosWriter.initialize(TermInfosWriter.java:91)
    at org.apache.lucene.index.TermInfosWriter.<init>(TermInfosWriter.java:83)
        at org.apache.lucene.index.TermInfosWriter.<init>(TermInfosWriter.java:77)
incrementalIndexing(sfile,tfile,indexDir)
    at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:381)
    at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:134)  File "indexer.py", line 141, in incrementalIndexing

    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3109)
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)
    writer.optimize(); writer.close()
lucene.JavaError: java.io.IOException: background merge hit exception: _70e:c4513 _70f:c1 into _70g [optimize]
    Java stacktrace:
java.io.IOException: background merge hit exception: _70e:c4513 _70f:c1 into _70g [optimize]
    at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:1749)
    at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:1689)
    at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:1669)
Caused by: java.io.FileNotFoundException: /home/alvas/europarl/index/_70g.tii (Too many open files)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
    at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:593)
    at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:435)
    at org.apache.lucene.index.TermInfosWriter.initialize(TermInfosWriter.java:91)
    at org.apache.lucene.index.TermInfosWriter.<init>(TermInfosWriter.java:83)
    at org.apache.lucene.index.TermInfosWriter.<init>(TermInfosWriter.java:77)
    at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:381)
    at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:134)
    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3109)
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)

My code: http://pastebin.com/Ep133W5f

The sample input files:
http://pastebin.com/r5qE4qpt , http://pastebin.com/wxCU277x

Answer 1

You are reassigning filedir at line 169, without closing the one you made on line 116. I think that’s a mistake, because you don’t need to be creating that new one, you can just reuse the old one. Plus, then you’d be creating it every loop or so and it would just leak handles, since they’re never getting closed.

If a few other methods, you’d creating a new filedir without closing it, line 106 in createEmptyIndex and line 97 in deleteFromIndex.

There is another one, that is less obvious, in retrieveUniqID. You’re creating searcher, but only closing it when the condition if cont == content: on line 87 is true. In the case where there is no match, and you return None on line 91, you’re never closing that searcher. In this case, since you’re passing a string to the IndexSearcher constructor, it’s internally creating a Directory that you’re not closing in certain cases. You can use a try/finally block to make sure that you always close it, if you’d like.

All of the above calls are made from a main loop in incrementalIndexing, so the number of leaked handles increases quickly.

Also, something to consider: all of those classes, the IndexSearcher, IndexReader, IndexWriter and Directory are thread-safe, and it’s expensive to create new ones each time. You might be better off with a small redesign to minimize the number of time you’ll need to open and close them. It might actually simply your code, since you could pass around the already created instances which would clean up a lot of the initializing clutter in the different methods.

Since it seems you also want to be able to access the documents you’ve indexed immediately, I would look into either getting your IndexReader (and thus your IndexSearcher as well) via the IndexWriter.GetReader() method, or by refreshing the reader, like so: reader = reader.Refresh().