Home/ Questions/Q 7676567
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-31T17:13:43+00:00

I have a single XML file that I want to index using Lucene.NET. The

  • 0

I have a single XML file that I want to index using Lucene.NET. The file is basically a large collection of logs.
Since the single file itself is beyond 5GB and I am developing code on a system with 2GB RAM, how can I perform the indexing when I am not parsing the file nor am I creating any other fields other than “text” which shall contain the file data?

I am using some code from CodeClimber and at the moment not sure what would be the best approach to index such a large single file.

Is there a way to pass on file data to the index in chunks? Below is the line of code that basically creates the text field and the associated file data

Document doc = new Document();
doc.Add(new Field("Body", text, Field.Store.YES, Field.Index.TOKENIZED));
writer.AddDocument(doc);

Thank you for the guidance

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You should use something like System.Xml.XmlReader that doesn’t load the whole xml into the memory. But indexing the whole xml as a single document doesn’t make sense since you will get either 1 or 0 document with each search.(found or not found). So to be able to pass data in chunks wouldn’t help you much. Therefore while reading your xml file you should split it into many documents(and fields) so that you can get some reasonable results when you search.

    how can I perform the indexing when I am not parsing the file nor am I creating any other fields other than “text” which shall contain the file data

    what a wonderful world it would be

    • 0