Question

How would I read two log files which are one TB in size without running out of the memory on my machine.I would be doing some comparison on both of them .I would like to do this in Java.Would the below code work?My concern is that the FileStream would not be able to hold the data of the log file.

public static void main(String args[])
{
  try{
     // Open the file that is the first 
     // command line parameter
     FileInputStream fstream = new FileInputStream("textfile.txt");
     // Get the object of DataInputStream
     DataInputStream in = new DataInputStream(fstream);
     BufferedReader br = new BufferedReader(new InputStreamReader(in));
     String strLine;
     //Read File Line By Line
     while ((strLine = br.readLine()) != null) {
        // Print the content on the console
        System.out.println (strLine);
     }
     //Close the input stream
     in.close();
  }
  catch (Exception e){//Catch exception if any
     System.err.println("Error: " + e.getMessage());
  }
}

Can anyone guide me to the proper way of doing this.

Answer 1

Your code will probably work, because you’re only loading each line into memory. You’ll lose the output in the stdout buffer once it’s read more than a few hundred lines, though.

The best thing to do for comparison is to load a number of items into a collection, then throw away the ones you don’t need when you’re done with them. This will keep memory usage low. If you want to be clever about it, keep an eye on your process’s memory usage and start clearing out when it reaches a fixed threshold.