Question

I have the following code in my application which does two things:

Parse the file which has ‘n’ number of data.

For each data in the file, there will be two web service calls.

 public static List<String> parseFile(String fileName) {
   List<String> idList = new ArrayList<String>();
   try {
     BufferedReader cfgFile = new BufferedReader(new FileReader(new File(fileName)));
     String line = null;
     cfgFile.readLine();
     while ((line = cfgFile.readLine()) != null) {
       if (!line.trim().equals("")) {
         String [] fields = line.split("\\|"); 
         idList.add(fields[0]);
       } 
     } 
     cfgFile.close();
   } catch (IOException e) {
     System.out.println(e+" Unexpected File IO Error.");
   }
 return idList;
}

When i try parse the file having 1 million lines of record, the java process fails after processing certain amount of data. I got java.lang.OutOfMemoryError: Java heap space error. I can partly figure out that the java process stops because of this huge data being provided. Kindly suggest me how to proceed with this huge data.

EDIT: Will this part of code new BufferedReader(new FileReader(new File(fileName))); parse the whole file and gets affected to the size of the file.

Answer 1

The problem you have is you are accumulating all the data on the list. The best way to approach this is to do it on a streaming fashion. This means do not accumulate all the ids on the list, but call your web service on each row or accumulate a smaller buffer and then do the call.

Opening the file and creating the BufferedReader will have no impact on memory consumption, as the bytes from the file will be read (more or less) line by line. The problem is at this point in the code idList.add(fields[0]);, the list will grow as large as the file as you keep accumulating all of the file data into it.

Your code should do something like this:

 while ((line = cfgFile.readLine()) != null) {
   if (!line.trim().equals("")) {
     String [] fields = line.split("\\|"); 
     callToRemoteWebService(fields[0]);
   } 
 }