Question

I am trying to read a text file and adding each line to a list. The list is used in prepared statement to insert the records to db. The problem is when there is 1000000 records the execution becomes too slow and some time gets Out of Memory error.

My code

while ((sRec = in.readLine()) != null) {
        myCollection.add(reversePad(sRec, hmSizeLookup, colLength, tableName));
        a = Utility.getPositions(sRec, hmSizeLookup, colLength);
        i = i + 1;
}
for (int j = 0; j < a.length; j++) {
    Field f = (Field) hmSizeLookup.get(String.valueOf(j));
    sAllFields += f.getColumnName();
    if (j < a.length) {
        sValues += "?";
    }
    if (j < a.length - 1) {
       sValues += ",";
       sAllFields += ",";
    }
}
sQ = "insert  into " + tableName + "(" + sAllFields + ") values(" + sValues + ")";
ds1.executePreparedStatement(sQ, myCollection);

How can read only 1000 lines at a time from the text file and need to perform the insert operation ? This should continue until the end of text file.

Any suggestions are appreciated,

Thanks

Answer 1

You need to do the inserting in the file loop.

Chop the insert code out to a function, every 1000 lines call it and then myCollection.clear()

After the loop you’ll need to call it again for any data after the last insert.

So something like:

int line = 0;
while ((sRec = in.readLine()) != null) {
        myCollection.add(reversePad(sRec, hmSizeLookup, colLength, tableName));
        a = Utility.getPositions(sRec, hmSizeLookup, colLength);
        i = i + 1;
    if (++line > 1000) {
        doTheInsert(myCollection);
        myCollection.clear();
        line = 0;
    }
}
doTheInsert(myCollection);