Question

First, I had a problem with getting the data from the Database, it took too much memory and failed. I’ve set -Xmx1500M and I’m using scrolling ResultSet so that was taken care of. Now I need to make an XML from the data, but I can’t put it in one file. At the moment, I’m doing it like this:

while(rs.next()){
                i++;
                xmlStringBuilder.append("\n\t<row>");
                xmlStringBuilder.append("\n\t\t<ID>" + Util.transformToHTML(rs.getInt("id")) + "</ID>");
                xmlStringBuilder.append("\n\t\t<JED_ID>" + Util.transformToHTML(rs.getInt("jed_id")) + "</JED_ID>");
                xmlStringBuilder.append("\n\t\t<IME_PJ>" + Util.transformToHTML(rs.getString("ime_pj")) + "</IME_PJ>");
//etc.
                xmlStringBuilder.append("\n\t</row>");
                if (i%100000 == 0){
                                    //stores the data to a file with the name i.xml
                    storeKBR(xmlStringBuilder.toString(),i);
                    xmlStringBuilder= null;
                    xmlStringBuilder= new StringBuilder();  
                }

and it works; I get 12 100 MB files. Now, what I’d like to do is to do is have all that data in one file (which I then compress) but if just remove the if part, I go out of memory. I thought about trying to write to a file, closing it, then opening, but that wouldn’t get me much since I’d have to load the file to memory when I open it.

Answer 1

Why not write all data to one file and open the file with the “append” option? There is no need to read in all the data in the file if you are just going to write to it.

However, this might be a better solution:

PrintWriter writer = new PrintWriter(new BufferedOutputStream(new FileOutputStream("data.xml")));

while(rs.next()){
    i++;
    writer.print("\n\t<row>");
    writer.print("\n\t\t<ID>" + Util.transformToHTML(rs.getInt("id")) + "</ID>");
    writer.print("\n\t\t<JED_ID>" + Util.transformToHTML(rs.getInt("jed_id")) + "</JED_ID>");
    writer.print("\n\t\t<IME_PJ>" + Util.transformToHTML(rs.getString("ime_pj")) + "</IME_PJ>");
    //...

    writer.print("\n\t</row>");
}

writer.close();

The BufferedOutputStream will buffer the data before printing it, and you can specify the buffer size in the constructor if the default value does not suit your needs. See the java API for details: http://java.sun.com/javase/6/docs/api/.