Question

I am creating two CSV files using String buffers and byte arrays.
I use ZipOutputStream to generate the zip files. Each csv file will have 20K records with 14 columns. Actually the records are fetched from DB and stored in ArrayList. I have to iterate the list and build StringBuffer and convert the StringBuffer to byte Array to wirte it to the zip entry.

I want to know the memory required by JVM to do the entire process starting from storing the records in the ArrayList.
I have provide the code snippet below.

StringBuffer responseBuffer = new StringBuffer();
    String response = new String();
    response = "Hello, sdksad, sfksdfjk, World, Date, ask, askdl, sdkldfkl, skldkl, sdfklklgf, sdlksldklk, dfkjsk, dsfjksj, dsjfkj, sdfjkdsfj\n";
    for(int i=0;i<20000;i++){
        responseBuffer.append(response);
    }
    response = responseBuffer.toString();
    byte[] responseArray = response.getBytes();
    res.setContentType("application/zip");
    ZipOutputStream zout = new ZipOutputStream(res.getOutputStream());
    ZipEntry parentEntry = new ZipEntry("parent.csv");
    zout.putNextEntry(parentEntry);
    zout.write(responseArray);
    zout.closeEntry();
    ZipEntry childEntry = new ZipEntry("child.csv");
    zout.putNextEntry(childEntry);
    zout.write(responseArray);
    zout.closeEntry();
    zout.close();

Please help me with this. Thanks in advance.

Answer 1

I’m guessing you’ve already tried counting how many bytes will be allocated to the StringBuffer and the byte array. But the problem is you can’t really know how much memory your app will use unless you have upper bounds on the sizes of the CSV records. I’m If you want your software to be stable, robust and scalable, I’m afraid you’re asking the wrong question: you should strive on performing the task you need to do using a fixed amount of memory, which in your case seems easily possible.

The key is, that in your case the processing is entirely FIFO – you read records from the database, and then write them (in the same order) into a FIFO stream (OutputStream in that case). Even zip compression is stream-based, and uses a fixed amount of memory internally, so you’re totally safe there.

Instead of buffering the entire input in a huge String, then converting it to a huge byte array, then writing it to the output stream – you should read each response element separately from the database (or chunks of fixed size, say 100 records at a time), and write it to the output stream. Something like

res.setContentType("application/zip");
ZipOutputStream zout = new ZipOutputStream(res.getOutputStream());
ZipEntry parentEntry = new ZipEntry("parent.csv");
zout.putNextEntry(parentEntry);
while (... fetch entries ...)
    zout.write(...data...)
zout.closeEntry();

The advantage of this approach is that because it works with small chunks you can easily estimate their sizes, and allocate enough memory for your JVM so it never crashes. And you know it will still work if your CSV files become much more than 20K lines in the future.