Question

I am writing a StringOutputStream class in Java because I need the pre-Base64-encoded data from an OutputStream to go to a String. I need this as a String because I am going to be putting it into a W3C XML Document.

Everything would be fine, but I’m dealing with (relatively) large objects. The resulting object turns out to be about 25 MB (before String representation). I am running this as an Applet, so I have 66 MB of heap space which gets exhausted quite quickly.

I have tried a few methods so far:

1. Append the received byte to a String object (using strObj.concat((byte) b) and strObj += new String((byte) b)) with and without buffering
2. Add the received byte to a StringBuffer
3. Add the byte to a byte array, then when the string is wanted, convert that byte array to a String

Number one works until about 11 MB, when the old String and the new String use up too much space when concat-ing.

Number two was a total failure, it only gets to about 7 MB.

Number three was (perhaps?) the best, it stores the whole stream, but when trying to get the String it, unsurprisingly, fails.

How would I make this work? Is it possible?

I think I have the space available to hold the resulting String, but it’s the copying that is the problem (since you need source and destination for a traditional copy). I know Strings are immutable, but is there any way to append some characters onto the end?

Here’s my three examples:

package com.myorg.SigningServer.Util.Security;

import java.io.IOException;
import java.io.OutputStream;
import java.util.Arrays;

import com.technicolor.SigningServer.Applet.SigningApplet;

public class StringOutputStream extends OutputStream {

byte[] array = new byte[1024*1024*22];
StringBuffer sb = new StringBuffer();
String output = "";
int prevByte = -1;
long numBytes = 0;

int bufferPos = 0;
int bufferSize = 512*1024;
byte[] buffer = new byte[bufferSize];

public void write2(int b) throws IOException {
    sb.append((byte) b);
}


public void write3(int b) throws IOException {
    array[(int) numBytes] = (byte) b;
    numBytes++;
}

public void write1(int b) throws IOException {
    numBytes++;
    bufferPos++;
    buffer[bufferPos] = (byte) b;
    if(bufferPos == bufferSize-1)
    {
        bufferPos = 0;
        System.gc();
        System.out.println("Generating string "+numBytes+"; String length "+output.length());
        output = output.concat(new String(buffer));
        System.gc();
    }
}

public void flush1() {
    output = output.concat(new String(Arrays.copyOf(buffer, bufferPos)));
    bufferPos = 0;
    System.gc();
}

public String toString2()
{
    return sb.toString();
}

public String toString3()
{
    return new String(array);
}

public String toString1()
{
    return output;
}
}

A few notes about the code: obviously, you rename the methods you want to use to write() and toString(). Also, the byte array is (currently) statically allocated, but that would be changed if I go that route (and is not used during the other methods).

Edit 1:
More information on my overall problem:

This is part of a larger application that takes data, signs it, and uploads it to a server. I have to read in a file, take the SHA-1 hash of it, encrypt it, and then construct an XML document (with a few other things in it, such as the time). Then that XML document must be signed (via XML DSig, aka javax.xml.crypto.dsig.XMLSignatureFactory) and uploaded back to the server.

The files to be signed are anywhere from 1KB to about 50 MB.

There are a few problems:

1. The current Java implementation of XML DSig does not parse and XML streams, just w3c Nodes. (I also cannot find any other implementations that do)
2. My boss wants this to not require minimal client-side installation, so that’s why an Applet was chosen (it is a signed applet, so it can access anything on the client).

Answer 1

Thanks to mdma, I realized that I would truly need a way to stream this, rather than storing in in memory.

Here’s what I did: the applet now encrypts the data as before, but signs it using PKCS7 (using BouncyCastle’s CMSSignedDataStreamGenerator). The data is streamed across the network without storing it on the client’s computer.

The detached PKCS7 signature generated by the PKCS7 signing is then put inside the XML. That XML is then signed with XML DSig and separately uploaded to the server.