I’m doing IPC between Java and some C code by just creating an external C process that takes some commandline arguments and writes its results to standard out. This is inside a server application, so the data returned needs to be fed directly back into the OutputStream of an HttpServletResponse, which I’m currently doing with a simple Apache commons IOUtils.copy(inputStream, outputStream). The current solution is convenient in that the external process can generate arbitrarily large responses without consuming arbitrary amounts of memory. Unfortunately, the cost of spawning the external process, as well as all the setup the actual process logic requires, is making the overhead unacceptable for small jobs. So I’d like to switch to JNI, which will get rid of the external process overhead and allow much of the setup of these tasks to be performed statically once at server startup time.
But I’m now wondering how to preserve the current streaming nature of the responses- I’d rather not simply make the JNI call return a byte array or even a ByteBuffer, because I’d like to keep memory usage capped and low. So, what techniques are available to call C logic and return essentially arbitrarily large byte arrays in a way that will let me get them into an OutputStream without consuming arbitrarily large amounts of memory, and without much overhead for setting up the C environment.
FWIW, this C code is basically running a passed in Lua script on a pile of records, and returning the output for each record.
The main one coming to mind is to just pass the OutputStream into the native method, and call write on it intermittently using JNI reflection as the C code generates its response. Are there better alternatives I’m missing? Files? Zeromq?
You could have your JNI method return an InputStream with a read callback into JNI, but I think passing the OutputStream into the JNI as you suggest is a cleaner solution.