I have a Java application that calls lots of different native methods of a legacy application through JNI. But JVM crashes with a stack dump at random places, outside any JNI call. Sometimes it crashes during GC, sometimes during class loading and other places. I suspect that one or more native methods is corrupting JVM heap or some other data structure. I need to know which call is this, so I can fix the native implementation.
The legacy application is a 3rd party DLL for which I don’t have sources nor symbol information. To make it callable from Java, I built a wrapper DLL that uses JNI calling conventions.
The perfect solution would be an extended JVM option that forces JVM to automatically check integrity of heap and its other data structures after each JNI call.
Do you know of something that can help?
P.S. Please don’t tell me to build a socket or pipe layer between JVM and the legacy application, because our requirements disallow that. This is about bug detection, not architecture design.
Because I went out of answers and couldn’t find a ready solution by myself, I ended up building a sandbox process in pure C++ just to identify the problem. My Java app instantiates the sandbox process using ProcessBuilder and then communicates with it using stdin and stdout. Instead of JVM, it’s the sandbox who actually loads and calls the legacy DLL. Then I monitored the sandbox process using Microsoft’s Application Verifier, which found a memory corruption problem – there was a call passing a buffer smaller than expected. After this was identified, I just increased the length of byte[] used as buffer in the Java app, and now JVM can make direct calls to DLL without use of sandbox.
Overall, I lost almost 10 days just because JVM doesn’t have an option to verify heap after each JNI call. But at least now if someone finds a crash we can quickly debug it using the sandbox.