Question

I’m currently retrieving image data from an iSight camera and I’d like to hand it over to Java for processing. I originally tried to put the data in a jbyteArray and return the jbyteArray. This works once per process. Calling the native function a second time will result in an invalid memory access.

Since I’m working with objective-c and Cocoa, I must use the JNF_COCOA_ENTER(…) and JNF_COCOA_EXIT(…) functions. Sadly, if I can’t return the jbyteArray because doing so would result in JNF_COCOA_EXIT(…) not getting called. It was suggested to use a direct ByteBuffer to pass the data from JNI land to java land. Unfortunately, all the resources and references I’ve been using don’t outline this straightforwardly enough for my brain to comprehend. I deeply apologize if this is a “duh” moment or has already been asked (I’ve searched with no luck), but…

1) What is the most efficient way to bring this image data to Java?

2) How should I use the ByteBuffer class to accomplish this? (if relevant)

Thanks!

Answer 1

Code might be helpful here. It’s not as difficult as you might imagine once you get an idea of what those macros do, and how to separate the java and cocoa concerns from each other.

What you need to remember is that JNF_COCOA_ENTER and JNF_COCOA_EXIT do two main things for you:

They establish a local scope (so variables defined inside are not available outside – this is to help you not do “dumb” things with variables you shouldn’t be touching) and also to set up an auto-release pool, so cocoa objects which are “autoreleased” inside that scope will be gone when the scope disappears. (this is part of why the local scope is useful/helpful) They’ll also do some exception parsing so that you can catch Cocoa exceptions in Java.

That said, the following code IS LEGAL, you just need to be quite careful about managing memory access and ownership of data. Avoid mixing Java ownership and Objective-C ownership if possible, or else have your object manage the ownership, and clean up when the java object is GCed.

jbyteArray bytes;

JNF_COCOA_ENTER(env);   

// Assign and full the bytes array here, doing any requisite transformations.
// Remember to COPY any data out of COCOA objects, as references will be dead soon!

JNF_COCOA_EXIT(env);

return bytes;

Using Java Objects from C is complex, but not unworkable. Still, the number of method calls is non-trivial, and the jump back and fourth is time consuming, so if you’ll be calling this method often, or it’s time critical, stick with the basic types as much as possible.

If you need to push data from Cocoa to Java from a delegate, things are a little bit more complicated, but not unworkable. Here’s a segment from a project I manage called QTCubed, which does exactly that. didOutputVideoFrame is the delegate method, and this object MUST be initialized with the target Java Object Reference, and Java Environment from a JNI called method. Once initialized, it’s then set as a delegate object, and receives updates from the camera.

@implementation QTKitCaptureDecompressedVideoOutput

- (QTKitCaptureDecompressedVideoOutput *)initWithEnv:(JNIEnv *) env javaObject:(jobject) javaObjectRef {
    [super init];
    // Save a reference to the VM
    (*env)->GetJavaVM(env,&g_vm);
    // Create a global reference to this object so we can access it later
    objectRef = (*env)->NewGlobalRef(env,javaObjectRef);

    return self;
}

- (void)captureOutput:(QTCaptureOutput *)captureOutput didOutputVideoFrame:(CVImageBufferRef)videoFrame withSampleBuffer:(QTSampleBuffer *)sampleBuffer fromConnection:(QTCaptureConnection *)connection {
    // Move into Java to deliver the data
    JNIEnv *env;
    (*g_vm)->AttachCurrentThread (g_vm, (void **) &env, NULL);

    void * rawData = [sampleBuffer bytesForAllSamples];
    int length = [sampleBuffer lengthForAllSamples];
    QTFormatDescription * formatDescription = [sampleBuffer formatDescription];
    QTTime duration = [sampleBuffer duration];

    float frameDuration = duration.timeValue/duration.timeScale;
    float fps = 1/frameDuration;

    jint format = [formatDescription formatType];
    NSValue * pixelSize = [formatDescription attributeForKey:QTFormatDescriptionVideoEncodedPixelsSizeAttribute];
    NSSize size = [pixelSize sizeValue];
    jint width = size.width;
    jint height = size.height;
    //NSLog(@"Outputting frame sized %d x %d of length %d with format: %#x",width,height,length,format);

    switch (format) {
            // 8 bit codecs
        case kCVPixelFormatType_1Monochrome:
        case kCVPixelFormatType_2Indexed:
        case kCVPixelFormatType_4Indexed:
        case kCVPixelFormatType_8Indexed:
        case kCVPixelFormatType_1IndexedGray_WhiteIsZero:
        case kCVPixelFormatType_2IndexedGray_WhiteIsZero:
        case kCVPixelFormatType_4IndexedGray_WhiteIsZero:
        case kCVPixelFormatType_8IndexedGray_WhiteIsZero:
        case kCVPixelFormatType_422YpCbCr8:
        case kCVPixelFormatType_4444YpCbCrA8:
        case kCVPixelFormatType_4444YpCbCrA8R:
        case kCVPixelFormatType_444YpCbCr8:
        case kCVPixelFormatType_420YpCbCr8Planar:
        case kCVPixelFormatType_422YpCbCr_4A_8BiPlanar:
        case kCVPixelFormatType_24RGB:
        case kCVPixelFormatType_24BGR:
        default:
        {
            // Re-use the existing array if possible
            if (byteFrameData == nil || (*env)->GetArrayLength(env,byteFrameData) < length) {
                // Clean up the previously allocated global reference
                if (byteFrameData != nil) {
                    (*env)->DeleteGlobalRef(env,byteFrameData);
                    byteFrameData = nil;
                }
                // Create an appropriately sized byte array to hold the data
                byteFrameData = (*env)->NewGlobalRef(env,(*env)->NewByteArray(env,length));
            }
            if (byteFrameData) {
                // Copy the raw data into the byteArray
                (*env)->SetByteArrayRegion(env,byteFrameData,0,length,rawData);

                // Get the class reference for our object
                jclass classRef = (*env)->GetObjectClass(env,objectRef);
                // Get the pushFrame methodId
                jmethodID methodId = (*env)->GetMethodID(env,classRef,"pushFrame","([BIIIF)V");
                // Call pushFrame with the byte array
                (*env)->CallVoidMethod(env,objectRef,methodId,byteFrameData,format,width,height,fps);
            }
            break;
        }   
            // 16 bit (short) storage of values
        case kCVPixelFormatType_16BE555:
        case kCVPixelFormatType_16LE555:
        case kCVPixelFormatType_16LE5551:
        case kCVPixelFormatType_16BE565:
        case kCVPixelFormatType_16LE565:
        case kCVPixelFormatType_16Gray:
        case kCVPixelFormatType_422YpCbCr16:
        case kCVPixelFormatType_422YpCbCr10:
        case kCVPixelFormatType_444YpCbCr10:
        {
            // Re-use the existing array if possible
            if (shortFrameData == nil || (*env)->GetArrayLength(env,shortFrameData) < length/2) {
                // Clean up the previously allocated global reference
                if (shortFrameData != nil) {
                    (*env)->DeleteGlobalRef(env,shortFrameData);
                    shortFrameData = nil;
                }
                // Create an appropriately sized byte array to hold the data
                shortFrameData = (*env)->NewGlobalRef(env,(*env)->NewShortArray(env,length/2));
            }
            if (shortFrameData) {
                // Copy the raw data into the byteArray
                (*env)->SetShortArrayRegion(env,shortFrameData,0,length/2,rawData);

                // Get the class reference for our object
                jclass classRef = (*env)->GetObjectClass(env,objectRef);
                // Get the pushFrame methodId
                jmethodID methodId = (*env)->GetMethodID(env,classRef,"pushFrame","([SIIIF)V");
                // Call pushFrame with the short array
                (*env)->CallVoidMethod(env,objectRef,methodId,shortFrameData,format,width,height,fps);          
            }
            break;
        }   
            // 32 bit (int) storage of values
        case kCVPixelFormatType_32ABGR:
        case kCVPixelFormatType_32AlphaGray:
        case kCVPixelFormatType_32ARGB:
        case kCVPixelFormatType_32BGRA:
        case kCVPixelFormatType_32RGBA:
        {
            // Re-use the existing array if possible
            if (intFrameData == nil || (*env)->GetArrayLength(env,intFrameData) < length/4) {
                // Clean up the previously allocated global reference
                if (intFrameData != nil) {
                    (*env)->DeleteGlobalRef(env,intFrameData);
                    intFrameData = nil;
                }
                // Create an appropriately sized byte array to hold the data
                intFrameData = (*env)->NewGlobalRef(env,(*env)->NewIntArray(env,length/4));
            }
            if (intFrameData) {
                // Copy the raw data into the byteArray
                (*env)->SetByteArrayRegion(env,intFrameData,0,length/4,rawData);

                // Get the class reference for our object
                jclass classRef = (*env)->GetObjectClass(env,objectRef);
                // Get the pushFrame methodId
                jmethodID methodId = (*env)->GetMethodID(env,classRef,"pushFrame","([IIIIF)V");
                // Call pushFrame with the int array
                (*env)->CallVoidMethod(env,objectRef,methodId,intFrameData,format,width,height,fps);
            }
            break;
        }
    }

    // Detatch from Java
    (*g_vm)->DetachCurrentThread (g_vm);
}

/* Clean up java references so they can be properly GCed in java */
- (void) dealloc {

    // Attach to java so we can release references
    JNIEnv *env;
    (*g_vm)->AttachCurrentThread (g_vm, (void **) &env, NULL);

    // release the references we hold

    if (objectRef != nil) {
        (*env)->DeleteGlobalRef(env,objectRef);
        objectRef = nil;        
    }
    if (byteFrameData != nil) {
        (*env)->DeleteGlobalRef(env,byteFrameData);
        byteFrameData = nil;        
    }
    if (shortFrameData != nil) {
        (*env)->DeleteGlobalRef(env,shortFrameData);
        shortFrameData = nil;       
    }
    if (intFrameData != nil) {
        (*env)->DeleteGlobalRef(env,intFrameData);
        intFrameData = nil;     
    }

    // Detatch from Java
    (*g_vm)->DetachCurrentThread (g_vm);

    g_vm = nil;

    [super dealloc];
}

@end

Here’s the method signature on the java side for the java object reference that is passed in:

protected void pushFrame(byte[] frameData, int formatInt, int width, int height, float frameRate);
protected void pushFrame(short[] frameData, int formatInt, int width, int height, float frameRate);
protected void pushFrame(int[] frameData, int formatInt, int width, int height, float frameRate);

Once you have a running application, USE INSTRUMENTS TO MAKE SURE YOU’RE DISPOSING OF REFERENCES PROPERLY!