Question

I’m developing an application for video processing using GStreamer and OpenCV. The task is to fetch images from an OpenCV source push/pull them into an GStreamer pipeline (does the converting and decoding and I have to stream the video data later) and process the images in an Java language environment.

So I build a native framework in C managing this task. Now I use JNI to access my C-Library, and I’ve attached the executing GStreamer main loop to the JVM. There are some threads unrelated to the image processing for which I use the SDL library but there are not calling any Java methods. Apparently it works so far. I get the raw images into Java and I also can display its byte data using a ByteBuffer (created on the native side) get method and a System.err.println.

The problem is to display these images somehow in a GUI. I use the Swing framework for my graphical environment.

So I made a quick and dirty image view by extending an JPanel as I usually would and which usually works when I use Java only.

I override the paint method to draw the image using the Graphics.drawImage method. The image is generated by copying the raw data, delivered by the native callback, into an integer array and then assign this integer array to a MemoryImageSource. I use the createImage method (of JFrame) to get an Image object. Everything returns apparently valid references. No exceptions are thrown and all. At least none visible to me.

Tried several ways as well as just use a byte array for the MemoryImageSource none generated images so far. The data however is there and I can read it!

I can copy the data, display the byte data but I’m unable to get an displayable image showing up in my custom JPanel view area. It is like the images are completely black or transparent or something.

This problem really beats me, I have no Idea how to convince this thing to work. I assume there are several problems with the threading here. But I have no further resources for this specific scenario.

What am I doing wrong, or better how to do it the right way.

Additional:
The application also sometimes just crashes or the GUI freezes if I change the size the window, so I avoid doing this for the moment. So there are problems with the native threads.

Answer 1

I use the OpenCV-Library to access cameras installed on a system. I grab the dimension, color information and the BGR-Samples from each grabbed frame and let a GStreamer AppSrc pull images into a video processing pipeline according to the frame rate the whole thing was initialized with. The initial pipeline looks like this:

OpenCV-Camera –> AppSrc –> VideoRate –> ffmpegcolorspace –> AppSink

Its going to be more complex soon but right now this is how it looks like.

The AppSink calls a function on the Gstreamer “new-buffer” signal (among some other no related to my problems matter). These callback implementations invoke Java object functions delivering the image color and dimension as well as its sample buffer. So this as brief information about what I’m doing.

Now to my problem:

It seems I solved my problem. I now use another way to create my images and now it works.

Instead of using the memory image source like so:

//constructor, callback whatever
imageSource = new MemoryImageSource(data.width,data.height,rawdata,0,data.width);
...
//Paint method of a JFrame or what ever ...
...
image img = createImage(imageSource);
setBounds(100,100,img.getWidth(null),img.getHeight(null));
createBufferStrategy(2);

BufferStrategy strategy = getBufferStrategy();

Graphics gr = strategy.getDrawGraphics();           
gr.drawImage(img, 0, 0, null);
gr.dispose();

strategy.show();

I use these few lines which work for me right now:

int[] nBits = {8, 8, 8};
ComponentSampleModel sampleModel = new ComponentSampleModel(DataBuffer.TYPE_BYTE, width, height, channels, width * channels, new int[] {2, 1, 0});
ColorSpace colorSpace = ColorSpace.getInstance(ColorSpace.CS_sRGB);
ColorModel colorModel = new ComponentColorModel(colorSpace, nBits, false, false,Transparency.OPAQUE,DataBuffer.TYPE_BYTE);

DataBufferByte db = new DataBufferByte(new byte[][] {buffer}, buffer.length);

WritableRaster raster = Raster.createWritableRaster(sampleModel, db, new Point(0, 0));
BufferedImage image = new BufferedImage(colorModel, raster, false, null);

buffer denotes herein raw byte samples array I get throught my method invoked from the native side.

I also ran into several other problems using swing like a freezing UI or a crash throwing these error messages:

[xcb] Unknown request in queue while dequeuing
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
java: ../../src/xcb_io.c:178: dequeue_pending_request: assertion »!xcb_xlib_unknown_req_in_deq« failed.

This is because I used the UIManager.setLookAndFeel() method to set the windows appearance to the systems default. I didn’t link this with the problem because I didn’t know that Swing uses the native UI framework if you use default system “Look And Feels”. Removing this resolves the problem. My research lead me to a forum post (in german) which describes this occurring on Java only (no native part) on Linux (Ubuntu) machines while the same application works on windows machines without modification. So removing it solved my problem. I think my “image not showing up issue” is somehow related to this but I don’t know and I don’t care since I works now.