Question

Hi i’m developing a VNC client app for iPhone (it’s for a school project) and I’m using the RFB protocol.

I’m at the point where I receive the matrix of pixels (with 32 bit per pixel) and I need to display them, but I need it to be fast and as a VNC client I need to update the image quickly when changes occur.

I tried with CGBitmapContextCreate and I’m able to create the image and display it but every time I need to update it I have to create a new image so it’s not a great solution.

I was wondering if there is something more useful or a better way to display my matrix of pixels.

Answer 1

I assume your server sends partial updates, and doesn’t send a full screen snapshot on every update.

The simple technique is to create one CGBitmapContext and keep it around. Update its data directly (using CGContextFillRect would be much slower) when you get new pixels and then create an image from it using CGBitmapContextCreateImage and assign that image to view.layer.contents. (Don’t forget to release the image after storing it in that property!)

As you have discovered, though, you have to create a new image every time. Why? A CGImage is immutable. To enforce immutability, Core Graphics copies the pixel data to private storage when it creates a CGImage. Then when you display the image in a layer, iOS has to copy the pixel data again to the window server (which is a separate process). Copying a whole screen’s worth of pixel data is slow: you’re copying anywhere from 614 KB (iPhone 3GS and earlier) to 12.6 MB (Retina iPad) depending on the device.

One way you can speed this up while still staying in the world of UIKit and Core Animation is to tile the screen with a grid of layers. Let’s say you tile it with layers that are each 64×64 pixels. You create a CGBitmapContext for each layer. Then, when you receive pixel data from the server, you update the data of the appropriate CGBitmapContexts and then update the contents of just the layers that need it. When just a small area of the screen changes, you will only update one or a few tiles, so you will copy much less data. At 64×64 pixels, you’re copying 16 KiB per tile. You may want to try a few different sizes to figure out what works best.

If this still isn’t fast enough, you will need to leave the world of UIKit and Core Animation and use OpenGL ES. On iOS, OpenGL ES provides the most direct access to the video card that you are allowed to have unless your email address ends with @apple.com. You can create a texture with your pixel data, and draw two triangles (arranged to form a rectangle) with that texture, covering a CAEAGLLayer. When you get new pixel data, you can write it into the texture using glTexSubImage2D and redraw the triangles. If you haven’t done much OpenGL programming, this will be a pretty big effort.