Question

We’re doing an augmented reality application using the kinect.

This has some challenges:

We’ll have a bunch of objects resting on the plane. Fortunately, the Kinect can give us the plane’s normal vector. We need to know where to set the OpenGL camera (ideally, this will be where the Kinect is located). For that, we need to know where the Kinect is in 3D space, because the two locations should be coincident.

If you need additional clarification, please tell me. I realise that my explanation may not be as clear as it needs to be.

Answer 1

Assume the Kinect at the origin. You also know the distance of point (on the plane) from the Kinect – that’s what the Kinect gives you as distance of this point. You also know the normal. So all you have to do is finding the plane equation for the plane, translate it into a matrix and invert that.

The plane equation given a point and normal vector is

n·(r-r_0) = 0 ; r_0 = 0
n·r = 0

We know r as

r = (0, 0, -distance_at(kinect_depth_image_width/2, kinect_depth_image_height/2))

Now remember that a plane can be expressed as

p(s,t) = r + u * s + v * t

where u, v are the base vectors of the plane. So we can rewrite this as matrix

u_x v_x n_x r_x
u_y v_y n_y r_y
u_z v_z n_z r_z
  0   0   0   1

u and v can be determined, by choosing arbitrary “Up”, and “Right” vectors, I suggest you use Y=(0,1,0) and X=(1,0,0), and perform a ortogonalisation against your known normal n, yielding u and v.

By inverting that matrix you have the transformation from your zero point on the plane to the kinect.