Question

I am trying to extract out 3D distance in mm between two known points in a 2D image. I am using square AR markers in order to get the camera coordinates relative to the markers in the scene. The points are the corners of these markers.

An example is shown below:

The code is written in C# and I am using XNA. I am using AForge.net for the CoPlanar POSIT
The steps I take in order to work out the distance:

1. Mark corners on screen. Corners are represented in 2D vector form, Image centre is (0,0). Up is positive in the Y direction, right is positive in the X direction.

2. Use AForge.net Co-Planar POSIT algorithm to get pose of each marker:

    float focalLength = 640; //Needed for POSIT
    float halfCornerSize = 50; //Represents 1/2 an edge i.e. 50mm
    AVector[] modelPoints = new AVector3[]
    {
         new AVector3( -halfCornerSize, 0,  halfCornerSize ),
         new AVector3(  halfCornerSize, 0,  halfCornerSize ),
         new AVector3(  halfCornerSize, 0, -halfCornerSize ),
         new AVector3( -halfCornerSize, 0, -halfCornerSize ),
    };
    CoplanarPosit coPosit = new CoplanarPosit(modelPoints, focalLength);
    coPosit.EstimatePose(cornersToEstimate, out marker1Rot, out marker1Trans);

3. Convert to XNA rotation/translation matrix (AForge uses OpenGL matrix form):

    float yaw, pitch, roll;
    marker1Rot.ExtractYawPitchRoll(out yaw, out pitch, out roll);

    Matrix xnaRot = Matrix.CreateFromYawPitchRoll(-yaw, -pitch, roll);
    Matrix xnaTranslation = Matrix.CreateTranslation(marker1Trans.X, marker1Trans.Y, -marker1Trans.Z);
    Matrix transform = xnaRot * xnaTranslation;

4. Find 3D coordinates of the corners:

    //Model corner points
    cornerModel = new Vector3[]
    {
        new Vector3(halfCornerSize,0,-halfCornerSize),
        new Vector3(-halfCornerSize,0,-halfCornerSize),

        new Vector3(halfCornerSize,0,halfCornerSize),
        new Vector3(-halfCornerSize,0,halfCornerSize)
    };

    Matrix markerTransform =  Matrix.CreateTranslation(cornerModel[i].X, cornerModel[i].Y, cornerModel[i].Z);
    cornerPositions3d1[i] = (markerTransform * transform).Translation;

    //DEBUG: project corner onto screen - represented by brown dots
    Vector3 t3 = viewPort.Project(markerTransform.Translation, projectionMatrix, viewMatrix, transform);
    cornersProjected1[i].X = t3.X; cornersProjected1[i].Y = t3.Y;

5. Look at the 3D distance between two corners on a marker, this represents 100mm. Find the scaling factor needed to convert this 3D distance to 100mm. (I actually get the average scaling factor):

    for (int i = 0; i < 4; i++)
    {
        //Distance scale;
        distanceScale1 += (halfCornerSize * 2) / Vector3.Distance(cornerPositions3d1[i], cornerPositions3d1[(i + 1) % 4]);
    }
    distanceScale1 /= 4;

6. Finally I find the 3D distance between related corners and multiply by the scaling factor to get distance in mm:

    for(int i = 0; i < 4; i++)
    {
       distance[i] = Vector3.Distance(cornerPositions3d1[i], cornerPositions3d2[i]) * scalingFactor;
    }

The distances acquired are never truly correct. I used the cutting board as it allowed me easy calculation of what the distances should be. The above image calculated a distance of 147mm (expected 150mm) for corner 1 (red to purple). The image below shows 188mm (expected 200mm).

What is also worrying is the fact that when measuring the distance between marker corners sharing an edge on the same marker, the 3D distances obtained are never the same. Another thing I noticed is that the brown dots never seem to exactly match up with the colored dots. The colored dots are the coordinates used as input to the CoPlanar posit. The brown dots are the calculated positions from the center of the marker calculated via POSIT.

Does anyone have any idea what might be wrong here? I am pulling out my hair trying to figure it out. The code should be quite simple, I don’t think I have made any obvious mistakes with the code. I am not great at maths so please point out where my basic maths might be wrong as well…

Answer 1

You are using way to many black boxes in your question. What is the focal length in the second step? Why go through ypr in step 3? How do you calibrate? I recommend to start over from scratch without using libraries that you do not understand.

Step 1: Create a camera model. Understand the errors, build a projection. If needed apply a 2d filter for lens distortion. This might be hard.

Step 2: Find you markers in 2d, after removing lens distortion. Make sure you know the error and that you get the center. Maybe over multiple frames.

Step 3: Un-project to 3d. After 1 and 2 this should be easy.

Step 4: ???

Step 5: Profit! (Measure distance in 3d and know your error)