Question

I’m currently working on Image stitching using OpenCV 2.3.1 on Visual Studio 2010, but I’m having some trouble.

Problem Description
I’m trying to write a code for stitching multiple images derived from a few cameras(about 3~4), i,e, the code should keep executing image stitching until I ask it to stop.

The following is what I’ve done so far:
(For simplification, I’ll replace some part of the code with just a few words)

     1.Reading frames(images) from 2 cameras (Currently I'm just working on 2 cameras.)
     2.Feature detection, descriptor calculation (SURF)
     3.Feature matching using FlannBasedMatcher
     4.Removing outliers and calculate the Homography with inliers using RANSAC.
     5.Warp one of both images.

For step 5., I followed the answer in the following thread and just changed some parameters:
Stitching 2 images in opencv

However, the result is terrible though.
I just uploaded the result onto youtube and of course only those who have the link will be able to see it.

http://youtu.be/Oy5z_7LeaMk

My code is shown below:
(Only crucial parts are shown)

VideoCapture cam1, cam2;
cam1.open(0);
cam2.open(1);

while(1)
{
    Mat frm1, frm2;

    cam1 >> frm1;
    cam2 >> frm2;

   //(SURF detection, descriptor calculation 
   //and matching using FlannBasedMatcher)


    double max_dist = 0; double min_dist = 100;

    //-- Quick calculation of max and min distances between keypoints
    for( int i = 0; i < descriptors_1.rows; i++ )
    { 
        double dist = matches[i].distance;
        if( dist < min_dist ) min_dist = dist;
        if( dist > max_dist ) max_dist = dist;
    }


    (Draw only "good" matches 
    (i.e. whose distance is less than 3*min_dist ))            

    vector<Point2f> frame1;
    vector<Point2f> frame2;

    for( int i = 0; i < good_matches.size(); i++ )
    {
      //-- Get the keypoints from the good matches
      frame1.push_back( keypoints_1[ good_matches[i].queryIdx ].pt );
      frame2.push_back( keypoints_2[ good_matches[i].trainIdx ].pt ); 
    }
    Mat H = findHomography( Mat(frame1), Mat(frame2), CV_RANSAC );
    cout << "Homography: " << H << endl;


    /* warp the image */
    Mat warpImage2;
    warpPerspective(frm2, warpImage2, 
    H, Size(frm2.cols, frm2.rows), INTER_CUBIC); 

    Mat final(Size(frm2.cols*3 + frm1.cols, frm2.rows),CV_8UC3); 

    Mat roi1(final, Rect(frm1.cols, 0, frm1.cols, frm1.rows)); 
    Mat roi2(final, Rect(2*frm1.cols, 0, frm2.cols, frm2.rows)); 

    warpImage2.copyTo(roi2); 
    frm1.copyTo(roi1); 
    imshow("final", final); 

What else should I do to make the stitching better?

Besides, is it reasonable to make the Homography matrix fixed instead of keeping computing it ?
What I mean is to specify the angle and the displacement between the 2 cameras by myself so as to derive a Homography matrix that satisfies what I want.

Thanks. 🙂

Answer 1

It sounds like you are going about this sensibly, but if you have access to both of the cameras, and they will remain stationary with respect to each other, then calibrating offline, and simply applying the transformation online will make your application more efficient.

One point to note is, you say you are using the findHomography function from OpenCV. From the documentation, this function:

Finds a perspective transformation between two planes.

However, your points are not restricted to a specific plane as they are imaging a 3D scene. If you wanted to calibrate offline, you could image a chessboard with both cameras, and the detected corners could be used in this function.

Alternatively, you may like to investigate the Fundamental matrix, which can be calculated with a similar function. This matrix describes the relative position of the cameras, but some work (and a good textbook) will be required to extract them.

If you can find it, I would strongly recommend having a look at Part II: “Two-View Geometry” in the book “Multiple View Geometry in computer vision”, by Richard Hartley and Andrew Zisserman, which goes through the process in detail.