Question

We are doing an image processing project entitled “Face Replacement System” and we are referring to http://thesai.org/Downloads/Volume1No6/Paper_22_A_Face_Replacement_System_Based_on_Face_Pose_Estimation.pdf.

We implemented “image warping”, in which source image (face) (which is to replace the target image) is shifted, rotated and scaled to match the pose of the target image (face). The result obtained after image warping has holes i.e. not all the pixels of target image are mapped to the source image. The paper does not mention any solution.

We tried a simple solution: to take average of the pixels from 8 connected neighbors for each hole. We tried other algorithms too. But the results are poor.

What is the best algorithm to implement in this case?

Answer 1

You are using affine transformations (“shifted, rotated and scaled”), so it is possible to calc back transform, and make “reverse mapping” – for each destination pixel find corresponding source pixel. So the appearance of the holes is entirely eliminated.

Edit: Bilinear interpolation for comment discussion

foreach pixel in newimage do
   (px, py) = Transform(pixel.x, pixel.y) //float
    x = Floor(px), y = Floor(py)  //integer base coordinates in old image, rounded to -Infinity
    tx = px - x, ty = py - y   //float parametric coordinates in old image cell
    Coeff00 = (1 - tx) * (1 - ty)
    Coeff01 = tx * (1 - ty) 
    Coeff10 = (1 - tx) * ty
    Coeff11 = tx * ty
    NewRedValue = OldRedValue[x, y] * Coeff00 + 
                  OldRedValue[x + 1, y] * Coeff01 + 
                  OldRedValue[x, y + 1] * Coeff10 + 
                  OldRedValue[x + 1, y + 1] * Coeff11
    the same for blue, green