Question

How can a find if a point lies within a 2D rectangle given 4 points?

Answer 1

Transform the point to a coordinate frame aligned with the rectangle, then the problem becomes axis-aligned and trivial.

If the rectangle consists of the following 4 points:

a  b
c  d

Then get the “x-axis” and “y-axis” of the rectangle as:

x = Normalize(d-c)
y = Normalize(a-c)

Then construct a rotation matrix using x and y as columns:

r = [ x | y ]

If you’re using 3-d coordinates, we need a z axis:

z = CrossProduct(x, y)
r = [ x | y | z ]

Your transform matrix from world coordinates to your rectangle’s axis-aligned coordinates becomes:

T = [ r^T | -r^T * c ]
    [ 0^T |     1    ]

Here we’ve chosen the lower-left corner c to be the local origin. “r^T” is r transposed. “0^T” is either a 2-d or 3-d row-vector filled with zeros. 1 is just a one. Note that this is just the inverse of the simpler rectangle-to-world transform, which is

T^-1 = [ r   | c ]
       [ 0^T | 1 ]

We can use T to transform the point to axis-aligned coordinates. Remember to pad p with a trailing 1, since T is a homogeneous matrix.

tp = T * p;  // Don't forget to pad p with a trailing 1 before multiplying.

// Checks that p isn't below or to the left of the rectangle.
for ( int d = 0; d < num_dimensions; ++d ) {
  if ( tp[d] < 0.0 ) {
    return false;
  }
}

// Checks that p isn't to the right of the rectangle
double width = Length(d-c);
if ( tp[0] > width ) {
  return false;
}

// Checks that p isn't above the rectangle.
double height = Length(a-c);
if ( tp[1] > height ) {
  return false;
}

// p must be inside or on the rectangle.
return true

If you’re using 3d coordinates, note that the above disregards the local z value of transformed point tp. Even if p is out of the plane of the rectangle, the above behaves as if it’s been projected to the rectangle surface. If you want to check for coplanarity, just do the following beforehand:

if ( fabs(tp[2]) > some_small_positive_number ) {
   return false;  // point is out of the rectangle's plane.
}