When testing where a point is respective to a tetrahedron there are 4 cases. It’s inside the tetrahedron, it is closest to a vertex, closest to a edge, or closest to a face. Here’s a diagram of it in 2D, https://i.stack.imgur.com/L8IkF.png . For example a point P is closest to a vertex A when the dot product of P – A and B – A, C – A and D – A are all negative. If u were to stretch a triangular face in the direction of the normal to create a prisim, the point would be closest to that face if it is contained within that space. It would be closest to an edge if it were in neither of those cases.
I want to be able to exclusively test whether or not it is closest to an edge without relying on the results of a vertex or face test. Is there a relatively efficient way to do this? The tetrahedron is not regular.
Do you mean, closest to a particular edge, or closest to an edge in general (as opposed to a vertex or plane)?
If the former, then a point is closest to (a particular) edge if it lies in the intersection of four half-spaces defined by two planes perpendicular to the adjoining faces and passing through the edge, and two planes perpendicular to the edge and passing through the adjoining vertices. So test your point against all four planes and if it lies in the appropriate half-space for all of them, it’s closest to that edge.
In detail, suppose we want to test whether the point P is closest to the edge AB, which lies between the faces ABC and BAD (vertices given clockwise). Then the four tests are:
(I wonder if there’s a cheaper way to do the third and fourth tests? If you know of one, let me know!)