Given two planes positioned and rotated arbitrary to each other, I’m trying to determine which vertices should be connected. The image below (planes in black, connecting edges in red) illustrates two examples on how the planes might look:
Comparing the distances between vertices won’t work as the planes are not (necessarily) positioned directly above each other, and comparing the xyz components won’t work either since they are not (necessarily) parallel and therefore might be twisted. I can’t really think of any other solutions so I’m wondering if I need to go back and approach the problem in another way?
After ensuring that your rectangles are not intersecting and are not orthogonal, try minimizing the total distance between the paired points. There are only 24 ways to pair the vertices, so a clever solution is probably not needed. The other obvious way to approach this would be to ensure that the 4 new faces formed by connecting the vertices do not intersect, ie. you’ve formed a topologically simple solid.