This seems non-trivial (it gets asked quite a lot on various forums), but I absolutely need this as a building block for a more complex algorithm.
Input: 2 polygons (A and B) in 2D, given as a list of edges [(x0, y0, x1, y2), ...] each. The points are represented by pairs of doubles. I do not know if they are given clockwise, counter-clockwise or in any direction at all. I do know that they are not necessarily convex.
Output: 3 polygons representing A, B and the intersecting polygon AB. Either of which may be an empty (?) polygon, e.g. null.
Hint for optimization: These polygons represent room and floor boundaries. So the room boundary will normally fully intersect with the floor boundary, unless it belongs to another floor on the same plane (argh!).
I’m kind of hoping someone has already done this in c# and will let me use their strategy/code, as what I have found so far on this problem is rather daunting.
EDIT: So it seems I’m not entirely chicken for feiling faint at the prospect of doing this. I would like to restate the desired output here, as this is a special case and might make computation simpler:
Output: First polygon minus all the intersecting bits, intersection polygons (plural is ok). I’m not really interested in the second polygon, just its intersection with the first.
EDIT2: I am currently using the GPC (General Polygon Clipper) library that makes this really easy!
What I think you should do
Do not attempt to do this yourself if you can possibly help it. Instead, use one of the many available polygon intersection algorithms that already exist.
I was strongly considering the following codebase on the strength of their demonstration code and the fact that they mentioned their handling of most/all of the weird cases. You would need to donate an amount (of you/your company’s choice) if you use it commercially, but it’s worth it to get a robust version of this kind of code.
http://www.cs.man.ac.uk/~toby/gpc/
What I actually did was to use a polygon-intersection algorithm that is part of the Java2D libraries. You can possibly find something similar in MS’s own C# libraries to use.
There are other options out there as well; look for “polygon clipper” or “polygon clipping”, since the same basic algorithms that handle polygon intersection also tend to be usable for the general clipping cases.
Once you actually have a polygon clipping library, you just need to subtract polygon B from polygon A to get your first piece of output, and intersect polygons A and B to get your second piece of output.
How to roll your own, for the hopelessly masochistic
When I was considering rolling my own, I found the Weiler-Atherton algorithm to have the most potential for general polygon-cutting. I used the following as a reference:
http://cs1.bradley.edu/public/jcm/weileratherton.html
http://en.wikipedia.org/wiki/Weiler-Atherton
The details, as they say, are too dense to include here, but I have no doubt that you’ll be able to find references on Weiler-Atherton for years to come. Essentially, you split all the points into those that are entering the final polygon or exiting the final polygon, then you form a graph out of all the points, and then walk the graph in the appropriate directions in order to extract all the polygon pieces you want. By changing the way you define and treat the “entering” and “exiting” polygons, you can achieve several possible polygon intersections (AND, OR, XOR, etc.).
It’s actually fairly implementable, but like with any computational geometry code, the devil is in the degeneracies.