Avast there fellow programmers!
I have the following problem:
I have two rectangles overlapping like shown on the picture below.

I want to figure out the polygon consisting of point ABCDEF.
Alternate christmas description: The red cookie cutter is cutting away a bit of the black cookie. I want to calculate the black cookie.
Each rectangle is a data structure with 4 2d-vertices.
What is the best algorithm to achieve this?
This is a special case of general 2D polygon clipping. A good place to start is the Weiler-Atherton algorithm. Wikipedia has a summary and links to the original paper. The algorithm seems to match the data structure you’ve described pretty well.
Note that it’s quite possible you’ll end up with a rectangle with a hole in it (if the red one is entirely inside the black one) or even two rectangles (eg if the red is taller and skinnier than the black). If you’re certain there is only one corner of the red rectangle inside the black one then the solution should be much simpler.