Let’s say we have two concave 2d polygons(A, B), that don’t intersect. The problem is to find the set of edge pairs(each pair consist of one edge from polygon A, and the edge from polygon B), that have following property: each item in pair must be visible to each other. One edge is visible to another, if there are no obstacles(in the pic there are three cases, marked with red cross, when this rule is broken) between them.

I know how to solve this problem in O(n^2) using rays and vertices of edges. But it is too slow.
I don’t think it can be done faster than O(n^2).
See the picture below. There is a hyperbola and two polygons. The vertices of each polygon are on a branch of the hyperbola.
In this case, the edges of the two polygons are pair-wise visible (except the two edges behind). Then the resulting set will contain O(n^2) pairs of edges.