I’ve tried on my own using a “between the equations of a line” approach, but I need to do the following:
I have a matrix, n by n, which store 2D histogram counts. I need to be able to specify points in order, and have the program count everything between these points.
For now at least, I would be most content with a simple rectangle (however, the rectangle can be rotated any number of degrees).

From my Paint.exe’d picture of the histogram, you can see I’d like to be able to count within the blue boxes. Counting the horizontal (top right) rectangle is not a problem (specify the boundaries in a For loop as start/end bins of the matrix).
I’m stuck on how to define the boundaries in code to count within the other (leftmost) blue box. I’m using Igor Pro (from WaveMetrics) to do this, so for the most part, this is non-specific to a language.
Basically this is for analyzing areas of interest in these graphs. There are tools to analyze images which come with “within a polygon/freeform” type things, but they cannot accurately get the counts from this matrix (they analyze based on image colors, not counts). Also, I cannot filter based on “is there more than X in this bin?” as the same rectangle must be applied to a baseline “noise” matrix.
Ideas? I’m really stuck on getting a core concept of how this would work..
EDIT: My attempt, which does not appear to work properly, specifically came up empty when I put in a “box”, similar to the right blue box above. I can’t necessarily varify the skewed rectangle either (as we have no real way of counting it anyways..)
// Find polygon boundaries
s1 = (y2-y1)/(x2-x1)
o1 = s1==inf || s1==-inf ? 0 : y2 - (s1*x2)
s2 = (y3-y2)/(x3-x2)
o2 = s2==inf || s2==-inf ? 0 : y3 - (s2*x3)
s3 = (y4-y3)/(x4-x3)
o3 = s3==inf || s3==-inf ? 0 : y4 - (s3*x4)
s4 = (y1-y4)/(x1-x4)
o4 = s4==inf || s4==-inf ? 0 : y1 - (s4*x1)
// Get highest/lowest points (used in For loop)
maxX = max(max(max(x1, x2), x3), x4)
maxY = max(max(max(y1, y2), y3), y4)
minX = min(min(min(x1, x2), x3), x4)
minY = min(min(min(y1, y2), y3), y4)
For (i=minX; i<=maxX; i+=1) // Iterate over each X bin
For (j=minY; j<=maxY; j+=1) // Iterate over each Y bin
// | BETWEEN LINE 1 AND LINE 3? | | BETWEEN LINE 2 AND LINE 4? |
If ( ( ((s1*i + o1) > j && j > (s3*i + o3)) || ((s1*i + o1) < j && j < (s3*i +o3)) ) && ( ((s2*i + o2) > j && j > (s4*i + o4)) || ((s2*i + o2) < j && j < (s4*i +o4)) ) )
totalCount += matrixRef[i][j] // Add the count of this bin to the total count
EndIf
EndFor // End Y iteration
EndFor // End X iteration
You can count the values in the diagonal rectangle the same way as in the horizontal rectangle, you will just have a more complex loop to determine where the boundaries are. You can do this by looping over a horizontal rectangle that includes the entire diagonal rectangle and only counting the values if they fall inside of the diagonal one.
If you know the points that make up the corners of the diagonal rectangle then you can get the equations for each boundary (slope-intercept equation). From there you see what side of each boundary the points are on. If they are on the sides that correspond to the inside of the rectangle you include them, if they are not then you don’t.
You are going to have to find the X and Y points of the borders for each point that you are working on. If you are only checking against the border on the bottom left you would have something like:
If you have the equations for the boundaries and you know what point you are analyzing you can use the known X value (from the point) to get the Y value of the boundary (how high it is in that column that you are looking at) and the know Y value (again, from the point you are analyzing) to get the X value of the boundary (how far in the boundary is at the height of the point).
For the full list of conditions you should have the point be: