Question

I have a simple rectangle here in my program on which I had to do hit testing. I am using openFrameworks but I think the problem here is related to OpenGL topics as well.

public class Shape : public class ofNode {
  Shape() {
    container.setFromCenter(getPosition(), 200,100);
    setPosition(ofVec3f(365, 50, 0)); //set the position of the node
    lookAt(ofVec3f(0,0,0));
  }

  virtual void draw() {
    ofRect(container); //the 200/100 rectangle will be drawn as per the container position and dimensions
  }
  ofVec3f getTopLeft() const {
    return getTopLeft() * ofNode::getGlobalTransformMatrix();
  }

  ofVec3f getTopRight() const {}    //similar - getTopRight from container and multiply
  ofVec3f getBottomLeft() const {}  //similar
  ofVec3f getBottomRight() const {} //similar

  bool hitTest(int tx, int ty) {
    // return true if mouse pointer is inside four vertices - hit test logic working fine
  }

  private:
    ofRectagle container;
};

The problem I have here is I am rotating and translating the shape inside my program:

void draw() {
  ofPushMatrix();
  ofTranslate(600,300,0);
  ofRotateY(rotate);
  shape->draw();

  ofCircle(shape->getTopLeft(), 10); //circle is drawn at the rectangle top left vertex on the screen but the coordinates while rotating and drawing do not change (as seen on cout)
  ofCircle(shape->getTopRight(), 10); //similar
  ofCircle(shape->getBottomLeft(), 10); //similar
  ofCircle(shape->getBottomRight(), 10); //similar
  ofPopmatrix();
}

On rotation being changed as in the above draw function, the points do not change. Hence, I do not have the appropriate four vertices for the hit test to figure out if the mouse is inside.
How do I get the position of the points in the screen which can be check against the mouse position for the hit test?

Answer 1

If you want to test if the mouse pointer is within the limits of the rectangle, I suggest you first transform them into screen space and do the test there, following the general idea I gave you in https://stackoverflow.com/a/14424267/524368

I’d also say, it’a about time to stop using OpenGL’s builtin matrix manipulation functions. There is absolutely no benefit in using them! Some people think they’d be GPU accelerated, but that’s not the case. As soon as you need those matrices at other points in your program it simply not practical to abuse OpenGL for this. OpenGL is not a math library after all. And in later versions the whole matrix stack has been dropped from OpenGL – good riddance.

OpenFrameworks comes with a fully featured matrix math library. I strongly suggest you use that. You can feed OpenGL the generated matrices with glLoadMatrix (fixed function pipeline) or glUniformMatrix (shader pipeline).

You can use it to implement a projection function as outlined by me in that other answer.

To test if a point lies within the boundaries defined by edges you can use a concept called “half edges”. Say your edges form a star domain loop. Then for each edge in the loop you take the cross product of the point with the edge. If the point is within the shape defined by the loop all the cross products will point into the same direction. Coplanar quadrilaterals (i.e. quadrilateals projected into screen space) always form a star domain.