I have a 3d object that I wish to be able to rotate around in 3d. The easiest way is to directly translate X and Y mouse motion to rotation about the Y and X axes, but if there is some rotation along both axes, the way the model rotates becomes highly counterintuitive (i.e. if you flip the object 180 degrees about one axis, your motion along the other axis is reversed).
I could simply do the above method, but instead of storing the amount to rotate about the two axes, I could store the full rotation matrix and just further rotate it along the same axes for each mouse drag, but I’m concerned that that would quickly have precision issues.
Create an accumulator matrix and initialize it with the identity.
Each frame, apply that to your modelview/world matrix state before drawing the object.
Upon mouse motion, construct a rotation matrix about the X axis with some sensitivity_constant * delta_x. Construct another rotation matrix about the Y axis for the other component. Multiply one, then the other onto the accumulator.
The accumulator will change as you move the mouse. When drawing, it will orient the object as you expect.
Also, the person talking about quaternions is right; this will look good only for small incremental changes. If you drag it quickly on a diagonal, it won’t rotate quite the way you expect.