I am using glm::rotate to rotate a transformation matrix for a cube in a scene.
"cube->t = glm::rotate(cube->t, stepTime * 50.f, glm::vec3(0.f, 1.f, 0.f));" is called once per frame, where cube->t is the matrix in question.
The strange thing is that over the course of 20 minutes (or two minutes if I rotate by stepTime * 5000.f instead of stepTime * 50.f), the cube scales noticeably on the X and Z axes, where the scaling on those two axes is the same at all times (the height of the cube never changes, but the width and depth change by exactly the same amount). In the case of 5000.f, becoming smaller, and with the normal 50.f or 100.f, becoming larger.
I thought this might be a question of rounding error, but other than that I have no idea what could be causing it. Is this rounding error? Can I solve it by normalizing the matrix on a regular basis? Does glm have a function for normalizing matrices, or do I have to write one myself?
If it is an incremental, cumulative, loss of precision: instead of in-place updating the cube, keep an original un-rotated cube and keep track of the total rotational angle. Then on each step create a cube from the original un-rotated cube, transformed by the total accumulated rotational angle. In this way your cube will not suffer from cumulative rounding problems.
Edit: In response to discussion with Miles, a more appropriate answer:
If you can separate the rotational and translation components of the transformation matrix, and the problem is primarily with the accumulation of rotation introducing shear or scaling, you can rectify the problem by extracting the angle from the rotational component and re-creating the rotation matrix with that angle but unit length.