I have a Cube which I want to rotate using mouse motion. So if I move my mouse to the left it rotates as if rotationX was increased. When I move right rotationX is increased. Moving Up and Down alters rotationY.
This looks fine as long as you modify only one dimension (either rotationX or rotationY). But if I have already changed rotationX and then try to modify rotationY the results are weird. If rotationX is 180 rotationY effects are reversed. It basically is quite unpredictable and certainly not suitable for user.
I have tried to look around (failed), read about quaternions (couldn’t understand), tried some out of blue and very complex trigonometric calculations (made a fool out of myself) and so I am stuck now. I’ve found this SO question Best way to translate mouse drag motion into 3d rotation of an object but it doesn’t help me a bit, I can’t understand any of the answers, really.
A complete, working solution/algorithm in AS3 would be the best, but some pseudo-code or just clear explanation will be really appreciated.
You are applying the rotationY to an already rotated object. So if you set rotationX to 180, and THEN set rotationY, you are really rotating the object standing “on its head”. If you want to apply both the x and y rotation at the same time, it is best to use a matrix. The link you provided is useful: it explains everything once you understand the concept of matrices. You should read up on this, especially if you intend to do more 3D in the future.
For now, just have a look at the Flash manual page for Matrix3D. It should provide all the information and examples you need to get going.