Original Post
I'm trying to implement a camera like the ones you find in FPS games where you can look in different separate directions with the mouse. I've managed to separate the direction rotation matrices, but this has introduced a weird bug: my "forward" vector isn't always the same as the camera's forward vector! This is my code:
CVector up(0.0f, 1.0f, 0.0f), look(0.0f, 0.0f, 1.0f), right(1.0f, 0.0f, 0.0f);
CMatrix rotX = Rotation(right, orientY); // rotation around the X axis, orientY radians
look = rotX * look;
up = rotX * up;
// look and up are correct
CMatrix rotY = Rotation(up, orientX); // rotate around the up axis, this yields the behavior I want.
// CVector(0.0f, 1.0f, 0.0f) makes the camera rotate on the sloping plane caused by rotX
look = rotY * look;
right = rotY * right;
up = rotY * up;
// the Y component of "look" changes when I rotate on the Y axis (change orientX), this is probably the
// source of the bug. It remains the same as after rotX if I use CVector(0.0f, 1.0f, 0.0f) as up in the
// rotY-rotation. I only want look.y to change when orientY has changed.
CMatrix mat(-right.getNormalized(), -up.getNormalized(), look.getNormalized());
transform = mat * Translation(position); // "transform" is used as a basis when rendering
I'm then adding the look vector to the position when moving forward. I tried to save the Y component from the first rotation and using that to override the Y component calculated in the second rotation, it worked but was ugly and there were smaller bugs. And this is weird: how can the look vector (which is the same as the Z axis of the camera's transformation matrix, I've checked) have another direction than what the camera points at - what I see? This is really getting frustrating, it looks OK to me.