How to get rid of camera z-axis rotation

I use front, side and up vector for my camera. after each update, I will set the y component of the side vector zero, so that no rotation around z axis is possible. After that, I will normalize the vector and recompute the front and up vector.

Could you please give the code responcible for recomputing the front and up vectors?

Or (better) the whole view matrix update.

D3DXVECTOR3 eye; //camera position

D3DXVECTOR3 at; //where you look at

D3DXVECTOR3 up(0,1,0); //assume the up vector is always pointing up, could have problem when you look straight up

D3DXVECTOR3 front = Normalize(at - eye);

D3DXVECTOR3 side = cross(front,up); //if you look through front vector, side vector will be on your right

use D3DXMatrixLookAtLH(....) or XMMatrixLookAtLH(...) to update the view matrix

Thanks, but side and front is not using in XMMatrixLookAtLH(...) or D3DXMatrixLookAtLH(...). Where and when I need to apply them?

D3DXVECTOR3 eye; //camera position

D3DXVECTOR3 at; //where you look at

D3DXVECTOR3 up(0,1,0); //assume the up vector is always pointing up, could have problem when you look straight up

D3DXVECTOR3 front = Normalize(at - eye);

D3DXVECTOR3 side = cross(front,up); //if you look through front vector, side vector will be on your right

use D3DXMatrixLookAtLH(....) or XMMatrixLookAtLH(...) to update the view matrix

Doesn't D3DXMatrixLookAtLH(....) or XMMatrixLookAtLH(...) take as income parameters the mentioned 'eye', 'at' and 'up' and construct 'front' and 'side' automatically?

If you rotate around the X axis, then rotate around the Y axis, then undo the rotation around the X axis, and finally undo the rotation around the Y axis, the resulting transformation is a rotation around the Z axis. This is a feature of rotations in 3D space, and you will observe it regardless of whether you use quaternions or anything else to represent them.

Perhaps instead of rotating around the X and Y axes, you meant to control pitch and yaw. If that's the case, use those angles to build the rotation from scratch every time.

D3DXVECTOR3 eye; //camera position

D3DXVECTOR3 at; //where you look at

D3DXVECTOR3 up(0,1,0); //assume the up vector is always pointing up, could have problem when you look straight up

D3DXVECTOR3 front = Normalize(at - eye);

D3DXVECTOR3 side = cross(front,up); //if you look through front vector, side vector will be on your right

use D3DXMatrixLookAtLH(....) or XMMatrixLookAtLH(...) to update the view matrix

Doesn't D3DXMatrixLookAtLH(....) or XMMatrixLookAtLH(...) take as income parameters the mentioned 'eye', 'at' and 'up' and construct 'front' and 'side' automatically?

After 'side = cross(front,up)', set 'side.y = 1', then normalize it, then recompute the up vector.

At this time, 'eye' and 'at' haven't changed, but 'up' has changed.

If you have rotation angle, álvaro's method would be simpler.

If you rotate around the X axis, then rotate around the Y axis, then undo the rotation around the X axis, and finally undo the rotation around the Y axis, the resulting transformation is a rotation around the Z axis. This is a feature of rotations in 3D space, and you will observe it regardless of whether you use quaternions or anything else to represent them.

Perhaps instead of rotating around the X and Y axes, you meant to control pitch and yaw. If that's the case, use those angles to build the rotation from scratch every time.

Could you please write working example for your suggestion?

We already tried to accum degrees, but as a result, camera moving not on orbit, but on trajectory 8 (rotating in all 3 axis at the same time).

If you're trying to do a 3rd person camera, I'll echo Álvaro and suggest you just keep track of camera yaw and position then construct the transform each frame.

Pseudo code:

class Camera
{
 float m_yaw; // Should be a value between 0 and 2 PI
 float m_pitch; // Value between -PI/2 and PI/2
 Vector3 m_position;
 Matrix  m_cameraMatrix;
 
 void Update()
 {
 	Vector2 deltaMousePos; // Assume this is a 2D vector that has the delta X/Y of our mouse cursor since the last frame.
 	
 	m_yaw += deltaMousePos.x;
 	m_pitch += deltaMousePos.y;
 	
 	// Construct our rotation based on our current yaw/pitch
 	Quaternion yRotation(Vector3::UnitY, m_yaw);
 	Quaternion xRotation(Vector3::UnitX, m_pitch);
 	
 	m_cameraMatrix = Matrix::Identity;
 	m_cameraMatrix *= (xRotation * yRotation);
 	m_cameraMatrix.translation = m_position; // Restore our position.
 }
 
}

I think we misunderstood each other:

I have free or whatever camera:

I calculate its rotation like this:

float heading = XMConvertToRadians(headingDegrees);
    float pitch = XMConvertToRadians(pitchDegrees);
    float roll = XMConvertToRadians(rollDegrees);

    XMVECTOR rot;

    /*rot=XMQuaternionRotationRollPitchYaw(pitch, heading, roll);*/
	/*m_orientation = XMQuaternionMultiply(m_orientation, rot);*/

	if (heading != 0.0f)
    {
		rot= XMQuaternionRotationAxis( XMLoadFloat3(&WORLD_YAXIS), heading);
        m_orientation = XMQuaternionMultiply(rot, m_orientation);
    }

    // Rotate camera about its local x axis.
    // Note the order the quaternions are multiplied. That is important!
    if (pitch != 0.0f)
    {
        rot = XMQuaternionRotationAxis( XMLoadFloat3(&WORLD_XAXIS), pitch);
        m_orientation=XMQuaternionMultiply(m_orientation, rot);
    }

Order of quaternion multiplication is changed because other way camera will rotate around Z-axis on each move.

So, now everything is ok, BUT...

If I move camera on diagonal in any side it will flip on 180 degrees after full turn and instead of normal view, I will receive exactly 180 degrees flipped camera view.

And its not all...

If I leave right quaternion multiplication order, There is no flipping at all while camera moving on diagonal, but camera become to rotate around Z-axis while fast moving on x-y axis.

So, solving one issue I receive another and vice versa.

My question is: How to get rid of both issues?

For now I have no idea how to lock camera flipping around while moving on diagonal direction.

Can you help me to solve this issue or this is not possible and I must tolerate and live with it?

Are headingDegrees, pitchDegrees and rollDegrees accumulators, or are they an instantaneous indication of how you are rotating?

Are headingDegrees, pitchDegrees and rollDegrees accumulators, or are they an instantaneous indication of how you are rotating?

This is just delta mouse position * rotation speed:

heading = -mouse.xPosRelative() * rotationSpeed;
		pitch = -mouse.yPosRelative() * rotationSpeed;
		roll = direction.x *cameraDX_Free->getCameraSpeedFlight() * elapsedTimeSec;
		cameraDX_Free->rotate(heading, pitch, 0.0f);

Even if you are using a free camera, I'm not sure why you would ever want roll (but it looks like you aren't actually doing anything with that value). As Álvaro said, you should probably be using those values as accumulators rather than just delta rotation angles. Otherwise if you are just applying those delta rotations each frame you're going to be rotating in whatever space you were previously in which could cause the weird behavior you are seeing - hence building the camera matrix EVERY frame so you keep your transform space "clean" since you are just rebuilding it based on a few values.

class Camera
{
  //... previous code
  
  void rotate(float yaw, float pitch)
  {
  	m_yaw += yaw;
  	// Wrap around if we exceed 2 PI, or go below 0
  	if (m_yaw > TWO_PI) 
  	{
  		m_yaw -= TWO_PI;
  	}
  	else if (m_yaw < 0.0f)
  	{
  		m_yaw += TWO_PI;
  	}
  	
  	m_pitch += pitch
  	// With pitch, we probably dont want to wrap around as that would cause some ugliness, so just clamp.
  	if (m_pitch > HALF_PI)
  	{
  		m_pitch = HALF_PI;
  	}
  	else if (m_pitch < -HALF_PI)
  	{
  		m_pitch = -HALF_PI;
  	}
  }
}

I am not sure what was wrong with my suggestion, which AdeptStrain explained perfectly well.

There is another thing you could do, which might be a smaller change to how you are doing things: If you want your orientation to maintain a certain up vector, just enforce that in every step. To do that you'd multiply `m_orientation' by the smallest rotation that maps the current up vector to the desired up vector (probably on the left, but I would have to think about it).

Even if you are using a free camera, I'm not sure why you would ever want roll (but it looks like you aren't actually doing anything with that value). As Álvaro said, you should probably be using those values as accumulators rather than just delta rotation angles. Otherwise if you are just applying those delta rotations each frame you're going to be rotating in whatever space you were previously in which could cause the weird behavior you are seeing - hence building the camera matrix EVERY frame so you keep your transform space "clean" since you are just rebuilding it based on a few values.

class Camera
{
  //... previous code
  
  void rotate(float yaw, float pitch)
  {
  	m_yaw += yaw;
  	// Wrap around if we exceed 2 PI, or go below 0
  	if (m_yaw > TWO_PI) 
  	{
  		m_yaw -= TWO_PI;
  	}
  	else if (m_yaw < 0.0f)
  	{
  		m_yaw += TWO_PI;
  	}
  	
  	m_pitch += pitch
  	// With pitch, we probably dont want to wrap around as that would cause some ugliness, so just clamp.
  	if (m_pitch > HALF_PI)
  	{
  		m_pitch = HALF_PI;
  	}
  	else if (m_pitch < -HALF_PI)
  	{
  		m_pitch = -HALF_PI;
  	}
  }
}

I already tried this, but this has no effect. Described Issues still present.

I think problem is that in curtain moment camera`s UP vector become something like 0,-1,0 instead of 0,1,0

So, my question here, how to set up correct UP vector after my manipulations with quaternions so, that no to break the view matrix?

Or this is issue with wrog view direction, because when I made 180 degrees rotation on diagonal, I still see the model. but I must not see it, because camera must look at the another point opposit to the model.

How do you think?

Even if you are using a free camera, I'm not sure why you would ever want roll (but it looks like you aren't actually doing anything with that value). As Álvaro said, you should probably be using those values as accumulators rather than just delta rotation angles. Otherwise if you are just applying those delta rotations each frame you're going to be rotating in whatever space you were previously in which could cause the weird behavior you are seeing - hence building the camera matrix EVERY frame so you keep your transform space "clean" since you are just rebuilding it based on a few values.

class Camera
{
  //... previous code
  
  void rotate(float yaw, float pitch)
  {
  	m_yaw += yaw;
  	// Wrap around if we exceed 2 PI, or go below 0
  	if (m_yaw > TWO_PI) 
  	{
  		m_yaw -= TWO_PI;
  	}
  	else if (m_yaw < 0.0f)
  	{
  		m_yaw += TWO_PI;
  	}
  	
  	m_pitch += pitch
  	// With pitch, we probably dont want to wrap around as that would cause some ugliness, so just clamp.
  	if (m_pitch > HALF_PI)
  	{
  		m_pitch = HALF_PI;
  	}
  	else if (m_pitch < -HALF_PI)
  	{
  		m_pitch = -HALF_PI;
  	}
  }
}

I already tried this, but this has no effect. Described Issues still present.

You are going to have to post some code to go with that. Building a rotation from scratch on every frame from yaw and pitch can't possibly leave you with a rotation around the z axis.

Here is my code:

take degrees for X and Y

if (mouseInput.MouseState.RightButton.Down)
            {
                basicCamera.RotateX(-mouseInput.MouseState.DeltaX * mouseRotationSpeed);
                basicCamera.RotateY(-mouseInput.MouseState.DeltaY * mouseRotationSpeed);
               
            }

Then do quaternion multiply:

public void RotateX(float _degree_angle)
        {
            if (cameraType == CamType.ThirdPerson)
            {
                _degree_angle = -_degree_angle;
            }

            qRotation = Quaternion.Multiply(Quaternion.RotationAxis(Vector3.UnitX, MathUtil.DegreesToRadians(_degree_angle)), qRotation);
            
            UpdateViewMatrix();
        }

        public void RotateY(float _degree_angle)
        {
            if (cameraType == CamType.ThirdPerson)
            {
                _degree_angle = -_degree_angle;
            }
            qRotation = Quaternion.Multiply(qRotation, Quaternion.RotationAxis(Vector3.UnitY, MathUtil.DegreesToRadians(_degree_angle)));
            
            UpdateViewMatrix();
        }

FInally update view matrix:

private void UpdateViewMatrix()
        {
            qRotation = Quaternion.Normalize(qRotation);

            if (CameraType == CamType.ThirdPerson)
            {
                ViewMatrix = Matrix.Translation(Vector3.Negate(position)) * Matrix.RotationQuaternion(qRotation);

                position -= new Vector3(ViewMatrix.M13, ViewMatrix.M23, ViewMatrix.M33) * radius;

                ViewMatrix = Matrix.LookAtLH(position, lookAt, new Vector3(ViewMatrix.M12, ViewMatrix.M22, ViewMatrix.M32));
            }
            else
            {
                ViewMatrix = Matrix.Translation(Vector3.Negate(position)) * Matrix.RotationQuaternion(qRotation);
            }
        }

P.S. quaternion multiply order in Y rotation changed because it prevent camera to move around its z-axis, but it causes it to turn upside down when moving on diagonal. If I dont change the order, then moving on diagonal is good, but camera is rotating on z-axis each time I move it around x and y axis.

Here is my code:

take degrees for X and Y

if (mouseInput.MouseState.RightButton.Down)
            {
                basicCamera.RotateX(-mouseInput.MouseState.DeltaX * mouseRotationSpeed);
                basicCamera.RotateY(-mouseInput.MouseState.DeltaY * mouseRotationSpeed);
               
            }

Then do quaternion multiply:

public void RotateX(float _degree_angle)
        {
            if (cameraType == CamType.ThirdPerson)
            {
                _degree_angle = -_degree_angle;
            }

            qRotation = Quaternion.Multiply(Quaternion.RotationAxis(Vector3.UnitX, MathUtil.DegreesToRadians(_degree_angle)), qRotation);
            
            UpdateViewMatrix();
        }

        public void RotateY(float _degree_angle)
        {
            if (cameraType == CamType.ThirdPerson)
            {
                _degree_angle = -_degree_angle;
            }
            qRotation = Quaternion.Multiply(qRotation, Quaternion.RotationAxis(Vector3.UnitY, MathUtil.DegreesToRadians(_degree_angle)));
            
            UpdateViewMatrix();
        }

FInally update view matrix:

private void UpdateViewMatrix()
        {
            qRotation = Quaternion.Normalize(qRotation);

            if (CameraType == CamType.ThirdPerson)
            {
                ViewMatrix = Matrix.Translation(Vector3.Negate(position)) * Matrix.RotationQuaternion(qRotation);

                position -= new Vector3(ViewMatrix.M13, ViewMatrix.M23, ViewMatrix.M33) * radius;

                ViewMatrix = Matrix.LookAtLH(position, lookAt, new Vector3(ViewMatrix.M12, ViewMatrix.M22, ViewMatrix.M32));
            }
            else
            {
                ViewMatrix = Matrix.Translation(Vector3.Negate(position)) * Matrix.RotationQuaternion(qRotation);
            }
        }

P.S. quaternion multiply order in Y rotation changed because it prevent camera to move around its z-axis, but it causes it to turn upside down when moving on diagonal. If I dont change the order, then moving on diagonal is good, but camera is rotating on z-axis each time I move it around x and y axis.

You aren't actually creating accumulators and building the matrix each frame from scratch. You're modifying "qRotation" every time you call RotateX/RotateY, so if you rotate 45 degrees on the X and Y, next frame if you do the same you're going to be rotating from within that previous "diagonal" space. Building it from scratch prevents you from ever rotating outside of the unit X/Y/Z space.

I still not really understand what do you mean from scratch. You mean I need to make qRotation Identity each time after multiply or what?

ViewMatrix is building from scratch each time.

Sorry, but I really don`t understand for now your concept. Maybe if you could change my code with yours (even pseudo code) it will help me to understand at last.