I found out how to properly rotate a quaternion on my own. The key was to find vectors for the axis I want to rotate around. Those are used to create quaternions from axis and angle, when angle is the amount to rotate around the actual axis.
The following code shows what I ended up with. It also allows to roll the camera, which might be useful some time.
void Rotate(btVector3 Amount, float Sensitivity)
{
// fetch current rotation
btTransform transform = camera->getWorldTransform();
btQuaternion rotation = transform.getRotation();
// apply mouse sensitivity
Amount *= Sensitivity;
// create orientation vectors
btVector3 up(0, 1, 0);
btVector3 lookat = quatRotate(rotation, btVector3(0, 0, 1));
btVector3 forward = btVector3(lookat.getX(), 0, lookat.getZ()).normalize();
btVector3 side = btCross(up, forward);
// rotate camera with quaternions created from axis and angle
rotation = btQuaternion(up, Amount.getY()) * rotation;
rotation = btQuaternion(side, Amount.getX()) * rotation;
rotation = btQuaternion(forward, Amount.getZ()) * rotation;
// set new rotation
transform.setRotation(rotation);
camera->setWorldTransform(transform);
}
Since I rarely found information about quaternion rotation, I'll spend some time further explaining the code above.
Fetching and setting the rotation is specific to the physics engine and isn't related to this question so I won't elaborate on this. The next part, multiplying the amount by a mouse sensitivity should be really clear. Let's continue with the direction vectors.
- The
up
vector depends on your own implementation. Most conveniently, the positive Y axis points up, therefore we end up with 0, 1, 0
.
- The
lookat
vector represents the direction the camera looks at. We simply rotate a unit vector pointing forward by the camera rotation quaternion. Again, the forward pointing vector depends on your conventions. If the Y axis is up, the positive Z axis might point forward, which is 0, 0, 1
.
- Do not mix that up with the next vector. It's named
forward
which references to the camera rotation. Therefore we just need to project the lookat
vector to the ground. In this case, we simply take the lookat
vector and ignore the up pointing component. For neatness we normalize that vector.
- The
side
vector points leftwards from the camera orientation. Therefore it lies perpendicular to both the up
and the forward
vector and we can use the cross product to compute it.
Given those vectors, we can correctly rotate the camera quaternion around them. Which you start with, Z, Y or Z, depends on the Euler angle sequence which is, again, a convention varying from application to application. Since I want to rotations to be applied in Y X Z order, I do the following.
- First, rotate the camera around the
up
axis by the amount for the Y rotation. This is yaw.
- Then rotate around the
side
axis, which points leftwards, by the X amount. It's pitch.
- And lastly, rotate around the
forward
vector by the Z amount to apply roll.
To apply those rotations, we need to multiply the quaternions create by axis and angle with the current camera rotation. Lastly we apply the resulted quaternion to the body in the physics simulation.