I've been working on mouse picker on a OpenGL project and followed this guide to transform my 2D mouse position to a 3D vector on click.
The problem I have is in my calculations for picking the object/mesh. It seems to work properly if I am in the origin position of the camera, but as soon as I start to move around with the camera my calculated bounding sphere for the mesh gets out of position. I assume that I need to do some kind of matrix transformation to fix this problem but I can't figure it out. I do have access to the view matrix but I'm not sure If I should implement it in my calculations.
A video of my problem. It can be a bit hard to understand the problem with no click sounds. It gets more visually clear towards the end when I am able to select the object by clicking next to it.
Data I have for calculations:
- The position of the camera (rayOrigin in calculations)
- The mesh and all of its vertices and also the position of the mesh in world space.
- The 3D vector for the mouse click calculated with the same method as the guide mentioned above.
The first thing I've done is to re-calculate all the objects vertices to fit the position of the mesh in world space. The position of the mesh should also be the center of the mesh (at least in the video I linked earlier).
vertecies[i] = vec3(worldMatrix * vec4(vertecies[i], 1));
In my second step I compare each vertex with the center position and calculate which vertex that has the largest magnitude from the center. This vectors lenght I use as radius for my bounding sphere.
My calculations for the bounding sphere ray test, rayDir is the mouse position vector, rayOrigin is the camera position:
vec3 oc = rayOrigin - center;
float doc = dot(oc , rayDir);
if (doc > 0 || (dot(doc, doc) < radiusSquared)) return false; // no hit
vec3 a = oc - doc * rayDir;
float aSquared = dot(a, a);
if (aSquared > radiusSquared) return false; // no hit
return true; // hit
My linear algebra isn't the best so I'm hoping you guys can help me out and tell me what I do wrong here.