Hello,
Perhaps I'm searching for the wrong terms, but I'm not quite finding an understandable answer to this question. Perhaps, I'm searching for the wrong terms, but of the ones I can find, they don't really give an good answer at all. At least one that I can make sense of. So, my genuine apologies if this question has been asked many times and I'm just failing to find their threads.
I'm trying to rotate a little 3D indicator arrow in the HUD that will point to an object, especially when the object is "offscreen." I understand how to rotate one object toward another generally speaking(in world coordinates), but I'm having difficulty since this arrow would take into account the ViewMatrix rotation, or screen space.
My initial attempts, I tried to use glm::project to parse out the screen coordinates and rotate toward those. This works great if the object is in view of the camera.
Something like this:
ProjectionMatrix= glm::perspective(45.0f, 4.0f / 3.0f, 0.1f, 100000.0f);
glm::vec3 projectedObject = glm::project(objPosition, ViewMatrix, ProjectionMatrix, glm::vec4(0,0,1920,1050));
quat rotation1 = RotationBetweenVectors(glm::vec3(0,0,1), glm::vec3(projectedObject.x - 960, projectedObject.y - 525, projectedObject.z) );
glm::mat4 Rotation;
Rotation = glm::mat4_cast(rotation1);
ModelMatrix = glm::translate(glm::vec3(0,-.5, -20)) * Rotation ;
MVP = ProjectionMatrix * ViewMatrix * ModelMatrix;
The RotationBetweenVectors function returns a quaternion representing the rotation toward the second object. From here.
But, if the object is offscreen, it doesn't work quite right, as it predictably goes back and forth as it hits the negative values. The z-axis also, predictably isn't quite working properly as glm:;project just gives a sort of positive/negative value, with little inbetween. In hindsight, I understand why this isn't working. I also tried the glm::lookat() function, but, since it doesn't take into account the current rotation of the viewmatrix, it also isn't quite working.
I can fanagle things to work a little better by doing:
quat rotation1;
if(projectedObject.z < 1)
rotation1 = RotationBetweenVectors(glm::vec3(0,0,1), glm::vec3(projectedObject.x - 960, projectedObject.y - 540, projectedObject.z) );
else
rotation1 = RotationBetweenVectors(glm::vec3(0,0,1), glm::vec3(-projectedObject.x - 960, -projectedObject.y - 540, -projectedObject.z) ); But, this is pretty inelegant, and only works for the X/Y axes, and not really the Z.
I guess I'm having difficulty quite visualizing what it is I need to do. I mean, I can visualize it, I'm just struggling to put this into an equation. But, in a nutshell, I need to take the current rotation of the ViewMatrix, and the calculate the rotation toward the destination object in screen space(or offscreen-space 
). And, this is for the full spectrum of rotations as the game is in space. So, there would be no preferred "UP" vector.
Anyway, if anyone has an insight on how to go about this, I'd be truly grateful for the help. Let me know if you need clarification or additional code at all, as this was probably a little rambling.
In the meantime, I'm going to go read articles on billboarding, as I feel like it would be the same principle, just instead of always facing the camera, The object would always face the targeted object.
Thanks all, 
