Screen coordinate unprojection to 3D coordinate(ray)

Hi Ruben,

You‘re very close. The z- and w-coordinate of the clipping space position are a little off.
The w-coordinate is always 1 and the z-coordinate is for the near plane -1 and for the far plane +1.

Eventually you have to project the world coordinates back to Cartesian space, so in the end it should look like this:
{
vec4 normalizedCoordNear = vec4(normalizedCoord2D.x, normalizedCoord2D.y, -1.0, 1.0);
vec4 clipCoordNear = normalizedCoordNear;
vec4 eyeCoordNear = projectionMatrixInverse * clipCoordNear;
worldCoordNear = viewMatrixInverse * eyeCoordNear;
worldCoordNear /= worldCoordNear.w;
}
{
vec4 normalizedCoordFar = vec4(normalizedCoord2D.x, normalizedCoord2D.y, 1.0, 1.0);
vec4 clipCoordFar = normalizedCoordFar;
vec4 eyeCoordFar = projectionMatrixInverse * clipCoordFar;
worldCoordFar = viewMatrixInverse * eyeCoordFar;
worldCoordFar /= worldCoordFar.w;
}

You can compare your results to what comes out of gluUnproject (should yield the same).

Hope it helps!
Cheers

Tsus, thank you so much! I appreciate.

I guess the next thing would be to build a ray that goes through unprojected near/far coordinates and intersect it with my plane? I'm already using GLM which does not seem to have Ray/Plane implementation. Is there any open library available that I can reuse its Ray and Plane implementations?

Would I be able to reuse the depth value to reconstruct 3D coordinate without intersecting the ray with my objects? Would I just need to put the depth value into the:

vec4 normalizedCoord = vec4(normalizedCoord2D.x, normalizedCoord2D.y, depthValue, 1.0);
?

Thanks! It really helped!

Hi Ruben!

I guess the next thing would be to build a ray that goes through unprojected near/far coordinates and intersect it with my plane?

Yes! This would be the most common approach.

I'm already using GLM which does not seem to have Ray/Plane implementation. Is there any open library available that I can reuse its Ray and Plane implementations?

Sure, I think so. Though, intersecting rays with planes or triangles is not so difficult. You could do that yourself, I have no doubt.

Depending on the amount of rays you have to cast you could use a ray-casting framework like Nvidia’s OptiX, since this would already bring you acceleration data structures like Kd-trees or SVBH trees for free, but for the most use cases OptiX might be a little overkill. (Except you’re doing photon mapping or ray tracing and friends.)

It’s probably the best (an easiest) to implement the things you need yourself, as it is probably not much.
A plane is usually stored in Hesse normal form [formula] ax_0+bx_1+cx_2-d=0 [/formula], whereas [formula](a,b,c)[/formula] is the normal and [formula]d[/formula] the distance to the origin, which means you can store everything in a vec4 (normal and distance).

A ray struct most often looks like this:
struct Ray {
public:
vec3 origin;
vec3 direction;
float startDistance;
float endDistance;
};
The start distance make sense, since you often need a numerical offset (e.g. in case you start a ray from the surface of an object and don’t want a self-intersection.). The end distance is useful if you want to limit the distance, e.g. pre-computing ambient occlusion without an endDistance would yield in a closed room total darkness, since in every direction is a blocker.

This article here outlines many intersection algorithms for all kinds of objects checked against each other (including rays and planes). I figure that might be helpful to you.

Would I be able to reuse the depth value to reconstruct 3D coordinate without intersecting the ray with my objects? Would I just need to put the depth value into the:

vec4 normalizedCoord = vec4(normalizedCoord2D.x, normalizedCoord2D.y, depthValue, 1.0);
?

Yes, exactly! If you put in the depth value (e.g. from your depth buffer) you get the world position back. Incidentally, it is sufficient to store only the depth in a g-buffer, since you can always reconstruct its original position. Though, you can’t figure out so easily which object you hit.

Hope this also helps!
Cheers!

Tsus,
I'm pulling the hair off from my head. But it still does not help. Something does not work well. I'm not getting correct numbers.
In my application I have a chess kind of board, cell has 100px dimension and when I click on the boundary I expect to get coordinate which is divisible by 100. But that does not happen.

Here is my code:


MathLib::mat4 projectionMatrix = GetPerspectiveMatrix();
MathLib::mat4 viewMatrix = GetViewMatrix();
MathLib::mat4 projectionMatrixInverse = glm::inverse(projectionMatrix);
MathLib::mat4 viewMatrixInverse = glm::inverse(viewMatrix);
MathLib::vec2 screenCoord = MathLib::vec2(x, y);
MathLib::vec2 normalizedCoord2D = MathLib::vec2(2.0 * screenCoord.x / viewportSize.x - 1.0, 2.0 * screenCoord.y / viewportSize.y - 1.0);
MathLib::vec4 worldCoordNear;
MathLib::vec4 worldCoordFar;
{
MathLib::vec4 normalizedCoordNear = MathLib::vec4(normalizedCoord2D.x, normalizedCoord2D.y, -1.0, 1.0);
MathLib::vec4 clipCoordNear = normalizedCoordNear;
MathLib::vec4 eyeCoordNear = projectionMatrixInverse * clipCoordNear;
worldCoordNear = viewMatrixInverse * eyeCoordNear;
worldCoordNear /= worldCoordNear.w;
}
{
MathLib::vec4 normalizedCoordFar = MathLib::vec4(normalizedCoord2D.x, normalizedCoord2D.y, 1.0, 1.0);
MathLib::vec4 clipCoordFar = normalizedCoordFar;
MathLib::vec4 eyeCoordFar = projectionMatrixInverse * clipCoordFar;
worldCoordFar = viewMatrixInverse * eyeCoordFar;
worldCoordFar /= worldCoordFar.w;
}
MathLib::ray viewRay(MathLib::vec3(worldCoordNear), MathLib::vec3(worldCoordFar - worldCoordNear));
MathLib::plane groundPlane(MathLib::vec3(0, 0, 1), 0);
MathLib::vec3 intersectedPoint;
bool result = viewRay.intersect(groundPlane, intersectedPoint);


The intersection function:

bool intersect(const Plane<Type> & p, glm::detail::tvec3<Type> & point) const
{
// http://www.siggraph.org/education/materials/HyperGraph/raytrace/rayplane_intersection.htm
Type vd = glm::dot(p.GetNormal(), _direction);
if (vd == 0)
return false;
Type v0 = -(glm::dot(p.GetNormal(), getOrigin()) + p.GetDistance());
Type t = v0 / vd;
point = _origin + _direction * t;
return true;
}


Thank you!
Ruben

Hi Ruben,

Frankly, both code snippets look fine. I cross-checked the intersection code with Octave and I don’t see mistakes in the unprojection. Could you compare your obtained world position of the near and far plane to the result of gluUnproject?

Btw, an equal to zero test with floats is quite unsafe. Better do:
if (abs(vd) < 0.00001)

Could you give us some input and output values (view and projection matrix, mouse coordinates, the unprojected points etc), so that we can check the results against code we have lying around, here?

Just out of curiosity… your up-vector is (0,0,1) right? Otherwise, your chess board wouldn’t be horizontal (according to the plane equation).

Thanks!

Tsus, I didnt have a chance to try the gluUnproject, but will try it out when I get back from work today.
Yes, the up vector is (0, 0, 1). Regarding the zero test I know that there is a potential problem, but just want to keep everything simple until it works and then will prettyfy and cover corner cases.

Thank you!!!

Here are some numbers:

Camera Position: [100.34367 239.79147 234.27966]
PerspectiveAspect: 1.7589803
PerspectiveFovy: 45.000000
PerspectiveZFar: 20000.000
PerspectiveZNear: 0.10000000
Viewport: 1518 x 863
Mouse: 115 x 402
ProjectionMatrix:
[1.37251, 0, 0, 0
0, 2.41421, 0, 0
0, 0, -1.00001, -0.200001
0, 0, -1, 0]

ProjectionMatrixInverse:
[0.728594, 0, -0, 0
0, 0.414214, 0, -0
-0, 0, -0, -1
0, -0, -4.99997, 5.00002]

ViewMatrix:
[-0.922488, 0.386026, -8.60301e-009, 2.01551e-006
-0.258441, -0.617598, 0.74282, -1.52588e-005
0.286748, 0.685242, 0.669491, -349.937
0, 0, 0, 1
]
ViewMatrixInverse:
[-0.922488, -0.258441, 0.286748, 100.344
0.386026, -0.617598, 0.685242, 239.791
-1.49012e-008, 0.74282, 0.669491, 234.28
-0, 0, -0, 1]

normalizedCoord2D: [-0.84848487 -0.068366162]
worldCoordNear: [100.373, 239.701, 234.211, 1]
worldCoordFar: [5965.37, -18037.6, -13690.1, 1]
intersectedPoint: [199.023, -67.7272, 0]