Ray-OBB intersection test valid only for 90 and 180 deg

Here are some things I noticed looking over your code:

• I don't know what all your variable names mean, and I haven't tried to fully analyze the functions for correctness, but are you sure your ray vs axis-aligned box test is implemented correctly? Have you tested it with a variety of input parameters?
• The math you're doing and the name 'dirfrac' seem to suggest that TestAABBIntersection() expects some 'prepared' data related to the direction vector, but in TestOBBIntersection() you just seem to be passing in the direction vector without any such preparation.
• You appear to be applying an Euler-angle rotation. If the Euler-angle conventions you're using in this code don't match the conventions used elsewhere, you could get erroneous results.
• I don't see you addressing the model's position anywhere in your code.

Maybe something there will be helpful.

you should use the reciprocal of the dir, not the frac

@Zakwayda

• I believe that ray-AABB is correctly implemented. I was already using it for a month or so. I have setup you can see on screen with model raypicking and axis arrows to move model arround. It work for different models and different positions and different camera angles.
  lb stands for left bottom (min bounds); rt stands for right top (max bounds); origin is camera position in world space.
• That's my fault of bad naming. It should also be dirfrac. That's what passed down.

MouseRaycastResult RaycastToModel(ModelClass* const model)
{
	//Go to [-1, 1] coordinates
	float x = GetCurrentMousePosition().first;
	float y = GetCurrentMousePosition().second;
	if (x > 1.0f)
		x = 1.0f;
	else if (x < -1.0f)
		x = -1.0f;

	if (y > 1.0f)
		y = 1.0f;
	else if (y < -1.0f)
		y = -1.0f;

	float pointX, pointY;
	XMMATRIX projectionMatrix, viewMatrix, inverseViewMatrix, worldMatrix, translateMatrix, inverseWorldMatrix;
	XMVECTOR origin, rayOrigin, direction, rayDirection;

	pointX = x;
	pointY = y;

	m_D3D->GetProjectionMatrix(projectionMatrix);
	pointX = pointX / projectionMatrix.r[0].m128_f32[0];
	pointY = pointY / projectionMatrix.r[1].m128_f32[1];

	m_Camera->GetViewMatrix(viewMatrix);
	inverseViewMatrix = XMMatrixInverse(nullptr, viewMatrix);

	direction = {
		(pointX * inverseViewMatrix.r[0].m128_f32[0]) + (pointY * inverseViewMatrix.r[1].m128_f32[0]) + inverseViewMatrix.r[2].m128_f32[0],
		(pointX * inverseViewMatrix.r[0].m128_f32[1]) + (pointY * inverseViewMatrix.r[1].m128_f32[1]) + inverseViewMatrix.r[2].m128_f32[1],
		(pointX * inverseViewMatrix.r[0].m128_f32[2]) + (pointY * inverseViewMatrix.r[1].m128_f32[2]) + inverseViewMatrix.r[2].m128_f32[2]
	};

	origin = { m_Camera->GetPosition().x, m_Camera->GetPosition().y, m_Camera->GetPosition().z };

	m_D3D->GetWorldMatrix(worldMatrix);
	translateMatrix = XMMatrixTranslation(model->GetPosition().x, model->GetPosition().y, model->GetPosition().z);
	worldMatrix = XMMatrixMultiply(worldMatrix, translateMatrix);

	inverseWorldMatrix = XMMatrixInverse(nullptr, worldMatrix);

	rayOrigin = XMVector3TransformCoord(origin, inverseWorldMatrix);
	rayDirection = XMVector3TransformNormal(direction, inverseWorldMatrix);

	rayDirection = XMVector3Normalize(rayDirection);

	const XMFLOAT3 dirfrac{ 1.0f / rayDirection.m128_f32[0], 1.0f / rayDirection.m128_f32[1], 1.0f / rayDirection.m128_f32[2] };
	const XMVECTOR dirfracNormalized = XMVector3Normalize({ dirfrac.x, dirfrac.y, dirfrac.z });
	const XMFLOAT3 dir = { dirfracNormalized.m128_f32[0], dirfracNormalized.m128_f32[1], dirfracNormalized.m128_f32[2] };

	return MouseRaycastResult{ {origin.m128_f32[0], origin.m128_f32[1], origin.m128_f32[2] }, dir };
}

I am finding ray by clicking mouse on screen and finding its direction. What I return is normalized inversed direction.

• I am using Euler-angle rotation for every model operations in my code, I believe
• lb, rt are min-max bounds in world-space. I take bounds in model-space and add current model position to it to transform to world-space

Does it make sense or am I mistaken on any part?

@Krypt0n

I am using dir returned in the code above. Is it right?

Yes, the reciprocal is correct. Why are you naming it frac? it's a bit confusing

(the rest of the ray-box intersection code looks correct to my eyes).

I was using many tutorials and forgot to rename some parameters after iterating on it.

Do you mean ray-AABB or ray-OBB? I have no idea why it doesn't work for all rotations, just 0/90/180 degrees.

40 minutes ago, komilll said:

I am using Euler-angle rotation for every model operations in my code, I believe

Generally speaking you also have to make sure Euler-angle conventions match from one place to another. I'm not saying they don't in your case, just that it can be a source of errors.

Quote

lb, rt are min-max bounds in world-space.

That was my guess, but e.g. 'lb' for left-bottom seemed a little confusing because it seems to suggest 2-d, whereas this is 3-d. Something like 'min' and 'max' (which you of course referred to above) would probably be more typical naming.

Quote

What I return is normalized inversed direction.

What's the purpose of normalizing the inverse direction? And is it the normalized inverse (rather than the original direction vector) that you're transforming by the model transform later? Not saying what you're doing is wrong - it's just not immediately obvious to me what all is going on (and there's too much code for me to analyze it fully at the moment), so I'm just curious if you can elaborate on that part of things.

I will check rotation operation using Euler-angle to make sure there aren't any errors.

Quote

What's the purpose of normalizing the inverse direction?

I think there isn't any. I was just testing some code and I'm pretty sure that normalized or not, it gives the same results. So I think you can ignore that, as it is just inversed non-normalized direction.

40 minutes ago, komilll said:

I think there isn't any. I was just testing some code and I'm pretty sure that normalized or not, it gives the same results. So I think you can ignore that, as it is just inversed non-normalized direction.

I do still wonder about the normalization, but I'd have to know more about the rest of the code to guess at what effect it might have, so I'll move on from that.

Something else that might be worth mentioning is that the slab test can be vulnerable to error if one or more direction components is zero or has very small magnitude. So we'd be talking about division by zero or numbers with small magnitude in this statement:

const XMFLOAT3 dirfrac{ 1.0f / rayDirection.m128_f32[0], 1.0f / rayDirection.m128_f32[1], 1.0f / rayDirection.m128_f32[2] };

You may be relying on language- or environment-specific behavior here, but I just thought I'd mention it in case you hadn't thought about it.

What I'm more curious about though is this. Here:

bool TestOBBIntersection(ModelClass* model, XMFLOAT3 origin, XMFLOAT3 dir, XMFLOAT3 lb, XMFLOAT3 rt, float & dist)

For 'dir', what are you passing in? The inverted (and normalized) direction vector? The original unmodified direction vector? Or something else?

@Zakwayda You had good point about what's actually passed in 'dir'. I created raycast debug line and found out where actual error is. Thank you!

I was returning dir as 1.0f / dir and then transforming it. I changed it to pass dir without inversion, transform it to OBB space and then inverse it and OBB works fine. However I found out, that it works only for location (0, 0, 0). Do you know what might be a problem?

1 hour ago, komilll said:

However I found out, that it works only for location (0, 0, 0). Do you know what might be a problem?

When you refer to only working at the origin, I'm not sure if you mean it only works when the ray originates at the origin, or when the model is positioned at the origin. In any case, in TestOBBIntersection(), you don't appear to be factoring in the model position, which seems like it could be related. Also, I don't know the details of the math library you're using, but keep in mind the distinction between positions and vectors when applying transforms.

@Zakwayda I cleaned code a little and here is a result:

bool TestOBBIntersection(ModelClass* model, XMFLOAT3 origin, XMFLOAT3 dir, XMFLOAT3 lb, XMFLOAT3 rt, float & dist)
{
	//Create inverse world matrix
	XMMATRIX worldMatrix = XMMatrixIdentity();
	//worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixScaling(model->GetScale().x, model->GetScale().y, model->GetScale().z));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationX(model->GetRotation().x * 0.0174532925f));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationY(model->GetRotation().y * 0.0174532925f));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationZ(model->GetRotation().z * 0.0174532925f));
	//worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, XMMatrixTranslation(model->GetPosition().x, model->GetPosition().y, model->GetPosition().z));
	const XMMATRIX inverseWorldMatrix = XMMatrixInverse(NULL, worldMatrix);

	//Transform origin point and direction ray vector
	const XMVECTOR originTransformed = XMVector3Transform({ origin.x, origin.y, origin.z }, inverseWorldMatrix);
	const XMVECTOR dirTransformed = XMVector3Transform({ dir.x, dir.y, dir.z }, inverseWorldMatrix);
	origin = { originTransformed.m128_f32[0], originTransformed.m128_f32[1], originTransformed.m128_f32[2] };
	dir = { dirTransformed.m128_f32[0], dirTransformed.m128_f32[1], dirTransformed.m128_f32[2] };

	//Draw debug ray
	m_raycastModel->Initialize(m_D3D, origin, { origin.x + dir.x * 100.0f, origin.y + dir.y * 100.0f, origin.z + dir.z * 100.0f });

	//Call AABB test for origin, ray transformed to OBB-space
	return TestAABBIntersection(lb, rt, origin, dir, dist);
}

MouseRaycastResult RaycastToModel(ModelClass* const model)
{
	//Go to [-1, 1] coordinates
	float x = GetCurrentMousePosition().first;
	float y = GetCurrentMousePosition().second;
	if (x > 1.0f)
		x = 1.0f;
	else if (x < -1.0f)
		x = -1.0f;

	if (y > 1.0f)
		y = 1.0f;
	else if (y < -1.0f)
		y = -1.0f;

	float pointX, pointY;
	XMMATRIX projectionMatrix, viewMatrix, inverseViewMatrix;
	XMVECTOR direction;

	pointX = x;
	pointY = y;

	m_D3D->GetProjectionMatrix(projectionMatrix);
	pointX = pointX / projectionMatrix.r[0].m128_f32[0];
	pointY = pointY / projectionMatrix.r[1].m128_f32[1];

	m_Camera->GetViewMatrix(viewMatrix);
	inverseViewMatrix = XMMatrixInverse(nullptr, viewMatrix);

	direction = {
		(pointX * inverseViewMatrix.r[0].m128_f32[0]) + (pointY * inverseViewMatrix.r[1].m128_f32[0]) + inverseViewMatrix.r[2].m128_f32[0],
		(pointX * inverseViewMatrix.r[0].m128_f32[1]) + (pointY * inverseViewMatrix.r[1].m128_f32[1]) + inverseViewMatrix.r[2].m128_f32[1],
		(pointX * inverseViewMatrix.r[0].m128_f32[2]) + (pointY * inverseViewMatrix.r[1].m128_f32[2]) + inverseViewMatrix.r[2].m128_f32[2]
	};

	const XMFLOAT3 dir = { direction.m128_f32[0], direction.m128_f32[1], direction.m128_f32[2] };
	return MouseRaycastResult{m_Camera->GetPosition(), dir };
}

XMFLOAT3 GetMinBounds(ModelClass * const model) const
{
	return{ model->GetBounds().minX + model->GetPositionXYZ().x, model->GetBounds().minY + model->GetPositionXYZ().y, model->GetBounds().minZ + model->GetPositionXYZ().z };;
}

XMFLOAT3 GetMaxBounds(ModelClass * const model) const
{
	return{ model->GetBounds().maxX + model->GetPositionXYZ().x, model->GetBounds().maxY + model->GetPositionXYZ().y, model->GetBounds().maxZ + model->GetPositionXYZ().z };;
}

Brief explanation - I got ray pure direction (not inverse, not normalized) from RaycastToModel. Origin is simply camera world-space position.
Then origin point and ray direction are transformed by inverse world matrix in TestOBBIntersection. lb/rt are equal to GetMinBounds and GetMaxBounds.

It still works for AABB in any position. OBB only for model position in (0, 0, 0). Camera origin (ray origin) might be doesn't matter.

Commented lines - scale is always (1, 1, 1) so that's identity matrix, we don't care. Uncommenting position multiplication makes even AABB not to work and OBB even for (0, 0, 0).

There's more there than I can easily sort out, but I'll point out one thing. XMVector3Transform() treats the input vector as a point (with w = 1), which probably isn't what you want for the direction vector. XMVector3TransformNormal() may do what you want there (in other words, use XMVector3Transform() for the origin and XMVector3TransformNormal() for the direction).

I suspect that may be an issue, but it may not be the only issue. There are many places that picking or raycasting (or any similarly complex geometric algorithm) can go awry, and there could easily be other issues at play here. But, I'd at least give the XMVector3Transform()/XMVector3TransformNormal() suggestion a try.

I can think of some other things that might make it easier to get this sorted out, but I'll refrain from offering them unless asked, as they're somewhat tangential and may not constitute the kind of feedback you're interested in.

@Zakwayda That's exactly where the problem was. Using XMVector3TrassformNormal for vector (not point) solved the problem. Thank you for all of your help. You're great!

For all of you looking for final solution:

XMFLOAT3 GetMinBounds(ModelClass * const model) const
{
	return{ model->GetBounds().minX, model->GetBounds().minY, model->GetBounds().minZ };
}

XMFLOAT3 GetMaxBounds(ModelClass * const model) const
{
	return{ model->GetBounds().maxX, model->GetBounds().maxY, model->GetBounds().maxZ };
}

MouseRaycastResult RaycastToModel(ModelClass* const model)
{
	//Go to [-1, 1] coordinates
	float x = GetCurrentMousePosition().first;
	float y = GetCurrentMousePosition().second;
	if (x > 1.0f)
		x = 1.0f;
	else if (x < -1.0f)
		x = -1.0f;

	if (y > 1.0f)
		y = 1.0f;
	else if (y < -1.0f)
		y = -1.0f;

	XMMATRIX projectionMatrix, viewMatrix;

	//Transform by projection matrix
	m_D3D->GetProjectionMatrix(projectionMatrix);
	x = x / projectionMatrix.r[0].m128_f32[0];
	y = y / projectionMatrix.r[1].m128_f32[1];

	//Create inverse matrix
	m_Camera->GetViewMatrix(viewMatrix);
	const XMMATRIX inverseViewMatrix = XMMatrixInverse(nullptr, viewMatrix);

	//Find ray direction in world space
	const XMVECTOR direction = {
		(x * inverseViewMatrix.r[0].m128_f32[0]) + (y * inverseViewMatrix.r[1].m128_f32[0]) + inverseViewMatrix.r[2].m128_f32[0],
		(x * inverseViewMatrix.r[0].m128_f32[1]) + (y * inverseViewMatrix.r[1].m128_f32[1]) + inverseViewMatrix.r[2].m128_f32[1],
		(x * inverseViewMatrix.r[0].m128_f32[2]) + (y * inverseViewMatrix.r[1].m128_f32[2]) + inverseViewMatrix.r[2].m128_f32[2]
	};

	//Origin is equal to camera position in world space;
	//Dir is transformed, not-normalized, not-inversed ray
	const XMFLOAT3 dir = { direction.m128_f32[0], direction.m128_f32[1], direction.m128_f32[2] };
	return MouseRaycastResult{ m_Camera->GetPosition(), dir };
}

bool GraphicsClass::TestAABBIntersection(XMFLOAT3 lb, XMFLOAT3 rt, XMFLOAT3 origin, XMFLOAT3 dir, float& distance)
{
	assert(lb.x <= rt.x);
	assert(lb.y <= rt.y);
	assert(lb.z <= rt.z);

	const XMFLOAT3 dirfrac = { 1.0f / dir.x, 1.0f / dir.y, 1.0f / dir.z };

	const float t1 = (lb.x - origin.x)*dirfrac.x;
	const float t2 = (rt.x - origin.x)*dirfrac.x;
	const float t3 = (lb.y - origin.y)*dirfrac.y;
	const float t4 = (rt.y - origin.y)*dirfrac.y;
	const float t5 = (lb.z - origin.z)*dirfrac.z;
	const float t6 = (rt.z - origin.z)*dirfrac.z;

	const float tmin = max(max(min(t1, t2), min(t3, t4)), min(t5, t6));
	const float tmax = min(min(max(t1, t2), max(t3, t4)), max(t5, t6));

	// if tmax < 0, ray (line) is intersecting AABB, but the whole AABB is behind us
	if (tmax < 0)
	{
		return false;
	}

	// if tmin > tmax, ray doesn't intersect AABB
	if (tmin > tmax)
	{
		return false;
	}
	distance = tmin;
	return true;
}

bool GraphicsClass::TestOBBIntersection(ModelClass* model, XMFLOAT3 origin, XMFLOAT3 dir, XMFLOAT3 lb, XMFLOAT3 rt, float & dist)
{
	//Create inverse world matrix
	XMMATRIX worldMatrix = XMMatrixIdentity();
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixScaling(model->GetScale().x, model->GetScale().y, model->GetScale().z));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationX(model->GetRotation().x * 0.0174532925f));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationY(model->GetRotation().y * 0.0174532925f));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationZ(model->GetRotation().z * 0.0174532925f));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, XMMatrixTranslation(model->GetPosition().x, model->GetPosition().y, model->GetPosition().z));
	const XMMATRIX inverseWorldMatrix = XMMatrixInverse(NULL, worldMatrix);

	//Transform origin point and direction ray vector
	const XMVECTOR originTransformed = XMVector3Transform({ origin.x, origin.y, origin.z }, inverseWorldMatrix);
	const XMVECTOR dirTransformed = XMVector3TransformNormal({ dir.x, dir.y, dir.z }, inverseWorldMatrix);
	origin = { originTransformed.m128_f32[0], originTransformed.m128_f32[1], originTransformed.m128_f32[2] };
	dir = { dirTransformed.m128_f32[0], dirTransformed.m128_f32[1], dirTransformed.m128_f32[2] };

	//Draw debug ray
	m_raycastModel->Initialize(m_D3D, origin, { origin.x + dir.x * 100.0f, origin.y + dir.y * 100.0f, origin.z + dir.z * 100.0f });

	//Call AABB test for origin, ray transformed to OBB-space
	return TestAABBIntersection(lb, rt, origin, dir, dist);
}

Glad to hear you were able to get it working ?