Hey, I had already implemented AABB raypicking using slab intersection, but I couldn't get it working with rotation to create OBB raypicking.

bool TestAABBIntersection(XMFLOAT3 lb, XMFLOAT3 rt, XMFLOAT3 origin, XMFLOAT3 dirfrac, float& distance)
{
	assert(lb.x <= rt.x);
	assert(lb.y <= rt.y);
	assert(lb.z <= rt.z);

	const float t1 = (lb.x - origin.x)*dirfrac.x;
	const float t2 = (rt.x - origin.x)*dirfrac.x;
	const float t3 = (lb.y - origin.y)*dirfrac.y;
	const float t4 = (rt.y - origin.y)*dirfrac.y;
	const float t5 = (lb.z - origin.z)*dirfrac.z;
	const float t6 = (rt.z - origin.z)*dirfrac.z;

	const float tmin = max(max(min(t1, t2), min(t3, t4)), min(t5, t6));
	const float tmax = min(min(max(t1, t2), max(t3, t4)), max(t5, t6));

	// if tmax < 0, ray (line) is intersecting AABB, but the whole AABB is behind us
	if (tmax < 0)
	{
		return false;
	}

	// if tmin > tmax, ray doesn't intersect AABB
	if (tmin > tmax)
	{
		return false;
	}
	distance = tmin;
	return true;
}

bool TestOBBIntersection(ModelClass* model, XMFLOAT3 origin, XMFLOAT3 dir, XMFLOAT3 lb, XMFLOAT3 rt, float & dist)
{	
	XMMATRIX worldMatrix = XMMatrixIdentity();
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationX(model->GetRotation().x * 0.0174532925f));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationY(model->GetRotation().y * 0.0174532925f));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationZ(model->GetRotation().z * 0.0174532925f));
	worldMatrix = XMMatrixInverse(NULL, worldMatrix);

	const XMVECTOR originTransformed = XMVector3Transform({ origin.x, origin.y, origin.z }, worldMatrix);
	const XMVECTOR dirTransformed = XMVector3Transform({ dir.x, dir.y, dir.z }, worldMatrix);
	origin = { originTransformed.m128_f32[0],originTransformed.m128_f32[1],originTransformed.m128_f32[2] };
	dir = { dirTransformed.m128_f32[0], dirTransformed.m128_f32[1], dirTransformed.m128_f32[2] };

	return TestAABBIntersection(lb, rt, origin, dir, dist);
}

What I am doing is multiplying ray origin and ray direction by inverse rotation matrix and then perform Ray-AABB test in OBB-space. It works only for 0, 90 and 180 degrees rotation. Where might be a problem?

Here are some things I noticed looking over your code:

• I don't know what all your variable names mean, and I haven't tried to fully analyze the functions for correctness, but are you sure your ray vs axis-aligned box test is implemented correctly? Have you tested it with a variety of input parameters?
• The math you're doing and the name 'dirfrac' seem to suggest that TestAABBIntersection() expects some 'prepared' data related to the direction vector, but in TestOBBIntersection() you just seem to be passing in the direction vector without any such preparation.
• You appear to be applying an Euler-angle rotation. If the Euler-angle conventions you're using in this code don't match the conventions used elsewhere, you could get erroneous results.
• I don't see you addressing the model's position anywhere in your code.

Maybe something there will be helpful.

you should use the reciprocal of the dir, not the frac

@Zakwayda

• I believe that ray-AABB is correctly implemented. I was already using it for a month or so. I have setup you can see on screen with model raypicking and axis arrows to move model arround. It work for different models and different positions and different camera angles.
  lb stands for left bottom (min bounds); rt stands for right top (max bounds); origin is camera position in world space.
• That's my fault of bad naming. It should also be dirfrac. That's what passed down.

MouseRaycastResult RaycastToModel(ModelClass* const model)
{
	//Go to [-1, 1] coordinates
	float x = GetCurrentMousePosition().first;
	float y = GetCurrentMousePosition().second;
	if (x > 1.0f)
		x = 1.0f;
	else if (x < -1.0f)
		x = -1.0f;

	if (y > 1.0f)
		y = 1.0f;
	else if (y < -1.0f)
		y = -1.0f;

	float pointX, pointY;
	XMMATRIX projectionMatrix, viewMatrix, inverseViewMatrix, worldMatrix, translateMatrix, inverseWorldMatrix;
	XMVECTOR origin, rayOrigin, direction, rayDirection;

	pointX = x;
	pointY = y;

	m_D3D->GetProjectionMatrix(projectionMatrix);
	pointX = pointX / projectionMatrix.r[0].m128_f32[0];
	pointY = pointY / projectionMatrix.r[1].m128_f32[1];

	m_Camera->GetViewMatrix(viewMatrix);
	inverseViewMatrix = XMMatrixInverse(nullptr, viewMatrix);

	direction = {
		(pointX * inverseViewMatrix.r[0].m128_f32[0]) + (pointY * inverseViewMatrix.r[1].m128_f32[0]) + inverseViewMatrix.r[2].m128_f32[0],
		(pointX * inverseViewMatrix.r[0].m128_f32[1]) + (pointY * inverseViewMatrix.r[1].m128_f32[1]) + inverseViewMatrix.r[2].m128_f32[1],
		(pointX * inverseViewMatrix.r[0].m128_f32[2]) + (pointY * inverseViewMatrix.r[1].m128_f32[2]) + inverseViewMatrix.r[2].m128_f32[2]
	};

	origin = { m_Camera->GetPosition().x, m_Camera->GetPosition().y, m_Camera->GetPosition().z };

	m_D3D->GetWorldMatrix(worldMatrix);
	translateMatrix = XMMatrixTranslation(model->GetPosition().x, model->GetPosition().y, model->GetPosition().z);
	worldMatrix = XMMatrixMultiply(worldMatrix, translateMatrix);

	inverseWorldMatrix = XMMatrixInverse(nullptr, worldMatrix);

	rayOrigin = XMVector3TransformCoord(origin, inverseWorldMatrix);
	rayDirection = XMVector3TransformNormal(direction, inverseWorldMatrix);

	rayDirection = XMVector3Normalize(rayDirection);

	const XMFLOAT3 dirfrac{ 1.0f / rayDirection.m128_f32[0], 1.0f / rayDirection.m128_f32[1], 1.0f / rayDirection.m128_f32[2] };
	const XMVECTOR dirfracNormalized = XMVector3Normalize({ dirfrac.x, dirfrac.y, dirfrac.z });
	const XMFLOAT3 dir = { dirfracNormalized.m128_f32[0], dirfracNormalized.m128_f32[1], dirfracNormalized.m128_f32[2] };

	return MouseRaycastResult{ {origin.m128_f32[0], origin.m128_f32[1], origin.m128_f32[2] }, dir };
}

I am finding ray by clicking mouse on screen and finding its direction. What I return is normalized inversed direction.

• I am using Euler-angle rotation for every model operations in my code, I believe
• lb, rt are min-max bounds in world-space. I take bounds in model-space and add current model position to it to transform to world-space

Does it make sense or am I mistaken on any part?

@Krypt0n

I am using dir returned in the code above. Is it right?

Yes, the reciprocal is correct. Why are you naming it frac? it's a bit confusing

(the rest of the ray-box intersection code looks correct to my eyes).

I was using many tutorials and forgot to rename some parameters after iterating on it.

Do you mean ray-AABB or ray-OBB? I have no idea why it doesn't work for all rotations, just 0/90/180 degrees.

40 minutes ago, komilll said:

I am using Euler-angle rotation for every model operations in my code, I believe

Generally speaking you also have to make sure Euler-angle conventions match from one place to another. I'm not saying they don't in your case, just that it can be a source of errors.

Quote

lb, rt are min-max bounds in world-space.

That was my guess, but e.g. 'lb' for left-bottom seemed a little confusing because it seems to suggest 2-d, whereas this is 3-d. Something like 'min' and 'max' (which you of course referred to above) would probably be more typical naming.

Quote

What I return is normalized inversed direction.

What's the purpose of normalizing the inverse direction? And is it the normalized inverse (rather than the original direction vector) that you're transforming by the model transform later? Not saying what you're doing is wrong - it's just not immediately obvious to me what all is going on (and there's too much code for me to analyze it fully at the moment), so I'm just curious if you can elaborate on that part of things.

I will check rotation operation using Euler-angle to make sure there aren't any errors.

Quote

What's the purpose of normalizing the inverse direction?

I think there isn't any. I was just testing some code and I'm pretty sure that normalized or not, it gives the same results. So I think you can ignore that, as it is just inversed non-normalized direction.

40 minutes ago, komilll said:

I think there isn't any. I was just testing some code and I'm pretty sure that normalized or not, it gives the same results. So I think you can ignore that, as it is just inversed non-normalized direction.

I do still wonder about the normalization, but I'd have to know more about the rest of the code to guess at what effect it might have, so I'll move on from that.

Something else that might be worth mentioning is that the slab test can be vulnerable to error if one or more direction components is zero or has very small magnitude. So we'd be talking about division by zero or numbers with small magnitude in this statement:

const XMFLOAT3 dirfrac{ 1.0f / rayDirection.m128_f32[0], 1.0f / rayDirection.m128_f32[1], 1.0f / rayDirection.m128_f32[2] };

You may be relying on language- or environment-specific behavior here, but I just thought I'd mention it in case you hadn't thought about it.

What I'm more curious about though is this. Here:

bool TestOBBIntersection(ModelClass* model, XMFLOAT3 origin, XMFLOAT3 dir, XMFLOAT3 lb, XMFLOAT3 rt, float & dist)

For 'dir', what are you passing in? The inverted (and normalized) direction vector? The original unmodified direction vector? Or something else?

@Zakwayda You had good point about what's actually passed in 'dir'. I created raycast debug line and found out where actual error is. Thank you!

I was returning dir as 1.0f / dir and then transforming it. I changed it to pass dir without inversion, transform it to OBB space and then inverse it and OBB works fine. However I found out, that it works only for location (0, 0, 0). Do you know what might be a problem?