🔒 Locked 🎮 Unity

"Picking" the closest mesh problem

Started by mcguile25 Apr 9, 2006 at 3:28 PM 4 replies 2.6k views

Original Post

Apr 9, 2006 at 3:28 PM

Hi all. I've got a problem with my "picking" code in DirectX. It works perfectly fine on any number of meshes _until_ I scale one of the meshes up or down. Here are some pictures and a description of what's happening in each (the mouse cursor is kind of hard to see, sorry bout that :D).
Here are two meshes. Obviously the yellow one was "picked" and is highlighted yellow. Both meshes are their original size, orientation and placement.

The next two screenshots have the floor mesh scaled by a factor of 5 all around. You can see that when selecting the vertical wall mesh, it still wants to select the ground mesh (the scaled mesh).

If I try this code with one model that is scaled, the picking works correctly, it only seems to not pick the right one when there are multiple meshes with one or more being scaled. Now, in my code, I am choosing the picked mesh by the closest distance to the cursor position. Here is some of my code. Is there anything glaringly wrong here?


bool CCollision::PickModel( LPDIRECT3DDEVICE9 displayDevice, LPD3DXBASEMESH mesh, D3DXMATRIXA16 *worldMatrix,
						    float *distance, D3DXVECTOR2 *cursorPos, D3DXVECTOR3 *hitPos )
{
	D3DXMATRIXA16 matProj, matView;
	displayDevice->GetTransform( D3DTS_PROJECTION, &matProj );
	displayDevice->GetTransform( D3DTS_VIEW, &matView );

	D3DXVECTOR3 v;
	v.x =  ( ( ( 2.0f * cursorPos->x ) / DISPLAY_SCREEN_WIDTH ) - 1 ) / matProj._11;
	v.y = -( ( ( 2.0f * cursorPos->y ) / DISPLAY_SCREEN_HEIGHT ) - 1 ) / matProj._22;
	v.z =  1.0f;

	D3DXMATRIX m;
	D3DXVECTOR3 rayOrigin,rayDir;

	D3DXMatrixInverse( &m, NULL, &matView );

	// Transform the screen space pick ray into 3D space
	rayDir.x = v.x*m._11 + v.y*m._21 + v.z*m._31;
	rayDir.y = v.x*m._12 + v.y*m._22 + v.z*m._32;
	rayDir.z = v.x*m._13 + v.y*m._23 + v.z*m._33;
	rayOrigin.x = m._41;
	rayOrigin.y = m._42;
	rayOrigin.z = m._43;

	// Use inverse of matrix
	D3DXMATRIX matInverse;
	D3DXMatrixInverse(&matInverse,NULL,worldMatrix);

	// Transform ray origin and direction by inv matrix into this object's model space
	D3DXVECTOR3 rayObjOrigin,rayObjDirection;

	D3DXVec3TransformCoord(&rayObjOrigin,&rayOrigin,&matInverse);
	D3DXVec3TransformNormal(&rayObjDirection,&rayDir,&matInverse);
	D3DXVec3Normalize(&rayObjDirection,&rayObjDirection);

	bool hasHit = false;
	float hitU, hitV = 0.0f;
	DWORD faceIndex = 0;

	D3DXIntersect(mesh, &rayObjOrigin, &rayObjDirection, reinterpret_cast<BOOL*>(&hasHit), &faceIndex, &hitU, &hitV, distance, NULL, NULL);

	// determine the hit position
	if ( hasHit )
	{
		*hitPos = rayObjOrigin + (rayObjDirection * (*distance));
		D3DXVec3TransformCoord( hitPos, hitPos, worldMatrix );
	}

	return hasHit;
}

jollyjeffers

Apr 9, 2006 at 4:29 PM

I don't think the problem is going to be in the code you posted, particularly if you say it works fine when theres only one mesh being tested...

It's more likely that your usage of that function is at fault. How are you iterating through the objects? are you break'ing from the loop as soon as you detect a collision?

What you probably need to do is collect a set of collisions and then take the nearest of those. The mouse coordinate defines a ray in 3D space, of which there could be many intersections.

Returning the distance parameter instead of (or as well as) a bool is the easiest way of implementing this. The nearest (and correct) intersection will have the lowest returned distance.

hth
Jack

mcguile25

Apr 9, 2006 at 4:43 PM

Thanks for the reply. First off, that function returns true if a collision occured or false if it didn't. Also, you pass in a pointer to a distance variable that gets filled in with the ray distance if a collision occured. Sorry I wasn't clear about that. Here is my loop code. Basically, if a collision occured, I check if it's the first match or less than the current closest distance and if so, set my highlighted index variable to 'i'. I don't break out of the loop prematurely at all.

	// distance variables	D3DXVECTOR3 collisionPoint;	float closestDistance = -1.0f;	float loopDistance = 0.0f;	// -------------	// we're going to see what object the user has clicked on and	// set that as the highlighted object.  we do this by keeping track of the closest	// -------------	D3DXVECTOR3 menuCursorPos = g_gameEngine.GetPersistentData()->GetMenuCursor()->GetPos();	D3DXVECTOR2 cursorPos( menuCursorPos.x, menuCursorPos.y );	D3DXMATRIXA16 matWorld;	int i = 0;	// ** models **	for ( i = 0; i < m_models.size(); ++i )	{		m_models[ i ]->CalcWorldMatrix( &matWorld );		if ( CCollision::GetInstance()->PickModel( m_displayDevice, m_models[ i ]->GetMesh(), &matWorld, &loopDistance, &cursorPos, &collisionPoint ) )		{			// store this model if it's closer than the current closest (or we're at the first closest mesh)			if ( closestDistance == -1.0f || loopDistance < closestDistance )			{				m_highlightedObjectIndex = i;				m_highlightedObjectType = MAP_EDITOR_TYPE_MODEL;				closestDistance = loopDistance;			}		}	}

mcguile25

Apr 9, 2006 at 4:48 PM

Also, to be more clear, this picking loop works perfectly fine for any number of meshes (not just 1 by itself).

The problem arises when I scale one or more of the meshes. Then things don't pick correctly. I made sure I was transforming the ray by the inverse of each object's world matrix. Hmm..

mcguile25

Apr 9, 2006 at 6:28 PM

Aha! I fixed it and this is what I originally was suspecting:

Since the objects have different scaling factors, I guess the distance returned was in a different space because of the scaling. So I multiply the distance returned by the scaling factor so that scaled meshes have a scaled distance as well.

It seems to work fine, but it kind of feels like a hack. Eh, oh well. If anyone else comes up with anything else, please let me know, thanks!

DXnut

Apr 9, 2006 at 6:56 PM

For detecting the closest object picked, I don't use the distance returned by the D3DXIntersect function. Instead, I have a vector that has the position of the mesh in world space and the position of the camera. Then I use the D3DXVec3Length function between these two positions to determine the closest to the camera.

Before doing the D3DXVec3TransformCoord function on the ray position, the ray's position is basically the same thing as your camera's position. So you could also just save it before transforming it for the intersect test, and use the saved one for the distance calculations.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.