A while ago I implemented both GJK and EPA. I've tested them with several numbers and as far as I could see they returned correct results. Except when they are rotated. GJK seems to return correct yes/no answers, but my guess is that the simplex itself could be incorrect while still giving correct results.

The return value is correct when the rotation does not influence the penetration/normal. For example, if the objects are on top of each other and one is rotated on the yaw axis, the correct results are returned. However, when it's rotated on the pitch, it either returns with the wrong values or gets into an infinite loop, depending on the input (I've seen both).

The full code on Github:
https://github.com/dorcsyful/Cinkes/blob/master/CinkesCollision/CGJKAlgorithm.cpp

The only thing I changed compared to the version that does not handle rotations is how the support points are calculated:

 CVector3 next = CVector3(1, 0, 0); //or whatever the current direction is
 CVector3 A = a_Object1->GetTransform().getRotation() *
        a_Object1->GetCollisionShape()->Support(a_Object1->GetTransform().getRotation().Transpose() * next)
        + a_Object1->GetTransform().getOrigin();
    CVector3 B = a_Object2->GetTransform().getRotation() *
        a_Object2->GetCollisionShape()->Support(a_Object2->GetTransform().getRotation().Transpose() *
            (next * (-1))) + a_Object2->GetTransform().getOrigin();
    CVector3 support = A - B;

I don't see any obvious problem with your support vector transformations, assuming that your matrix math library is doing what it says, and your transform basis is defined to rotate from local to world space. So, probably you have a bug somewhere else in the GJK code that only shows up with rotations.

@Aressera Is there anywhere else in the algorithm that requires special handling with rotations? Since GJK and EPA work with the Minkowski difference as opposed to the actual shapes I figured I only need to include rotation matrices in the support function. As everyone else on the internet, my GJK is also based on Casey's video and I really can't see any part that'd require multiplications. Do you maybe have a suggestion on how I can check if the simplex GJK returns is correct? Like I said it might return a correct yes/no answer but I want to check the whole simplex, so I can see if it's GJK or EPA that's broken.

dorcsyful said:
Do you maybe have a suggestion on how I can check if the simplex GJK returns is correct?

Store your minkowski vertices as point A and vector (A-B), rather than only A-B like you do now. Then, you can know the points on shape A and B in world space that make up the simplex on each object. I would try drawing the simplex on top of the colliding objects (as 2 wireframe tetrahedra), and make sure that one face of the tetrahedron includes the collision point.