Tight fit shadow frustum issue

I see a bunch of issues:

• It looks like you use camera up vector to create the light view matrix. That will fail if the light direction is the same as the vector, meaning the cross product is 0. It's better to construct a consistent orthonormal basis from the light direction alone. You just need to figure out another vector that is perpendicular to the light direction.

• Are your rendering coordinates (projection matrix) left handed? The code treats Z+ as forward. If your projection matrix code doesn't match then that could cause issues.

• You shouldn't create a separate light view matrix for each cascade. That is unnecessary and may cause different pixel orientations with each cascade. Just use the orientation matrix that you start with in the first line of code. Change the position only for each cascade.

• You will have shadow stability issues when the camera moves because you don't ensure that the light view is always rendering to a consistent texel grid. To fix that you need to ensure there is always a consistent mapping from world space to texels in the shadow map. Therefore, you need to ensure that the shadow map is sized so that it can contain the cascade frustum in any orientation. To do that, you need to use the bounding sphere of the frustum to determine the units:texels scale factor, so that it is always consistent. You must also round the light-space AABB bounds to the nearest shadow texel.

• Since your light view will no longer fill the whole texture (because the scale needs to be consistent), you need to determine the UV rectangle that the light should render to within the texture, and then use that viewport when rendering the shadow map. Then, you need to use a corresponding UV matrix for the cascade when sampling the shadow map, so that the [-1,1] post-projection coordinates can be mapped to the correct UV region within the shadow map that is occupied by the current viewport for the cascade.

• General coding: You are needlessly making lots of allocations for fixed-sized arrays because you use std::vector. Memory allocation is one of the most expensive things you can do. Better to use standard arrays on the stack, or std::array if you feel pedantic about safety. Only use std::vector if you don't know the size in advance.

This is my working code. For reference, lightBasis is a 3x3 rotation matrix for the light (from local to world), which has the Z column as the negative light direction. This code uses Z- as front (OpenGL), so you may need to negate some things and swap min/max Z if you use left-handed coordinates.

// Get the maximum size of the frustum in any direction.
// This must be consistent over time to avoid shimmering.
const Float32 frustumSphereRadius = camera->getViewVolumeBoundingSphereRadius( cameraNear, cameraFar );

// Determine the shadow map size based on the frustum size and required texel density.
const Float32 frustumSize = 2.0f*frustumSphereRadius;
const Float32 inverseFrustumSize = 1.0f / frustumSize;
const Float32 unitsPerTexel = frustumSize / Float32(shadowMapSize);
const Float32 texelsPerUnit = Float32(shadowMapSize) * inverseFrustumSize;

// Get the corners of the camera frustum in world space.
SIMDReal4 corners[Camera::FRUSTUM_CORNER_COUNT];
camera->getViewVolumeCorners( corners, cameraNear, cameraFar );

// Transform corners into light space and compute the local AABB.
SIMDReal4 lightSpaceCorners[Camera::FRUSTUM_CORNER_COUNT];
SIMDAABBR lightSpaceAABB = (lightSpaceCorners[0] = lightBasisInverse * corners[0]);
for ( Index i = 1; i < Camera::FRUSTUM_CORNER_COUNT; i++ )
	lightSpaceAABB |= (lightSpaceCorners[i] = lightBasisInverse * corners[i]);

// Round the AABB center to the nearest texel and transform to world space.
const SIMDReal4 lightSpaceCenter = unitsPerTexel * math::floor( lightSpaceAABB.getCenter()*texelsPerUnit + 0.5f );
const SIMDReal4 worldSpaceCenter = lightBasis * lightSpaceCenter;

// Remove center offset from the AABB.
lightSpaceAABB -= lightSpaceCenter;

// Round the AABB to the nearest texel, including Z coordinate.
lightSpaceAABB.min = unitsPerTexel * math::floor( lightSpaceAABB.min*texelsPerUnit + 0.5f );
lightSpaceAABB.max = unitsPerTexel * math::floor( lightSpaceAABB.max*texelsPerUnit + 0.5f );

// Determine the viewport box covered by the frustum. This defines a UV region in the texture.
const SIMDReal4 viewportMin = lightSpaceAABB.min*inverseFrustumSize + 0.5f;
const SIMDReal4 viewportMax = lightSpaceAABB.max*inverseFrustumSize + 0.5f;
lightViewport.min = Vector2f( viewportMin[0], viewportMin[1] );
lightViewport.max = Vector2f( viewportMax[0], viewportMax[1] );

// Create an orthographic camera to render the light's view.
lightView.setProjectionType( ProjectionType::ORTHOGRAPHIC );
lightView.setPosition( worldSpaceCenter );
lightView.setOrientation( lightBasis );
lightView.setOrthoBounds( lightSpaceAABB );

const Vector2f uvScale = lightViewport.getSize() * 0.5f;
const Vector2f uvCenter = lightViewport.getCenter();
const SIMDMatrix4R uvMatrix(
		SIMDReal4( uvScale.x, 0.0f, 0.0f, 0.0f ), 
		SIMDReal4( 0.0f, uvScale.y, 0.0f, 0.0f ), 
		SIMDReal4( 0.0f, 0.0f, 0.5f, 0.0f ), 
		SIMDReal4( uvCenter.x, uvCenter.y, 0.5f, 1.0f ) );
// The matrix transforming from world space to light view space, then projected, then transformed to the viewport in UV space.
cascadeMatrices[cascade] = uvMatrix * lightView.getProjectionMatrixSIMD() * lightView.getTransformMatrixInverseSIMD();