I am using RT shader (DXR) to generate g-buffers with code below. There was no problems with smaller scene (pink room from cwyman tutorial - http://cwyman.org/code/dxrTutors/tutors/Tutor5/tutorial05.md.html ). However, when I moved to bigger scene (Sun Temple by Epic Games - https://developer.nvidia.com/ue4-sun-temple ), I've started noticing problems with texture sampling. Please compare textures at the end of this post. The greater the distance to world origin (0, 0, 0), the more jaggy artifacts appear and quality becomes worse.

At the beginning, I was suspecting acceleration structures. But skybox quality is becoming worse, so it's probably unrelated to AS.

   inline void GenerateCameraRay(uint2 index, uint2 dimensions, float4x4 projectionToWorld, inout  float3 origin, out float3 direction)
    {
    	float2 xy = index + float2(0.5f, 0.5f); // center in the middle of the pixel.
    	float2 screenPos = (xy / (float2) dimensions) * 2.0 - 1.0;
    
        // Invert Y for DirectX-style coordinates.
    	screenPos.y = -screenPos.y;
    
        // Unproject the pixel coordinate into a ray.
    	float4 world = mul(float4(screenPos, 0, 1), projectionToWorld);
    
    	world.xyz /= world.w;
    	direction = normalize(world.xyz - origin);
    }

    [shader("raygeneration")]
    void RayGen()
    {
    	float3 rayDir;
    	float3 origin = g_sceneCB.cameraPosition.xyz;
        
        // Generate a ray for a camera pixel corresponding to an index from the dispatched 2D grid.
    	GenerateCameraRay(DispatchRaysIndex().xy, DispatchRaysDimensions().xy, g_sceneCB.projectionToWorld, origin, rayDir);
    
    	RayDesc ray = { origin, 1e-4f, rayDir, 1e+38f };
    	RayPayload payload = { 0 };
    	TraceRay(SceneBVH, RAY_FLAG_NONE, 0xFF, 0, 1, 0, ray, payload);
    }
        
    [shader("miss")]
    void Miss(inout RayPayload payload)
    {
    	float3 rayDir = WorldRayDirection();
    	rayDir.z = -rayDir.z;
    
    	RTOutputAlbedo[DispatchRaysIndex().xy] = float4(skyboxTexture.SampleLevel(g_sampler, rayDir, 0).rgb, 0);
    	RTOutputNormal[DispatchRaysIndex().xy] = float4(0, 0, 0, 0);
    	RTOutputSpecRoughness[DispatchRaysIndex().xy] = float4(0, 0, 0, 0);
    }

@swiftcoder Do you think that FP precision might cause such artifacts?

Can I “move world” around camera without rebuilding AS?

komilll said:
Can I “move world” around camera without rebuilding AS?

The world positions of instances are contained in the TLAS, and that is designed to be easily/swiftly built, so technically you can rebuild that every few frames from scratch, with every instance in it moved appropriately, to minimize visible precision errors. ( The nVidia articles I've come across flat out say that do not bother with trying to update an existing TLAS, it's easier just to rebuild it, what you would gain with the shorter build time, would be lost when tracing through it, because an AS update usually produces a less optimal structure to trace through. )

To the original problem: This doesn't look like a DXR problem, more like a math problem with the ray direction calculation. You calculate the ray direction, pass it to TraceRay, and then ask it back in the miss shader with WorldRayDirection. Now I have no idea what WorldRayDirection does inside, but it would be strange if it did anything more than just giving back the exact vector you passed to TraceRay. ( And you can test this if you want, instead of writing the result UAVs in the miss shader, you can put a bool flag in the payload that is initialized to false, and that the miss shader can set to true, to indicate that a miss happened. After TraceRay you read it back from the payload, and if it indicates a miss, you do that same sampling and UAV writing, but using the original rayDir as the input. If there is no difference in the resulting image, then the ray direction calculation is what is busted in some way. ) That would mean that the problem manifests it self already when you look up a sample from the skybox texture with the calculated ray direction, indicating that the problem lies in the math/inputs around that area.

@LandonJerre WorldRayDirection() is giving same effects as using boolean payload. Problem was that perspective matrix had very short Z_NEAR and Z_FAR values (0.01f and 200.0f). By changing it to much higher range (10.0f, 20000.0f), problem with jaggies and artifacts disappears. Thank you for help.