Terrain tessellation and frustum culling

You can't cull the geometry being generated. After you've culled via GeoClipmapping or GeoMipmapping, you will inevitably end up with triangles partly or fully off the screen. Tessellation may increase the number of triangles that are off-screen, but they are discarded early in the pipeline so you just take the minor loss.


L. Spiro

You are right, it's only draw a full quad grid and you offset the y on the domain shader after the tesselation pass, domain shader is basically the vertex shader after tesselation pass.
You can frustum culling this tesselation using a check in the tesselation pass in the hull shader like that :

bool AABBBehindPlaneTest(in float3 center, in float3 extents, in float4 plane)
{
  float3 n = abs(plane.xyz);
  float r = dot(extents, n);
  float s = dot(float4(center, 1.0f), plane);
  return (s + r) < 0.0f;
}

bool AABBOutsideFrustumTest(in float3 center, in float3 extents, in float4 frustumPlanes[6])
{
  for(int i = 0; i < 6; ++i)
  {
    if(AABBBehindPlaneTest(center, extents, frustumPlanes[i]))
    {
      return true;
    }
  }
  return false;
}

PATCH_OUTPUT PatchFunc(InputPatch<HS_INPUT,4> patch, uint patchID : SV_PrimitiveID)
{
  PATCH_OUTPUT pt;
  float3 Min = float3(patch[2].Position.x, patch[0].BoundsY.x, patch[2].Position.z);
  float3 Max = float3(patch[1].Position.x, patch[0].BoundsY.y, patch[1].Position.z);
  float3 boxCenter  = 0.5f*(Min + Max);
  float3 boxExtents = 0.5f*(Max - Min);
  if(AABBOutsideFrustumTest(boxCenter, boxExtents, FrustumPlanes))
  {
    pt.EdgeTess[0] = 0.0f;    
    pt.EdgeTess[1] = 0.0f;
    pt.EdgeTess[2] = 0.0f;
    pt.EdgeTess[3] = 0.0f;
    pt.InsideTess[0] = 0.0f;
    pt.InsideTess[1] = 0.0f;
    return pt;  
  }
  else
  {
    // Tesselation pass
  }
}

You are right, it's only draw a full quad grid and you offset the y on the domain shader after the tesselation pass, domain shader is basically the vertex shader after tesselation pass.
You can frustum culling this tesselation using a check in the tesselation pass in the hull shader like that :

bool AABBBehindPlaneTest(in float3 center, in float3 extents, in float4 plane)
{
  float3 n = abs(plane.xyz);
  float r = dot(extents, n);
  float s = dot(float4(center, 1.0f), plane);
  return (s + r) < 0.0f;
}

bool AABBOutsideFrustumTest(in float3 center, in float3 extents, in float4 frustumPlanes[6])
{
  for(int i = 0; i < 6; ++i)
  {
    if(AABBBehindPlaneTest(center, extents, frustumPlanes[i]))
    {
      return true;
    }
  }
  return false;
}

You use this function in the "patchconstantfunc" :

PATCH_OUTPUT PatchFunc(InputPatch<HS_INPUT,4> patch, uint patchID : SV_PrimitiveID)
{
  PATCH_OUTPUT pt;
  float3 Min = float3(patch[2].Position.x, patch[0].BoundsY.x, patch[2].Position.z);
  float3 Max = float3(patch[1].Position.x, patch[0].BoundsY.y, patch[1].Position.z);
  float3 boxCenter  = 0.5f*(Min + Max);
  float3 boxExtents = 0.5f*(Max - Min);
  if(AABBOutsideFrustumTest(boxCenter, boxExtents, FrustumPlanes))
  {
    pt.EdgeTess[0] = 0.0f;    
    pt.EdgeTess[1] = 0.0f;
    pt.EdgeTess[2] = 0.0f;
    pt.EdgeTess[3] = 0.0f;
    pt.InsideTess[0] = 0.0f;
    pt.InsideTess[1] = 0.0f;
    return pt;  
  }
  else
  {
    // Tesselation pass
  }
}

So, basically you always send the full grid and you can frustum test to avoid to do the tesselation pass if the part of the grid is not visible.

Makes sense. The 'BoundsY' for a quad is calculated on the CPU once by reading the heightmap and making an AABB?

Can this be used in conjunction with other terrain tessellation alghorithms (like http://www.frostbite.com/wp-content/uploads/2013/05/adaptive_terrain_tessellation.pdf)?

Yes you create the boundsY which is a vector2(minY, maxY) on the creation of the buffers reading the heightmap.
Your link is about the density map, I don't see why it would breaks the density map algorithm.
Remark : Bounds also help for the ray intersection to bypass part of terrain to be faster.