If I understand correctly, the DXT5_NM format for normal maps specifies a DXT5 image that has the original red/x component in the alpha channel, and the red and blue channels zeroed out (or set to a constant value). The z component of the normal is meant to be reconstructed in the shader as z = sqrt(1 - x*x - y*y).
The RXGB formal on the other hand, is a DXT5 image with x in the alpha channel, the red channel empty, and the original green and blue channels left in tact. This improves the compression quality for the x component and doesn't require any reconstruction.
Does all this sound correct? I ask because I'm playing around with the "nvcompress" utility (part of the new GPU accelerated nvidia texture tools) and it's interpretation of these formats seems to be different. First of all in the readme file it lists this as a support output format:
" -bc3n BC3 normal map format (DXT5n/RXGB)".
Which sounds like they think both of these formats are the same. Then, when I actually convert a file to this format, and look at the individual channels in GIMP, there appears to be data in the red, green and alpha channels and nothing in the blue. So this doesn't even agree with what I thought was the accepted definition of DXT5_NM.
So, does anyone know what's going on here? Is there just a loose definition for all these formats, or is nvcompress wrong? It's not that I can't work with the format it's creating after having examined the output, but it seems like they ought to follow the established conventions for these formats (assuming there are any).
EDIT:
Examining the layers in GIMP more closely it looks like there isn't actually data in the red channel, it's just filled with 100% red. GIMP was showing it to me modulated by the alpha channel so it looked like normal map data. So I guess that answers this question. The only thing left is the difference (if any) between DXT5_NM and RXGB.
