DXT normal map compression

You should read Understanding BCn Texture Compression Formats from Nathan Reed.

It should clear your questions. The only thing to add is that, before we had BC5 (which was only introduced in DX10; unless you used the ATI hacks); we used the technique you linked above: DXT5 and store the components in the green and alpha channel.

The alpha channel in DXT5 is encoded exactly the way BC5 does for its components. However the RGB components are not. They're encoded like in BC1, which delivers lower quality for our purposes. The Green channel is used over the Red and Blue because it has one more bit of precision.

This means BC5 beats DXT5 in terms of quality because the latter wastes precision on unused Red & Blue channels.

Is the internal ATI/NVIDIA format adding Green channel bytes and how many? Because it is the same Green/Alpha packing the only difference could be that they do something like DTXNormal = [4][8][4][Alpha].

Are you taking about the end of the tutorial where they mention ATI2/3DC? As posted above, this format is also known as BC5 - so you'll be able to find info on it under that name.
It doesn't store 4 channels, only 2!
So you get 8bit green & 8bit alpha, without wasting bits on red/blue like you do with the DXT5 trick.

The way this works with the shader is, the channels are called Luminace and Alpha (not green and alpha). When you sample from the texture, the shader receives float4(L,L,L,A) - Luminace is copied into R, G & B, which is handy because as mentioned in the tutorial, it's then compatible with he same shader code as "DXT5-swizzle" normal maps.