Decoding grayscale intensities

Do you need to determine which sub-pixel samples correspond to the non-zero portion of your coverage values? If so, you will need to do some analysis of the neighbors of each pixel, probably similar in nature to what is done for MLAA.

What exactly do you need to do with the coverage values???

For coverage all samples are originally 0 or 1. How subpixel samples are used to create the final value for a pixel depends on the antialiasing method. With standard MSAA, a box filter is used to combine samples which weights them all instantly. So if you had 4 MSAA samples 1 passed sample would result in a value of 0.25, 2 would result in 0.5. Other filter types will weight subsamples differently based on their position within the pixel, and might even factor in subsamples from neighboring pixels. At any rate, you're not going to have enough information from just the resolved pixel value to determine the original values of subsamples, except for in a few trivial cases.

Cheers guys, the application I'm using it with is getting the alpha channel data from an image, how the anti-aliasing is computed is unknown to me, I then use this data for use with the dead reckoning distance transform which uses source-destination 2d points to get the distance, this works well for binary images but for certain effects, outlining for example, the none full coverages need to be taken into account (for good results at least)

I've already tried using pixels t,b,l,r of the current pixel with some success but obviously it's a flawed hack which didn't quite work, however, depending on how much analysis I'd need to do for each none full coverage, might outdo the speed gains using dead reckoning.

Some paper on how Photoshop does it would be the holy grail for me. ;)

Ok thanks guys I've managed to parp my brain fart and all is clear, the solution is to use sobel, well, I'm playing around with different kernels to improve accuracy, but it's a job done.