🔒 Locked

Reinhard's tone mapping operator

Started by g0nzo Aug 2, 2006 at 4:10 PM 9 replies 22.4k views

Original Post

Aug 2, 2006 at 4:10 PM

Hi! I'm trying to implement Reinhard's tone mapping operator, but I get strange results and I'm not really sure how it should look like: - normally you calculate average luminance of the scene (world luminance - Lw) - then you map to the middle-grey zone: L(x,y) = key/Lw * pixelLuminance(x,y) - then you scale all pixel luminances to range 0-1: Lscaled(x,y)=L(x,y)/(L(x,y) + 1) This way you obtain scaled luminance of the pixel. And now what? I tried just to multiply pixel color by this value, but I get strange results (bright pixels get clamped to 0 very quickly, dark are very dark until *really* high key is set).

In Goodnight et al. "Interactive Time-Dependent Tone Mapping Using Programmable Graphics Hardware" they use something different: color.rgb = pow((color.rgb/luminance), alfa) * Lscaled, where alfa is in range 0.4-0.8 and luminance is pixel luminance. Results are quite good:

In HDRFormats demo the pixel lumianance is not used at all, only the color is used: color.rgb = key / Lw * color.rgb; color.rgb = color.rgb + / (1.0f + color.rgb); and results are also good (that's not fair [smile]).

In Reinhard's own source code (C), he uses some strange RGB->XYZ and XYZ->RGB color space conversion (one of these XYZ values is luminance, I don't know what other 2 are), but I can't figure out what he's doing exactly.

I even tried to divide scaled lumiance of the pixel by non-scaled pixel luminance before multiplying it with the pixel color and the result where more similar to Reinhard's:

All images were taken using exactly same values of average luminance, key and white point (measured by Reinhard's program, so scaling luminance on GPU doesn't have any impact on them). I don't understand from Reinhard's code how he's doing color space conversions, so I can't figure out how he's changing lumianance to RGB values. Is there a correct way to do it? When I was trying to implement Reinhard's operator I hoped to obtain similar results, but it's not so easy [smile] BTW. Does anyone know if the Goodnight's GPU implementation of Reinhard's "dodging and burning" tone mapping operator is available somewhere? [EDIT] I've found RGB -> XYZ conversion explained: http://en.wikipedia.org/wiki/CIE_1931_color_space (however values used in Reinhard's code are different). But is this conversion really necessary? If I have scaled pixel luminance and its RGB color, what is the correct (or simple, but giving accurate results) way to obtain new RGB values? Right now I'll probably use Goodnight's method as it gives good results. [Edited by - g0nzo on August 3, 2006 4:26:14 AM]

stephanh

Aug 3, 2006 at 9:04 AM

The conversion from RGB to CIE XYZ and then to CIE Yxy separates the luminance (Y) information from the color/chromaticity one (x,y) and you can adjust the luminance without affecting the color.

You get from CIE XYZ (a) to Yxy (b) by:

b.Y = a.Y;
b.x = a.X / (a.X + a.Y + a.Z)
b.y = a.Y / (a.X + a.Y + a.Z)

Havent looked a reinhards TMO recently, dont know if you have to scale the luminance or replace it by your new L channel. But you do that in Yxy space and then convert back to RGB via XYZ.

btw. color faq is a good resource for such things: http://www.poynton.com/ColorFAQ.html

regards,
stephan

nooan

Aug 3, 2006 at 10:36 AM

Thanks Stephan for your wisdom and congratulations for your outstanding project

g0nzo

Aug 3, 2006 at 11:05 AM

Thanks for the answer and for the link!

However is this RGB <-> XYZ conversion really necessary to obtain accurate results? I haven't seen this conversion in any realtime Reinhard's tone mapping implementation. Maybe it's just that without this conversion results are "good enough", so nobody is using it?

Anyway thanks again, I'll try to implement it and check if performance gets really crappy.

[EDIT]
Here's the HLSL implementation of Reinhard's simple tone mapping operator with RGB<->XYZ<->Yxy conversions:

float4 PS_Reinhard02( VSOutput input ) : COLOR{  // Get color of the pixel  float4 color = tex2D( sceneTextureSampler, input.TexCoords );  	  // Get lumianance values from 1x1 lumianance texture  float4 luminanceSample = tex2D( luminanceTextureSampler, float2( 0.5f, 0.5f ));  // Average scene luminance is stored in R channel  float avgLuminance = luminanceSample.r;   // RGB -> XYZ conversion  const float3x3 RGB2XYZ = {0.5141364, 0.3238786,  0.16036376,                             0.265068,  0.67023428, 0.06409157,                             0.0241188, 0.1228178,  0.84442666};				                      float3 XYZ = mul(RGB2XYZ, color.rgb);    // XYZ -> Yxy conversion  float3 Yxy;  Yxy.r = XYZ.g;                            // copy luminance Y  Yxy.g = XYZ.r / (XYZ.r + XYZ.g + XYZ.b ); // x = X / (X + Y + Z)  Yxy.b = XYZ.g / (XYZ.r + XYZ.g + XYZ.b ); // y = Y / (X + Y + Z)      // (Lp) Map average luminance to the middlegrey zone by scaling pixel luminance  float Lp = Yxy.r * exposure / avgLuminance;                         // (Ld) Scale all luminance within a displayable range of 0 to 1  Yxy.r = (Lp * (1.0f + Lp/(whitePoint * whitePoint)))/(1.0f + Lp);    // Yxy -> XYZ conversion  XYZ.r = Yxy.r * Yxy.g / Yxy. b;               // X = Y * x / y  XYZ.g = Yxy.r;                                // copy luminance Y  XYZ.b = Yxy.r * (1 - Yxy.g - Yxy.b) / Yxy.b;  // Z = Y * (1-x-y) / y      // XYZ -> RGB conversion  const float3x3 XYZ2RGB  = { 2.5651,-1.1665,-0.3986,                              -1.0217, 1.9777, 0.0439,                                0.0753, -0.2543, 1.1892};  color.rgb = mul(XYZ2RGB, XYZ);  color.a = 1.0f;  return color;}

The funny thing is that it works (if it didn't I'd have a huge problem, because I don't know how to debug shaders [smile]). I didn't noticed any performance decrease (however it's obviously a lot more to calculate), so it's quite strange that it's not used in any HDR sample I've seen (i.e. in DX SDK).

And I'll repeat my question from the first post:
Is Goodnight's implementation of Reinhard's tone mapping local operator available somewhere?

[Edited by - g0nzo on August 3, 2006 4:05:21 PM]

LeGreg

Aug 4, 2006 at 10:57 PM

Quote:
Original post by g0nzo
Is there a correct way to do it?

Actually there isn't.
It's all subject to interpretation.

All that is to understand is that you've got a destructive compression; so all those tone mapping schemes are about control of where you want to lose your information (or where you want to keep it).

Doing RFinal = RScaled/(RScaled + 1) for each component, has several advantages:
one is that the destination intensity for each channel fits nicely into your [0.1] interval without adjustment, the only adjustment necessary is to scale the source to alter the perceived lightness (the so called exposure). Second is that there is no saturation happening before quantization (and saturation or clamping can cause some nasty looking artefacts).
The problem with that approach on the other hand is that each channel is handled separately (which is also great for performance), but it also decreases color saturation (all your resulting images look desaturated, grayish in the higher intensity which some people think it's ok because it looks more like film, and you can enhance color saturation/contrast as a post process).

Now if you consider that you applied this formula only to the luminance, you on the other hand haven't decided what each color component will be so you still have an extra degree of freedom. One natural approach would be to preserve exactly the ratio between each color channel, so that gives you a simple equation to solve to get the final color. There are problems associated with that, for example if the luminance is well between 0 and 1 intensity, allowing large differences between channel at high intensities means that some color channel can exceed 1, which will result in clamping and saturation artefacts (which may be ok or not, depending on the look that you're looking for).

Or you can try an approach in between. Or you can post scale the second approach to get rid of artefacts, or you can post scale the first approach to increase color saturation at the expense of new artefacts. Or you can use a totally different tone mapping function (a sigmoid function). Or you can allow for crosstalk between channels, etc. etc. There is no end to what you can or would want to do. It's all about what look you want for your image.

LeGreg

g0nzo

Aug 6, 2006 at 7:05 AM

Thanks for the explanation, it's really helpful!

Quote:

All that is to understand is that you've got a destructive compression; so all those tone mapping schemes are about control of where you want to lose your information (or where you want to keep it).

So i.e. y=x/(x+1) function compresses high luminances more, while sigmoid function compresses high and low luminances more?

I won't make any more changes to it for now, as I get results similar to Reinhard's own program, which was the main goal.

Now I'll try to implement Reinhard's local tone mapping operator, but it looks much more complicated (for me and for GPU [smile]).

Do you know maybe some other than Reinhard's tone mapping operators (global or local) that are quite easy to implement on GPU?

LeGreg

Aug 6, 2006 at 2:11 PM

There are a lot of schemes, all with their advantages and inconvenients.

The ILM guys when they have to present their intermediary work, they use a very big volumetric texture (because they can), that they address using the original color as (r,g,b) = (u,v,w), scaled to the max possible luminance or clamped. This gives them the resulting argb color, so they can factor in a lot of complicated calculations in one texture read but this requires a pretty big texture to give good approximate results. There is also some magic involved using exponentiations and the like, but I'm not too sure about the details.

One easy thing to do if you want to have a more "local" operator, is to calculate the mean luminance on a more restricted area than the whole image using gaussianized weights, each pixel will then have its own exposure based on the luminance of its neighbours. You can try varying the size of the gaussian filter to see what gives you the best result for your image (it also depends on its size on the screen and the distance to the viewer from the screen!). Also you may have to do an averaging between the global one and the local one, because the local one will tend to reduce contrasts on the whole image. Also you see that if you push this to the limit and have one exposure calculated with only one pixel, you'll end up with a totally decontrasted image.

Hope that helps,
LeGreg

g0nzo

Aug 6, 2006 at 4:49 PM

Thanks again!

Quote:

One easy thing to do if you want to have a more "local" operator, is to calculate the mean luminance on a more restricted area than the whole image using gaussianized weights, each pixel will then have its own exposure based on the luminance of its neighbours.

Actually that's more or less like the Reinhard's local tone mapping works.
From Goodnight's paper about implementation of local operator on the GPU (chapters 4.2 and 4.3):

Quote:

First, the image is convolved with a set of Gaussian convolution kernels defined at multiple spatial scales, giving a set of responses Vi. Subtracting adjacent responses gives an estimate of the local contrast at multiple spatial scales.

The idea is to find "the largest area around a given pixel where no large contrast changes occur", so if the difference V_i-V_i+1is higher than a given threshold, it means that there's a high change in luminance and V_i is used as local luminance.

Here's an image that ilustrates it:

However there's one thing that I don't really understand:

In Reinhard's paper he mentiones that they start with radius of 1 pixel (which is just its luminance value) up to 43 pixels, using factor 1.6 ("Our choice of center-surround ratio is 1.6, which results in a difference of Gaussians model that closely resembles a Laplacian of Gaussian filter [Marr1982]" - unfortunately it doesn't say much to me), so they are doing it up to 8 times (1.6^8=43). Great, but 1px x 1.6 is 1.6 pixels. I'm just a newbie and maybe I just didn't understand it correctly, but how am I supposed do create a kernel from this (even if I'd round it up to 2 pixels)?

Also, using large kernels for every pixel (which would occur if the image has similar lumianance over large areas) would be probably very slow , wouldn't it be in fact faster to use FFT on the GPU?

LeGreg

Aug 6, 2006 at 5:39 PM

Of course it's too slow, but doing it on the gpu allows you to cut corners and make it fast again.

First, you don't necessarily operate on the whole resolution image. Since you calculate an approximation of luminance over large areas you can assume it varies at a low frequency and so operating on a reduced resolution and then interpolating values in between may be ok for you.

Second, if you use a separable filter you actually make the convolution calculation O(n) instead of O(n^2), which is considerably faster.
A separable filter is a convolution filter that can be divised into two separate filters, one operating horizontally and one operating vertically (making it a 1D filter that operates on each row then and each column). Not all 2D filters are separatable, but you are allowed to choose one. If you do a 1D gaussian on pixels horizontally and then a 1D gaussian on pixels vertically then you've got yourself a separable gaussian filter. This costs you an extra pass of course but sometimes it's worth the cost if your kernel operates on a large area.

Third, running on GPUs allow you to use texture filter hardware for free in addition to the computational power in the pixel shader, so as soon as your computation looks like something that can be filtered in hw try to fit it in. You can also use the anisotropic filtering capabilities if you need to decimate pixels quickly (though you cannot specify the weights there).

Hope that helps,
LeGreg

g0nzo

Aug 7, 2006 at 3:55 AM

Thanks once again.

But I'm not sure how should I implement it.

First (optional) step would be to downsample a texture using StretchRectangle or just by rendering to a smaller texture using linear filter (I'm not sure which is faster). Second, to calculalate the luminance of each pixel and store it in i.e. R32F format, so I won't have to calculate it every time (if texture lookup is faster than calculating dot product)

To calculate one gaussian convolution I need 2 passes (horizontal and vertical). I need to store results of 2 adjacent convolutions, so I can get their difference, so I need 2 buffers for this.

I should probably store the differences somewhere, so that I can just check its value at the beginning and if it's greater than a treshold, omit calculating convolutions for this pixel. But I need also to store calculated local luminance. Could I use a texture with 2 channels (G32R32F?) for this? In the green channel I would i.e store the difference and in the red one, value of local lumianance.

Maybe I could use also a texture with 2 channels instead of 2 separate buffers for storing adjacent convolutions?

If I was downsampling a texture in the first step, would I need to upsample the texture with local luminances at the end, or could I just take values from downsampled version?

Also dynamic memory allocation is not possible in shaders? So I can't just create 2 arrays for storing offsets and weights of n and m sizes and pass these values to shader? I would have to declare an array for every kernel size?

Does it all make any sense? [smile]

jollyjeffers

Aug 7, 2006 at 12:03 PM

I'm a little late to this thread and only scan-read it, so apologies if I repeat anything [smile]

Have you looked at my 'HDRPipeline' (aka 'HDRDemo') sample in the DirectX SDK (since the December '05 release)?

I implemented Reinhard's tonemapping operator as best as I could in that sample code - straight from the original research paper for the most part. I also used the two channel GxxRxxF format (where xx = 16 or 32) to store per-pixel maximum/average luminances. The other SDK samples skip the maximum luminance term (hence the simplified shader code) but I found it made quite a difference to the results...

Be careful with the filtering hardware and HDR data - only the latest generations support FP16 filtering/blending, and none (that I know of) support FP32 filtering/blending. My aforementioned sample has some code that will correctly select FP16/FP32 based on hardware support for filtering.

There was an ATI paper that I can dig up the URL for that showed ways of using the filtering hardware to increase the convolution kernel filter size. If you position your sampling offsets directly between two texels you effectively get a double sample for the cost of a single; thus you can either use 1/2 as many samples or you can double the size of your convolution kernel. Only works when you have filtering hardware, but is definitely worth the effort.

Quote:
Also dynamic memory allocation is not possible in shaders? So I can't just create 2 arrays for storing offsets and weights of n and m sizes and pass these values to shader? I would have to declare an array for every kernel size?

No, you cant do dynamic allocation in shaders. You can probably use uniform and/or #define constructs to minimize the code you have to write. When compiling the shader you can pass in a set of defines (e.g. your offset dimension) and have it generate the appropriate code.

hth
Jack

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.