Energy conservation of diffuse term

It's because the energy conservation considers the integration across the hemisphere, while you're talking about a single ray.

To explain in simple terms, imagine a rectangle of 5x1 in size where the height is the amount of energy going out of from a single ray, each sub-millimeter of the width is a single ray (infinite rays means infinitely small increments across the width), and the area of the rectangle is the total energy going out (the sum of all rays)

Now with this analogy, the area of the rectangle is the one that must be equal to one, and your question would be like asking why do you need to change your formula if the height at any given point is 1. The thing is, we need the area to be 1, not the height.

If height = 1 and width = 5; then the rectangle area is not energy conserving because it's equal to 5x1 = 5.

Thus, you need to divide your output by 5 (or multiply by 0.2) from each ray so that the area is now 1 --> 5 * 1 * 0.2 = 1

This is a simple analogy in 2D terms using a basic rectangle. BRDF is exactly the same thing but over a hemisphere in 3D space. To mathematically obtain the right factor you need to be familiar with limits of integration.

An example of analytically obtaining the right factor can be seen here; and you can look at an even harder example from Fabian Giesen.

Note that integration is an advanced topic, usually taught in University-level courses. Some integrals are so hard to solve that we scratch our head multiple times, or just turn to Monte Carlo solutions (a fancy word for saying try multiple times until you start approaching the result)

...constant of lambert's BRDF should that not already be energy conserving ? Since we only define the diffuse color in between [0-1]. Do We need to do this because we combine it with a specular brdf ?

IIRC, for lambert to be energy conserving the constant/colour must be from 0 to 1/pi.
Anyway, say you've got an energy conserving diffuse brdf, and an energy conserving specular brdf -- individually they each reflect <=100% of the input energy, so when combined together they reflect <=200%...
The fresnel term you mention is used to blend the two brdf's together so that when they're combined, the total reflectance is <=100% again.

I see, maybe it was just confusing to me since in real-time rendering 3rd edition they wrote:

"The requirement for a BRDF to be energy conserving is simply that R(l) be no greater than one for all possible values of l"

and not specifically the sum of all possible values. So I thought it is enough for every single ray to be between 0 and 1 for the BRDF to be energy conserving.

That's an interesting phrase from RTR, I might have to re-read their explanation, maybe they suggest the correct meaning through the context around that sentence? Do you have a page number?

Yeah, AFAIK, it should read something like: "that R(I) integrated over the hemisphere will be no greater than I".

Yeah it's at the bottom of page 227.

They do show that R(l) is equal to the BRDF times the cosine of the angle between n and l, integrated over all possible outgoing directions in the hemisphere. Still that sentence confused me.

Just throwing this out there, I think the problem is more that most of the classical BRDFs used in rendering are designed to be 'one-shot' in the sense they don't offer a breakdown for diffuse and specular terms. When a theoretical graphics programmer is using said BRDFs as intended, there's no energy competition between the two terms, and our 'diffuse Fresnel' problem goes away. In fact, the reason we have these models more has to do with the rather crappy indirect lighting situation most games find themselves in-- we have to get a little more bang for our buck from existing point lights, etc. so we have a sort of multilayered BRDF approach that's designed to show detail not immediately in the specular area of influence.

EDIT: Yeah, I wrote this on reduced sleep and in a rush to get to work; I'm not sure where I was going with the whole GI/multilayer BRDF thing. See MJP's post below for a nice explanation of what I think I was originally going for

Huh? The lack of GI is not the reason we break the BRDFs into Diffuse & Specular terms.

Specular formulas simulate lighting coming with a particular polarity pattern while Diffuse lighting simulate lighting coming with polarization patterns on a different axis.

This is a real life phenomenon unrelated to Indirect/Global Ilumination.

In general the combination of a diffuse and specular is just to simulate materials that are actually composed of multiple layers consisting of different reflective properties. The classic example would be a coated plastic, where you have a very smooth specular reflection on the surface layer while underneath you have a layer that's much rougher with subsurface scattering that causes the reflected light to take on the albedo color. You can generalize this to having a material being composed of a sum of BRDF's, instead of a more rigid diffuse/specular relationship. For instance on cars you typically have a clear coat on top, and a more metallic surface underneath that is still very much view-dependent which necessitates another specular lobe. In all cases you just need to be careful in how you set up the interactions between the two BRDF terms if you want to maintain energy conservation.

Huh? The lack of GI is not the reason we break the BRDFs into Diffuse & Specular terms.

Specular formulas simulate lighting coming with a particular polarity pattern while Diffuse lighting simulate lighting coming with polarization patterns on a different axis.

This is a real life phenomenon unrelated to Indirect/Global Ilumination.

I always thought that the traditional difference between specular and diffuse terms was simply that the specular term is only the light that is perfectly reflected from the light source, rather than being reflected over some distribution of angles (although I've also colloquially heard specular be considered any light reaching the camera dependent on the angle between the camera and the reflector as well; in this case diffuse would be limited to "ideal" diffuse, and certain kinds of materials like Minnaert shading, which give off light in certain directions but not dependent on the angle of entry, would still be considered "specular"). My recollection of how polarity works seems to suggest that it isn't sufficient to make that distinction, at least in all cases.

In all cases you just need to be careful in how you set up the interactions between the two BRDF terms if you want to maintain energy conservation.

I'm a bit curious about this. Since real-life materials are perfectly capable of absorbing light and then re-emitting it as energy other than visible light, what does it really mean to have a material that doesn't conserve energy? I guess materials that absorb light in especially strange, angle-sensitive ways are probably rare, but it seems plausible that some arrangement of microfacets could potentially be described by materials which are obviously "wrong."

I guess materials that absorb light in especially strange

Note that energy conserving means that the material's output is <= 100%, not == 100%. A material that absorbs light and doesn't show it back is very common
What is not common though, is a material that can output more than incoming light.

I'm a bit curious about this. Since real-life materials are perfectly capable of absorbing light and then re-emitting it as energy other than visible light, what does it really mean to have a material that doesn't conserve energy? I guess materials that absorb light in especially strange, angle-sensitive ways are probably rare, but it seems plausible that some arrangement of microfacets could potentially be described by materials which are obviously "wrong."

Fluorescense is a great example of materials that can output more energy than its incoming energy. That's because they add an additional source of power other than incoming light (i.e. chemical reactions). Note that conservation of energy (as in the universe as a whole) is preserved, since they're just releasing energy they had been storing in some other form; but for the given time an observer was watching, he or she will have observed that outgoing light > incoming light.
Burning ashes could be another example. The extra light emitted by the ashes comes from heat.
In games, we just call those emissive materials and don't usually care about "correctness". Though this is starting to change (since Physically Based BRDFs + HDR + GI = We can use real life values, like sun's measured power, a bulb's measured power, etc)

I guess materials that absorb light in especially strange

Note that energy conserving means that the material's output is <= 100%, not == 100%. A material that absorbs light and doesn't show it back is very common
What is not common though, is a material that can output more than incoming light.

I'm a bit curious about this. Since real-life materials are perfectly capable of absorbing light and then re-emitting it as energy other than visible light, what does it really mean to have a material that doesn't conserve energy? I guess materials that absorb light in especially strange, angle-sensitive ways are probably rare, but it seems plausible that some arrangement of microfacets could potentially be described by materials which are obviously "wrong."

Fluorescense is a great example of materials that can output more energy than its incoming energy. That's because they add an additional source of power other than incoming light (i.e. chemical reactions). Note that conservation of energy (as in the universe as a whole) is preserved, since they're just releasing energy they had been storing in some other form; but for the given time an observer was watching, he or she will have observed that outgoing light > incoming light.
Burning ashes could be another example. The extra light emitted by the ashes comes from heat.
In games, we just call those emissive materials and don't usually care about "correctness". Though this is starting to change (since Physically Based BRDFs + HDR + GI = We can use real life values, like sun's measured power, a bulb's measured power, etc)

Good point; I meant to include materials that emit more light than they receive as well as the inverse but neglected to do so, and I realize that both cases do not change the total amount of energy, only the total amount of light (or visible light).

In any case I agree that the core requirement is that the integration of light emitted over all angles is generally no greater than the amount of light received; I do also think that in practice, when people refer to energy-conserving materials, they typically are also suggesting a more holistic, physically-based approach. For instance, one can imagine satisfying the core requirement (light out <= light in) with outdated, ad hoc, and generally "bad" shading methods just by scaling or clamping the parameters and/or output, but people would probably not be especially impressed with such an "energy-conserving" material that met the definition in only the narrowest sense.

Specular formulas simulate lighting coming with a particular polarity pattern while Diffuse lighting simulate lighting coming with polarization patterns on a different axis.

When a light wave (polarised or not) hits a surface, it's reflected and refracted. Specular simulates the reflected parts, and Diffuse simulates the refracted parts.

Light that's polarised in different ways is still partially reflected and partially refracted (so: requires both diffuse and specular formulas).

When a light wave (polarised or not) hits a surface, it's reflected and refracted. Specular simulates the reflected parts, and Diffuse simulates the refracted parts.
Light that's polarised in different ways is still partially reflected and partially refracted (so: requires both diffuse and specular formulas).

I wouldn't really call it "refracted", it's more a notion of incoherent scattering. In other words, "specular" means "not scattered", whereas refraction *is* a form of local specular reflection. This distinction isn't particularly important in everyday computer graphics but there are some physical effects that apply to specularly reflected light, but not scattered light (and vice versa), so I think the notion of coherence between reflected rays for specular/diffuse light is really the distinctive feature we are trying to quantize here.

Anyway, on topic, yes, the point is that the total energy reflected off a surface patch is less than (or equal to) the total energy falling on it. We don't care about "energies" of particular light rays, they could be incredibly large. If you shine a laser in your eye - don't do this, by the way - it's going to be super bright, with an intensity exceeding hundreds of watts per steradian, yet that laser has a finite amount of power (e.g. half a watt) being converted into light. Now widen the beam.. and it doesn't look as bright anymore. So what we're really interested in is how much power is radiated from a given surface *in every direction*, not just a single one, which involves summing up (or integrating) over the sphere or hemisphere of directions.

I'm a bit curious about this. Since real-life materials are perfectly capable of absorbing light and then re-emitting it as energy other than visible light, what does it really mean to have a material that doesn't conserve energy? I guess materials that absorb light in especially strange, angle-sensitive ways are probably rare, but it seems plausible that some arrangement of microfacets could potentially be described by materials which are obviously "wrong."

That's because your typical BRDF doesn't handle those sorts of effects. Basically, most computer graphics renderers assume that the flow of light in a scene has entered a steady state, that is, it is constant and unchanging as long as the geometry remains the same, which leaves no obvious way to simulate fluorescence and other time-dependent effects (it can be done, though, especially when ray tracing). In this sense the notion of "energy conservation" dictated by those renderers can be somewhat limited physically.

When a light wave (polarised or not) hits a surface, it's reflected and refracted. Specular simulates the reflected parts, and Diffuse simulates the refracted parts.
Light that's polarised in different ways is still partially reflected and partially refracted (so: requires both diffuse and specular formulas).

I wouldn't really call it "refracted", it's more a notion of incoherent scattering. In other words, "specular" means "not scattered", whereas refraction *is* a form of local specular reflection.

Slight nit-pick, but isn't it "refracted/diffuse" in that the incoming light enters the surface before coming back out of the surface, thereby picking up the colour of the underlying object - as opposed to the reflected light which never enters the surface and thereby remaining the pure light colour?

When a light wave (polarised or not) hits a surface, it's reflected and refracted. Specular simulates the reflected parts, and Diffuse simulates the refracted parts.
Light that's polarised in different ways is still partially reflected and partially refracted (so: requires both diffuse and specular formulas).

I wouldn't really call it "refracted", it's more a notion of incoherent scattering. In other words, "specular" means "not scattered", whereas refraction *is* a form of local specular reflection. This distinction isn't particularly important in everyday computer graphics but there are some physical effects that apply to specularly reflected light, but not scattered light (and vice versa), so I think the notion of coherence between reflected rays for specular/diffuse light is really the distinctive feature we are trying to quantize here.

Not to beat a dead horse, but in a strict sense I think I actually agree with Hodgman. I think "specular" reflection is when a ray is reflected perfectly off of a surface (meaning in only in precisely one direction) whereas any other type of "reflection" is refraction -- at least insofar as the ray/rays leave in multiple directions (or alternately in a nond-eterministic way over a distribution).

Like I mentioned, this definition of specular excludes "rough specular" type materials (but not really, as they are ultimately just an approximation of many different microfacets, each of which are perfect mirrors). It's also common to use "specular" to refer to any material where the perceived luminance varies by viewing angle, in which case diffuse refers only to "ideal" diffuse materials; I think this is closer to the definition you're using, especially if we also add to the "diffuse" label materials where the perceived luminance is allowed to vary by viewing angle, but not as a function of the angle that light enters (Minnaert, etc.).

I wouldn't really call it "refracted", it's more a notion of incoherent scattering. In other words, "specular" means "not scattered", whereas refraction *is* a form of local specular reflection.

If you're looking at light hitting the surface as a wave that's encountering a change in IOR, then refraction is correct. Fresnel's laws say that some of the wave is reflected and some is refracted -- those are the two options.
For the reflected portion of the light wave, the angle of incidence and angle of reflection are always equal, which makes it a "specular reflection". This light can't be discoloured by the surface due to absorption, because it never enters the surface.

The rest of the wave (the part that isn't reflected) only has one other option - to be refracted (i.e. transmitted through the IOR boundary). The angle of transmission is bent slightly (meaning it's not "specular"), though this is irrelevant to us when rendering solids.
Once this light has been refracted/transmitted into the opaque object, then scattering/diffusion/absorption can occur. It bounces around inside the object, mostly being absorbed but with parts of it managing to make it back to the surface again to be re-emitted in some random direction.

So it's also valid to say that diffuse lighting equations account for scattered/diffuse light, but scattered/diffused light is refracted light (the only way that light can get into the diffuser, assuming it's not emissive, is to be transmitted/refracted through it's surface).

The Lambert diffuse model ignores specular reflection (all light is refracted/transmitted into the surface), it's then assumed to be perfectly diffused (every possible exitance angle has equal probability), and if you use a white "diffuse texture", then it's assumed that no absorption occurs and that all of this refracted light eventually makes it back out.
Likewise, the Phong specular model ignores diffusion/refraction (all light is reflected off the surface without entering it at all).
To account for all of the light that hits the surface - the reflected portion and the refracted/transmitted portion, you need to combine both these BRDF's together (or use another BRDF that does account for both "diffuse" and "specular" light).

During the diffusion process inside the surface, it's also usually assumed that the polarisation of this light also becomes randomly distributed (it becomes "unpolarized" light, or has mixed polarisation). Whereas on the other hand, reflected light doesn't really interact with the surface, so it's assume that it's polarization is essentially unchanged. Most renderers ignore polarization though, and simply assume that all light is an equal mix of all possible polarizations.

The Lambert diffuse model ignores specular reflection (all light is refracted/transmitted into the surface), it's then assumed to be perfectly diffused (every possible exitance angle has equal probability), and if you use a white "diffuse texture", then it's assumed that no absorption occurs and that all of this refracted light eventually makes it back out.

And we come full circle!

L_out(y) = Integral( BRDF(x,y) * Visibility(x) * L_in(x) * cos(angle between x and normal) * dx ) over hemisphere

Energy conservation it's just

Integral( L_out(x) * dx ) <= Integral( L_in(x) * dx ) (here out and in flux density values are)

You can substitute constant L_in and Visibility equaling to 1. And in the case of diffuse term BRDF is just albedo.

It'll be

Integral( L_out(x) * dx ) <= Integral( L_in(x) * dx ) (condition of not emission)

Integral( Integral( BRDF(x,y) * Visibility(x) * L_in(x) * cos(x,n) * dx ) * dy ) <= Integral( 1 * dx )

Integral( BRDF(x,y) * 1 * 1 * cos(x,n) * dx ) * Integral( 1 * dy) <= 2 * Pi

Integral( BRDF(x,y) * cos(x,n) * dx ) * Integral( 1 * dy) <= 2 * Pi

diffuse BRDF is just constant

Integral(a * cos(x,n) * dx ) * 2 * Pi <= 2 * Pi

a * Integral(cos(x,n) * dx ) <= 1

a * Pi <= 1

a <= 1 / Pi

it's constraint for albedo

Like this way you can check different BRDF.

And also when you use direct light source in Integral over hemisphere its integration is changed by computing by single ray via delta function.

L_in(x) = delta(x_{light directoin} - x) * c, where c is flux density of light source, being measured in Watt / m² units, and delta func is measured steradian^-1.

Then

L_out(y) = Integral( BRDF(x,y) * Visibility(x) * L_in(x) * cos(x,n) * dx )

L_out(y) = Integral( BRDF(x,y) * Visibility(x) * delta(x_{light directoin} - x) * c * cos(x,n) * dx )

L_out(y) = BRDF(x_{light directoin},y) * Visibility(x_{light directoin}) * c * cos(x_{light directoin},n)

where Visibility(x_{light directoin}) is filtered sample from shadow map (for example).