Dynamic Shader Compilation

i was thinking of implementing this in my own engine (dynamic shader construction) but decided to give it a miss, as when i move to shader 3.0 model nvidia and others recommend creating an uber shader which would definately make things easier :)

Quote:

Original post by supagu i was thinking of implementing this in my own engine (dynamic shader construction) but decided to give it a miss, as when i move to shader 3.0 model nvidia and others recommend creating an uber shader which would definately make things easier :)

I agree to an extent. My "uber shader" will still be dynamically created though. I have tons of effects that may not even be used in the entire game. Compiling them into one large shader would be a rather significant waste. This system is also for my 2.0 and 1.1 fallback however, so I can't rely on the "uber shader" until 3.0 is a part of my minimum system requirements. Thanks for the reply.

directX december update provides a set of classes for combining shader fragments at run time, correct? would this be the sort of thing your after?

Quote:

Original post by RipTorn directX december update provides a set of classes for combining shader fragments at run time, correct? would this be the sort of thing your after?

Essentially, yes. I plan on implementing my own version however, as my engine is cross API. I also think I can do slightly better than the D3DX interfaces, mostly due to the assumptions I can make about my current engine's setup. Thanks for the reply.

The basic design for my material system was originally inspired by Yann's material system that has been previously discussed her over multiple infamous threads. Essentially, I just define the attributes of an object from a rather high level (e.g. Diffuse, Specular, Reflective, Emissive, etc... ). The decision as to which built-in shader fragments are to be used is dependant upon the attributes defined in the material script. However, the effects are scaled based on runtime performance and host hardware.

As an example, the engine will dynamically scale the number of lights that are used to light a surface, it will also drop bones with small weights in some animations if cost of animation is determined to be the current bottle-neck. I can go into more detail if necessary, but to broaden the topic, I am really just attempting to design a scalable and efficient, generic material/rendering system, and would like to hear from anyone that has something to say. Think of this as a brainstorming thread.

currently I'm still in the 'theoretical in my head' stage of planning how I go about a materials/shader system.

I havn't read any of yahns threads about his system, so I'm not sure how similar/different my ideas are...


Anyway. what I'm currently planning (and feel is going to be much easier now I've moved to .net, but thats off topic [wink]) - is to have some sort of shader/material editing gui... The idea being there are are four main parts to the interface..
In effect, A vertex shader editor, a pixel shader editor, a list of optional attributes, and of course a preview. Sortof.

What I want to do, but havn't quite figured out the best way to do it, is some way to move away from the C-style functions for current shaders, and move more towards just a block of higher level code.

How this becomes useful, is to do with the list of 'optional attributes'...
Ideally, how I see things happening (for example), is that there will be a list of texture units here, and you can then set what each is.
But more so, not only do you set that they are 2d textures, but also what each channel represents. So the RGB might be your surface normal in tangent space, and the alpha might be a height map, with an optional scaler.
Therefore, instead of using tex2D (which I'd probably hack around to prevent it working) - you would use texture1(...), or normal1(...) or such. That way, how texture1() actually works can be changed depending on the unit. For example, there might be (in a really simple case) a tick box saying 'use parallax mapping' - or for later on, actual recursive ray tracing into the texture (like some papers are currently proposing).. another example is that normal() might automatically convert from texture space to world space

Other attributes may just be generic things, like the view vector... but this too could be taken a step further.. For example an attribute may be added that is the view vector in tangent space.
The idea is to do all the more complex and geometry specific maths in the background of the shader (such as tangent spaces) - and hide how data is passed in, etc. The programmer could maybe then specify if something is calculated per-pixel or per-vertex...

This, of course, doesn't say how I'm going to go about doing lighting, and frankly I'm still thinking about that. How I do it is going to have a big impact on the overall shape of the rest of the code, so I've got a while to go before I narrow it down and finalize something. But I'd most like to seperate material/lighting calculation from the shader code, so lighting may just end up being another attribute for the pixel shader...

On top of that, I'd like to be able to figure out an easy way to chain these shaders together. Ie, for pixel shaders have intemediate steps, ie, this shader + that shader, ot this*that. And simple add-on vertex shaders that simply fudge the output or input (for example, an animation vertex shader).
Being able to selectivly combine these at run time would obviously be a boon too. For an easy example, like in half-life 2 how a burnt enemy ends up all charred.

So.. as you can no doubt see, I have a long way to go in designing this system.
But I've got enough time so it's all good. I get the feeling that dealing with the limitations of pixel/vertex shaders is going to be the biggest hurdle (currently aiming at shader model 2.0 and above only)

Thats what I'm planning anywho.

Quote:

Original post by Sages In the process of redesigning my engine's material system I realized that my new design broke the hacked dynamic shader compiler I had set up. I am planning on having several functions that are just combined to create complete specialized shaders. This should decrease the total number of shaders I would need to use at runtime, which allows me to have larger batch sizes and consume less graphics memory. I was just wandering if any of you had implemented something similar, or if you had any suggestions as to what I could do instead of using this method. Comments, questions or concerns? Thanks guys

I'm not sure why you think this will help you decrease batch size - as it doesn't actually decrease the number of shaders you need to use from the APIs perspective. Considering the huge amount of overhead in creating a shader, creating new ones on a frame by frame basis wouldn't be advisable.

However, I've seen such systems used on (upcoming) comercial engines, but I wouldn't recommend it - nor would I expect this to be the way it is handled in the future. A single shader will be complex enough to handle a variety of options, and be able to reconfigure on a per-triangle basis for much better batching.

For today's hardware, however, an alternative is to offline compile the permutations, and have the handy when needed. This is the approach that Half-Life 2 took. All that dynamic compiling in the runtime does for you is decrease the amount of memory you need to store your shaders.

Hello Sages,
I have been using a dynamic shader compilation scheme that preprocesses cg files at runtime to make them match input vertex types, ps/vs concections, optional parameters, etc...
It behaves a bit like a C++ template system, where you define generic parameters such as constants, types, and functions.
It works quite well, and allows to write very few shaders that will be bloeated into specialized shaders.
Even if ps3.0 is goind to help a bit, you cannot use overly generic shaders for performance reasons.

Above that shader systems, we use a material dispatcher that behaves a bit like an adaptive .fx file and creates multipass shaders from material definitions and environmental parameters (light count, fog of war, ...)

My point is a shaders generator, or "meta-compiler", is a must-have.

Sorry for the delay in this, I didn't have access to a computer while I was on vacation. I'm back now though (obviously), and would like to continue this topic.

Quote:

Original post by EvilDecl81 Considering the huge amount of overhead in creating a shader, creating new ones on a frame by frame basis wouldn't be advisable.

Sorry, I wasn't very clear in my explanation. I'm not actually going ot be creating the shaders on a per frame basis. It would only be at level load and/or on demand. Essentially, I create the optimal shaders when needed, but I would cache them, so on the next frame the shader wouldn't need to be re-created.

Quote:

Original post by janos My point is a shaders generator, or "meta-compiler", is a must-have.

I agree to an extent. I'm essentially planning to do what you seem to have implemented. Is your system completely data driven or would you have to compile to create a new material. Would you have to re-compile to create a new factor (like a new constant, or vertex input)? If you would like to elaborate on your design I would enjoy and appreciate a more in depth conversation. Please contact me through one of the various methods mentioned on my site (in the signature).

I'm on my way to getting the design of the system worked out, I'll probably start implementing after I have had a few more ideas from you people. Please note that all ideas are appreciated regardless of your expertise on the subject.

Dynamic shader generator could be a great tool but as the way of developing / using shader will change a lot in the next 2 years I think making a simple system that just do the job is a better move for now ...

...but it's just an hobbyist opinion :)

I posted a thread about something quite similar a few months ago (clicky), didn't start working on the code yet, just some on-paper work. and some suggestions were made in the thread that might interest you, even if it's not exactly like what you wanted to do, it has some similarities (it fully generates shader code from BRDF/BSSRDF approximation functions, or other kind of lighting functions, and uses an AST to optimize the general lighting function for the specific shader characteristics, and to ease the linkage between physics and graphics shaders (or any other kind of separate shader module that needs to modify the shader being currently generated)).

Quote:

Original post by Sages

Any high-level shader language having macroses will do the job.
Write explicit/distinct shaders for different materials (diffuse, diffuse+specular, etc.) and prepare each of them for dynamic conditions like shadowing, lights, etc. (that will be managed by the SG) + a reference (in the shader or in the material) for a shader-types, like 'shadowmap' shader, 'glow' shader, etc.
Then the art pipeline will export a mesh with a shader for mesh's material, and SH will setup that shader to behave (of course, if the shader is 'smart enough') correctly, like compiling it with special macro-definitions.
So, each shader on-disk will spawn a family of shaders that you must select from at runtime (or your SG).