material system

For example material obviously should know about material related variables in shader but there are other variables that are not related to how things are seen like matrices so which part of the game should be responsible for setting these kind of variables.

Matrices are material parameters. There have been a few discussions in the past about that and the hint was about considering parameters as opaque blobs. I cannot quite point you to the exact thread, hopefully someone will search for it.

I'd elaborate a bit on that concept though. Some special values must obviously have a special binding. I think D3D somehow has this built in (I think it's called DXSAS but never looked at it). In general, I considered my parameters a sequence of opaque blobs with an header. If the header mandated the use of an environment value - such as MVP or light info - then the blob would count 0 bytes and I would pull out the value from runtime.

Who does the pulling? Someone very close to the renderer, but not quite like it. I've called my pullers uniform fetchers.

Just to give you some input, here's my Material class (stripped of its member functions):

class Material
{
public:

enum CullFace
{
DONT_CULL,

CULL_FRONT,
CULL_BACK
};

enum DepthTest
{
DONT_TEST,

DEPTH_TEST_NEVER,
DEPTH_TEST_ALWAYS,

DEPTH_TEST_EQUAL,
DEPTH_TEST_NOT_EQUAL,

DEPTH_TEST_LESS,
DEPTH_TEST_LESS_EQUAL,

DEPTH_TEST_GREATER,
DEPTH_TEST_GREATER_EQUAL
};

enum BlendMode
{
DONT_BLEND,

BLEND_MODULATE,
BLEND_ADDITIVE,
BLEND_TRANSPARENT
};

vec4 ambient;
vec4 diffuse;
vec4 specular;

bool colorWrite;
bool depthWrite;

CullFace cullFace;
DepthTest depthTest;
BlendMode blendMode;

protected:

// contains shader program id, etc.
shared_ptr<MaterialProxy> m_proxy;

// texture layers contain textures and texture coordinate
// generators used to scroll, stretch and rotate textures
vector<shared_ptr<TextureLayer>> m_textureLayers;
};

I use a uniform buffer object to update material colors etc. efficiently, I think there's a similar concept available in DX.

There have been a few discussions in the past about that and the hint was about considering parameters as opaque blobs. I cannot quite point you to the exact thread, hopefully someone will search for it.

Here you go. This is most likely the more complete thread about material systems.

No, that was not quite the post I was referring to. It's related, seems to involve the management of the various materials but it does not seem to go much in depth about what a material is.
I think this could be slightly more in my line of thinking... but I'm not 100% sure. It seems to contain all the details.

As a side note: I would avoid the material structure above. It is adeguate for D3D7 and FFP: an extremely inflexible way to think at things now.

My best suggestion to OP is to approach this iteratively, by focusing on the needs of a particular product first. For a start, Vertex shader + pixel shader + (a subset of) ROP settings coupled with constant data will likely suffice.

First of all thanks for the advices. And here is my strategy. First I think I should change my shader variables.
Currently they are like below and I have proper ID3DX11Variable type for each them.

float3 lightDirection;
float3 lightColor;

And in order to use them as raw data with ID3DX11EffectConstantBuffer it should be like this

cbuffer LightInfo
{
float3 lightDirection;
float3 lightColor;
};

And shader class should have a list of ID3DX11EffectConstantBuffer and should know what they contain. And material class should have actual buffers for these variables. I'm not sure what is the best way to make shader class know about contents of buffers though. And another thing I'm curious about is that shader variables are filled by different parts of the game. I mean view and projection matrices come from camera class, some values come from object's class etc. And what is the best approach to update material contents?
That was my strategy and my problems . Is it a viable strategy or does it need to change?

In my engine I have a Frame Constant Buffer that is only updated once per frame and containts matrices like View, Proj, ViewProj matrices and other constants that only change once per frame... This CB is accessible to all shaders in the CB slot 0. Then I have a ActorCB that contains the constants of a actor, and the number of times it is updated = number of actors drawn, and the same goes for the other Constant Buffers

In my engine I have a Frame Constant Buffer that is only updated once per frame and containts matrices like View, Proj, ViewProj matrices and other constants that only change once per frame... This CB is accessible to all shaders in the CB slot 0. Then I have a ActorCB that contains the constants of a actor, and the number of times it is updated = number of actors drawn, and the same goes for the other Constant Buffers

Yes I was thinking about something like this but how do you know what is inside the buffers. I mean different shaders have different CBs. And some of them might not have per frame matrix data. Are you using another file that has information about CBs or is there a way to find out CB when compiling the shader.

I have a file called ShadersHeader.hlsl that looks something like this:

//Samplers
SamplerState gTriLinearSampler : register(s0);
SamplerState gTriPointSampler : register(s1);
SamplerState gAnisoSampler : register(s2);
//Constant Buffers
cbuffer cbImmutable : register(b0)
{
//Constants that never change during gameplay
}
cbuffer cbPerFrame : register(b1)
{
float4x4 gView;
float4x4 gProj;
float4x4 gViewProj;
float4x4 gInvViewProj;
float4 gEyePositionFarPlane;
}


Then I #include this header in every shader file that needs some/all of this constants. So when I update the constant buffer at slot 0/1 the constants will be available to every shader... you just have to make sure you don't override constant buffer slots.
I'm not sure how the effects framework will deal with this, but it will probably work the same way, anyway why don't you write your own effect framework? You will gain more control over the shaders and probably a performance boost.

EDIT: In Effects11 I think you can share samplers/constant buffers, and other resources, across techniques by using Effect Groups, although I'm not sure how they work...

Thanks for the reply. I'm not against writing my own effect framework but currently I don't need something like that and before I wrote a simple effect framework so I know how it works. And about my question, I don't have problems with CBs that are used by all shaders. My question is lets say there are two shaders A and B. And they have different variables in CBs. So to create corresponding variables in shader and material class I should know about what their CBs contain. And my solution to that is having xml file for every shader which contains info about shader variables. Something like this.

//SHADER A
cbuffer buffer
{
float4 a;
float3 b;
};

//xml
<variable name="a" type="float4" />
<variable name="b" type="float3" />



//SHADER B
cbuffer buffer
{
matrix a;
float b;
};

//xml
<variable name="a" type="matrix" />
<variable name="b" type="float" />


And this way I can know what it contains and its size. But I'm not sure this is the easiest and the most efficient way to do this.

You can do that, but you can also just use reflection to figure out which variables are in a constant buffer. Everything about a constant buffer can be queried with the reflection API's, you don't need your own layout scheme to find out that information.

You can do that, but you can also just use reflection to figure out which variables are in a constant buffer. Everything about a constant buffer can be queried with the reflection API's, you don't need your own layout scheme to find out that information.

Thanks! I think thats exactly what I need. And if anyone curious I found this about reflection. Also I think it might be possible to get these info with effects framework (I've just found that). And before I jump into coding is this a good approach to material and shader system. Because another question popped up in my head "How should I update buffers?" and I'm not sure about that . Because after I created buffers with the info I got from reflection (or effects framework if possible) I need to send necessary information to material class to update buffers. And obviously different materials need different information. How can I create uniform interface for sending necessary information to material class?

I keep it simpler in my case. Since you're writing a deferred shading engine, you are kind of limited in the material types you can create anyway.

I let people specify things like diffuse map, specular and gloss map, normal map, blend mode, etc...

It's easy to work with and gives me exactly what I want. Ogre3D has a much more powerful material system from what I've seen.

Also I use a bitmask depending on features in the shader. I have a map of bitmask to shader and if that shader is already loaded and compiled I use it. The bitmask also decides which #define keywords are defined when compiling the uber shaders.

Ok it is mostly finished. Here is how I did it. Hopefully someone having the same problems can get an idea and you can help me to improve it.

This is my shader class. Shader class keeps a list of ID3DX11EffectVariable and 2 variables for per frame and per object constant buffers so I can update them individually or all together if I want. And what I like about my shader class is it can create input layout automatically. If anyone needs that I can post it too. And I have a question about FreeTextures function. Since I'm using deferred renderer some of my textures are used as render targets so I need to free them before using as render targets. And here is how I do it but I don't know if this is a good way to do it. Because when I apply pass to device context doesn't it updates all of the variables?


void Shader2::FreeTextures()
{
std::map<std::string, ID3DX11EffectShaderResourceVariable*>::iterator i;
for(i = textures.begin(); i != textures.end(); i++)
{
i->second->SetResource(0);
}
ID3DX11EffectTechnique* tech = techniques[0];
tech->GetPassByIndex(0)->Apply(0, graphicsDevice->GetDeviceContext());
}



class Shader2
{
friend class Material;
public:
struct VariableContainer
{
ID3DX11EffectVariable* variable;
UINT size;
};
Shader2(void);
~Shader2(void);
void Initialize(GraphicsDevice* gDevice, std::string path);
void Release();
void SetVariable(const std::string name, void* data);
void SetVariable(const std::string name, bool data);
void SetVariable(const std::string name, float data);
void SetVariable(const std::string name, D3DXMATRIX data);
void SetTexture(std::string name, ID3D11ShaderResourceView* texture);
void SetPerFrameConstantBuffer(void* data, UINT size);
void SetPerFrameConstantBuffer(void* data, UINT offset, UINT size);
void SetPerObjectConstantBuffer(void* data, UINT size);
void SetPerObjectConstantBuffer(void* data, UINT offset, UINT size);
void FreeTextures();
void Render();
private:
void CreateShader();
void CreateInputLayout();
private:
GraphicsDevice* graphicsDevice;
//Effect
ID3DX11Effect* effect;
//Input layout
ID3D11InputLayout* inputLayout;
/*per frame constant buffer is always on register1 and it's name is perFrameConstantBuffer
*per object constant buffer is always on register2 and it's name is perObjectConstantBuffer
*/
ID3DX11EffectConstantBuffer* perFrameConstantBuffer;
ID3DX11EffectConstantBuffer* perObjectConstantBuffer;
//individual variables with their names
std::map<std::string, VariableContainer> variables;
//texture variables with their names
std::map<std::string, ID3DX11EffectShaderResourceVariable*> textures;
//techniques
ID3DX11EffectTechnique** techniques;
UINT techniqueCount;
//file path to shader source code
std::string filePath;
};


This is my material class. It keeps list of variable values and corresponding ID3DX11EffectVariables. I keep them both because it is faster this way since I have to search for the parameter name only once.Other than that nothing interesting here currently but I will add render state options, physical properties and things like that.


class Material
{
public:
struct VariableBuffer
{
Shader2::VariableContainer* variableContainer;
void* buffer;
};
struct TextureBuffer
{
ID3DX11EffectShaderResourceVariable* shaderResource;
ID3D11ShaderResourceView* texture;
};
Material(void);
~Material(void);
void Initialize(Shader2* shader, std::string materialName);
void Release();
void SetVariable(const std::string name, void* variable);
void SetVariable(const std::string name, float variable);
void SetVariable(const std::string name, bool variable);
void SetVariable(const std::string name, D3DXMATRIX variable);
void SetTexture(const std::string name, ID3D11ShaderResourceView* texture);
void ApplyMaterial();
private:
//shader for this material
Shader2* shader;
//material name
std::string name;
//textures and variables with their names
std::map<std::string, TextureBuffer> textures;
std::map<std::string, VariableBuffer> variables;
};