Direct3D Shading Language

Writing an entire engine without having any experience with shaders is really not a good idea, start with basic games before you work on a huge advanced project like an engine.

Direct3D uses a C-like language called HLSL, which is similar to the shading language GLSL used by OpenGL. Since you're talking about shaders in the context of post-processing effects I'll assume you're using fixed function pipeline functionality in D3D9 to do your 'regular' rendering, since D3D10 and beyond require you to write shaders for all your rendering tasks.

A good place to start with HLSL is the DirectX SDK, which contains a good amount of samples and tutorials on how to implement basic functionality with shaders.
I recommend you practice and experiment with shaders a lot, since they can be quite tough to wrap your head around in the beginning.
It might be a good idea to abandon the use of the fixed function pipeline altogether and force yourself to do all your rendering tasks with shaders. This will give you a much better understanding of what shaders are and what they can do.

The ShaderX books are not meant for teaching people shader programming. They contain explanations and implementation of advanced rendering techniques, so they won't be useful to you at this point.

Thanks for the reply.
Actually I'm currently stuck only with the rendering part
I have already implemented Event System working with Lua, Bullet physics and scene graph. Also memory pool and resource cache, managing my memory, Loading assets from zip and custom engine format. I don't have only world editor and "proper" rendering..
Now I'm going to check the DXSDK Samples...
P.S. I actually don't have idea about any rendering teckniques. For example how to get better frame rate. Or how to render +8 lights(Maybe I can calc the most valuable lights for one object and enable them before rendering it ;/ ). Can I find everything I need in the samples or I will need external info source?

I don't want to turn this into a 'how to build a game engine' thread, but I do hope you realize that an engine is much more than just a few systems slapped together, right?

The DXSDK samples will provide you with the basics of how to write shaders, that's pretty much it. They have advanced samples, but I've never found them to be really useful.

I actually don't have idea about any rendering teckniques. For example how to get better frame rate. Or how to render +8 lights(Maybe I can calc the most valuable lights for one object and enable them before rendering it[/quote]

Worrying about the performance of a system without having even written it is something you really shouldn't do, premature optimization mostly leads to more problems than it solves.

How your rendering system works is completely up to you. There are a lot of ways to set up a renderering system, each of them having a set of advantages and disadvantages, and each implementation of these setups can differ based on what your requirements are.
Since you're writing an engine you'll probably want to provide the option for a user to decide which setup applies to his/her needs, and add the ability for the user to implement a custom setup if required, just like you should with any other system.

The 8 lights maximum (it's actually a guaranteed minimum rather than a maximum) is a limit imposed by the fixed function pipeline, this limit does not exist in a shader-based approach where the lighting calculations are completely up to you. How lights are handled and how many lights you can include in a scene again depend on your rendering setup. There are techniques like deferred rendering and light pre-pass rendering which potentially allow for thousands of active dynamic lights in a scene while sacrificing other qualities like material variation. This is in contrast to forward rendering where you can have a huge range of materials while compromising on the amount of lights you could realistically use.

There are enough tutorials on how to build a basic shader-based rendering setup like proper forward rendering or deferred rendering, but when you get to the much more complex features a proper general purpose rendering engine should have you'll pretty much be on your own.

working on Directx 9.0 c while running one sample error warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them how to solve error with which DX SDK

I don't want to turn this into a 'how to build a game engine' thread, but I do hope you realize that an engine is much more than just a few systems slapped together, right?[/quote]
Yeah I understand it. Its actually something like hobby project, I want to prove myself I can do it(and to find out what I can't do so I can start learning it).

Worrying about the performance of a system without having even written it is something you really shouldn't do, premature optimization mostly leads to more problems than it solves.[/quote]
I found that for my personal problem. After reading so many books and everyone in them is talking abour performance. There is always one voice that is shouting in my head that I shoud do things well in favour of performance, even when I don't know how to do it. However I've have always known that this is wrong.

Also I'm not english speaker(and I'm sure u already know that) and most terms sound quite non-natural or strange in bulgarian so I can't "understand" them in my own language. I need to fully understand nature of one process so I can get what it actually mean. (for example deferred rendering... I've heard it but not image appear in my head)
However after I saw it from your reply I can now google it and find tutorial for it. Thats what I want to thank you for.

I don't think an "engine" means what you think it means. An engine is not related to rendering only, but to the framework that is used to create an application as a whole as well as a collection of tools for asset and content creation.

As far as learning DirectX, check the SDK documentation. It is very helpful. You should be comfortable with the pipeline and the different resources needed by each stages.

Also, the best book I found is Frank Luna's "Introduction to 3D Game Programming with DirectX 11". I highly recommend it. He has intro to the linear algebra stuff you need to know and introduces the xna math library. He also goes into a lot of general techniques such as shadows / reflections using the stencil buffer, lighting, deferred rendering, SSAO, and much more. He also has a chapter dedicated to DirectCompute. By the time you finish the book, you'll have a very firm grasp on DX11.

I don't think an "engine" means what you think it means. An engine is not related to rendering only, but to the framework that is used to create an application as a whole as well as a collection of tools for asset and content creation

He clearly stated in his second post that he has been working on other systems besides rendering too.