engine design

Its hard to answer your question will such little information.
A generic answer would be to have your engine process everything and then shoot the processed information to your graphics API and thats where your shader code would be.

You may need to provide more information so that others can input more specifically.

Continue writing games. Then look back and factor out any common and reusable code. This will serve as the basis of your engine.

The 2 common methods are shader stitching and shader permutations.

Stitching means that your shaders are spread over many files, each file containing code for just one small feature/algorithm in the shader. A routine is written to build a complete shader by combining these files based off what features are desired to be in the final shader. For example, if lighting is enabled, it will stick the code from “Lighting.shader” into the main loop at a certain location. Inside Lighting.shader it will replace a line with the code from BlinnPhong.shader or OrenNayar.shader depending on which lighting model is supposed to be used. On the other hand, if no lighting is desired, Lighting.shader is not inserted into the final shader code.


I prefer permutation.
In the case of permutation, you have just one large shader file for each major rendering type. One for per-vertex lighting, one for per-pixel lighting, one for shadow-map creation, etc.
Inside these are a bunch of macros that enable or disable certain features such as lighting. When a model wants to render itself it passes some flags off to the model shader manager which it translates into macros which cause code in the file to appear or disappear, effectively enabling/disabling the requested features.

The reason I prefer this method is because the code is all in one file and it is easier to maintain. Dealing with macro spaghetti is the lesser of the 2 evils.


L. Spiro

Continue writing games. Then look back and factor out any common and reusable code. This will serve as the basis of your engine.

+1.

This link helps to further this idea: http://scientificninja.com/blog/write-games-not-engines

I have 2 problems with the above reply.
Firstly there is nothing wrong with someone wanting to write a game engine as an educational tool and the topic-poster had a very specific question about game-engine development. “Write games, not engines” is not the universal solution. If it was then guess what: Games would never be made since no one spent the time to write the engines behind them. Not only that but I would be out of both a job and a hobby. Some people, myself included, appreciate the lower-level technology behind game development and enjoy learning about it more than we enjoy making the actual games. It is also true that some people dive too deep before they are ready, but this is not the way to inform them of that fact.

The second problem I have is, Josh, Mr. Petrie, please change the title of your article. The things you say in your article are not wrong, but people coming into it start off with a false idea about what is trying to be proved and that is away with what they walk. I already posted multiple quotes from people who read your article and somehow took away from it that writing engines is a plague that must be avoided at all costs. They couldn’t even figure out how to write a game because every idea they had involved making some semblance of an engine and from your article they learned that that was evil.

The content may be accurate but the title is so misleading that I simply can’t support it in any way until the title is changed. The above reply is just another example of it doing more damage than good.


L. Spiro

Firstly there is nothing wrong with someone wanting to write a game engine as an educational tool and the topic-poster had a very specific question about game-engine development. “Write games, not engines” is not the universal solution. If it was then guess what: Games would never be made since no one spent the time to write the engines behind them

This is a gross over-approximation. As much as I sympathize with the concept, I always took the recommendation as a hint to focus on design and logic rather than just logic. I don't think anyone suggesting that ever took this to the extreme of not writing logic.

there is two kind of way treating shader...

one is make shader with designer for 3-4 month ( I highly recommend this way) In making game, theres is quiet just someking of ways of using shader...

so Art director and graphics programmer discuss on shader and for visual and for performance.. they work together and make some standard for it. after that sub artist use that shader and ask few modification sometime ..but most game have very similiar feeling about screen and light it's not often modifying standard shader.. just FX part need new shader constantly ( but it restricted very much)

this way is so simple and very productive and queit good for most game developpment.. even situation on Unreal3's shader tool supported , Art team don't make many formular on shader.. It's so true..changing basic fomular is huge chaos and so much work in Art team .

Secound way is as many big comercial engine build shader compositing tools.. you already know about Unreal 3 and others .( like unitity3) it's base cores is text manipulation and compile it and show to viewport...

It's have plexibility but, extremely hard to optimize and I doubt it's productivity.. It's good for just they don't have good shader programmer at all and need extremly complex scene and object.. really big game..

I hope it can help you... but.. don't worry about shaders it's just simple script ( of course It's very similiar ASM but.. think it so easy and be relax )

Continue writing games. Then look back and factor out any common and reusable code. This will serve as the basis of your engine.

I'd much rather think of the "showcase game" of my technology as a way to shape the engine I've written for it. I would rather start with an engine that's broad in scope and use my first game with it as a way to iron out bugs in the engine, discover weak points in my engine code, and streamline the parts of the engine code that need it most so that when I make a game in the future, I have a solid tech to start with that I know will work and I don't have to remove a bunch of extraneous hacks that I had to use to force my previous game to even function.

Frankly I agree with the above post. There are certain things that are obviously common to all games, such as loading MP3 or OGG files, PNG or WHATEVER graphics files, etc.
3D file formats are always custom. Loading a 3D file and any animations with it are going to end up being common to any 3D game you make no matter what (your own format, but always the same format for your own games, just with extensions added later).
Sound? Is pan not a feature of every sound engine ever? Is volume not a feature of every sound engine ever made?


There are obvious ways to break down engine components into things that you will end up using across all of the games you plan to make, and the fundamental idea of “make games not engines” is simply flawed.


Make games and then look for reusable code? Why would anyone do that when we know from the start that we need a 3D file format, animation format, camera-animation format, sound format, image format, etc. We know from the start that we need to support basic features such as sound panning, sound volume, camera panning, camera animations, 3D animations, etc.

Whether or not we get it right on the first try is unrelated to our objective. If our plan is to make a game, we will still fail to make the best animation format on our first try. Same if our goal is to make an engine.
Either way it takes practice practice practice.


So let those who want to make game engines work on game engine and practice practice practice.
Let those who want to make games work on games and practice practice practice.


L. Spiro

Firstly there is nothing wrong with someone wanting to write a game engine as an educational tool and the topic-poster had a very specific question about game-engine development. “Write games, not engines” is not the universal solution. If it was then guess what: Games would never be made since no one spent the time to write the engines behind them. Not only that but I would be out of both a job and a hobby. Some people, myself included, appreciate the lower-level technology behind game development and enjoy learning about it more than we enjoy making the actual games. It is also true that some people dive too deep before they are ready, but this is not the way to inform them of that fact.

There is indeed nothing wrong with people wanting to throw code together under the gise of some sort of 'learning' experiance BUT I'm willing to bet that ANYONE doing that is going to end up with a mess which isn't fit for purpose WITHOUT a project driving it.

Of course games would be made without an 'engine' behind it - If you start from scratch you don't spent months putting together some grand unified design before working on the game, you attack the game, figure out what the game needs and then write the code for that bit refactoring and rearranging as scope changes. At the end you have a nice chunk of code, a shipped product AND code you can reuse as you know it is fit for purpose.

Yes, there are many people who like hacking on low level stuff, hell I'm one of them, but thinking you can sit in isolation throwing out code which is going to be useful is foolish, more so if you are a beginner/someone new to the process as you have NO IDEA what you want and need to do. To use the overused car example; it would be like someone sitting in a shed who has never designed a car trying to design and build a powerful engine block - sure they might be able to do it but when they come to try and fit it into a car they might be in for a shock...

If you take a step back you'll very quickly see that EVERY useful game engine which has appeared in the last 15 years has, at some point, started off as a game or been driven very firmly by game requirements; the Unreal engine was not developed in isolation and thats part of what makes it good.

And I don't throw this out there as a 'oh this could happen...' I've SEEN it happen in a company where a bunch of very clever people sat down to write an engine in isolation without a game to drive it (only a small test scene) and it didn't survive first contact with a game; large parts had to be rewritten (not refactored, chucked and redone), more had to be refactored, massive holes in the design came to light once someone tried to use it. Even things as fundimental as 'spawn an object in the world at runtime' was missing because it wasn't needed for the tiny world based test case. This wasn't some bunch of newbies, this was people who'd been in the industry sometime, had shipped games and still managed to screw up to the point where only one game will be shipped on this engine before it is going to be (effectively) thrown away and rebuild/salvaged using input from the games which will need to use it in the future.
(It also being done slower this time; before the 'renderer' interface was 'designed' in 2 days... this time out we are about a month in and still not settled on a design, although we are converging on something which everyone seems happy with.)

So if professionals can screw up then I give someone who is new at it no chance of producting something workable without a game as their primary focus.

Sure, if they want to spend months/years on something which they have to massively refactor at the end of the process when they realise 'shit, this doesn't work...' then fine, I don't really care, although I reserve the right to point and laugh at them when they realise this.

Thus, if you want to write something which can be used to power future games then write a game - you'll learn just as much if not more while producing WORKING and USEFUL code at the end of it. Even if you don't 'ship' the game at the end, or get 6 months in and decide to switch game project the point remains; with something guiding you'll still have usable code to run with.

Of course, if you don't want to power a game with your code then do whatever; it doesn't really matter, if you want to just get pretty images on the screen then you can just throw things togeter adhoc... *shrugs*

Whether or not we get it right on the first try is unrelated to our objective. If our plan is to make a game, we will still fail to make the best animation format on our first try.

OK, so you invent an animation format AWESOMEANIM, you get it hooked up and working, you can see a character being animated in a window... how do you know if this is any good for a game to use? What constraints did you place on it? Did they make sense outside of a test renderer? How optimal is it? Is it going to slow down in a 'real' situation rather than your whitebox test space?

So yes, you might well fail to make 'the best' animation format but you won't know this until you (or someone) tries to use it at which point, depending on how badly you've screwed up, you might well end up throwing away 90% of what you've done in order to support what needs to be done. At which point you might argue 'ah! but we get to learn more!' which I can't argue with however I doubt you'll learn that much more as chance are the core principles are going to be the same, you'll just "learn" how to do them differently with a little bit of extra knowledge on top.

The same can be applied to sound pipelines, rendering pipelines (scene selection, sorting, rendering queues etc; all might look 'great' in your whitebox and fail in real life), UI system, message system.. the list goes on.

Without something to aim at you WILL screw up and you WILL produce code which is worthless and you WONT really learn anything more doing it again... well, aside from 'real requirements are important!' which is something so so so many programmers miss because instead of thinking "lets ask who needs to use it.." they start off with "I'm the best one here, so I'll just run with my ideas and not bother asking those who need to use it..." and then act surprised/annoyed when their "solution" turned out to be bullshit.

While I agree with most of your post L. Spiro, I also advocate "practice practice practice".

Make games and then look for reusable code? Why would anyone do that when we know from the start that we need a 3D file format, animation format, camera-animation format, sound format, image format, etc. We know from the start that we need to support basic features such as sound panning, sound volume, camera panning, camera animations, 3D animations, etc.[/quote]
I need to comment on this though. I feel like you've used the word "we" as if everyone is as experienced as you in writing software. Not everyone understands the concepts of these core engine features - nor does every game even require some of the features you've listed above.

How does one even implement 'camera panning' without some concrete requirement of how it's going to be used? I'll usually fall into the "what if the end-user wants to do this or that" trap... and will ultimately end up writing a bunch of code that never gets used. Personally, I'd rather write a feature for a requirement of the game, instead of some generic feature. And then, if I notice this may be a nice chunk of code to use again, factor it out.

Writing software without concrete requirements can be very difficult, which is why I believe writing engines can be so difficult. Writing a game/demo/unit test is so much easier because those applications inherently come with a set of requirements which have purpose, and can help keep a programmer focused.

I'd much rather think of the "showcase game" of my technology as a way to shape the engine I've written for it. I would rather start with an engine that's broad in scope and use my first game with it as a way to iron out bugs in the engine, discover weak points in my engine code, and streamline the parts of the engine code that need it most so that when I make a game in the future, I have a solid tech to start with that I know will work and I don't have to remove a bunch of extraneous hacks that I had to use to force my previous game to even function.[/quote]
I can respect that darkhaven3. +1.

I need to comment on this though. I feel like you've used the word "we" as if everyone is as experienced as you in writing software. Not everyone understands the concepts of these core engine features - nor does every game even require some of the features you've listed above.

Then maybe that individual shouldn't be writing their own game engine just yet. It is common knowledge that a good game in this modern era generally requires that there be a standard format or formats for images in place that you will read from the disk, audio of some form, and some kind of user interface. I believe all of these can be cleanly separated into core components of a game engine, and you don't have to be the most intelligent person in the world or even a programmer at all to understand these requirements.

How does one even implement 'camera panning' without some concrete requirement of how it's going to be used? I'll usually fall into the "what if the end-user wants to do this or that" trap... and will ultimately end up writing a bunch of code that never gets used. Personally, I'd rather write a feature for a requirement of the game, instead of some generic feature. And then, if I notice this may be a nice chunk of code to use again, factor it out.[/quote]

This implies that the programmer has no idea what kind of game he will be programming, or will be somehow working on-the-fly without some kind of design document about how the game should be implemented written down first. If you are writing a text adventure, you know you will not need camera control of any kind. If you are writing some kind of 2D or 3D game, you will certainly need some kind of "camera" in as abstract of a sense as I can put it. Don't need a particular camera feature for your game? #ifdef it out in your game header files or something.

Writing software without concrete requirements can be very difficult, which is why I believe writing engines can be so difficult. Writing a game/demo/unit test is so much easier because those applications inherently come with a set of requirements which have purpose, and can help keep a programmer focused.[/quote]

If you don't have a game design document that clearly outlines the scope and format of the game before you start writing code, I believe you have more serious issues to attend to than deciding how to write your engine. These are all issues that seem easily resolved at the planning stage; if I'm writing a game for Gameboy Advance, I can have it on good faith I'm not going to be needing things like vertex coloring and DX11 tesselation. Likewise, if I'm writing a game on PC and I want it to look fancy, I can implement these features and I will know that I will need to have this information stored in some kind of consistent format.

In short, while I respect that you might find it difficult to write a game engine, it feels like you are assuming that everyone just writes game engines without some kind of idea as to what kind of game the engine will be used for.

In short, while I respect that you might find it difficult to write a game engine, it feels like you are assuming that everyone just writes game engines without some kind of idea as to what kind of game the engine will be used for.[/quote]
Certainly not everyone. I did not mean to make that assumption. My initial comment meant to state that if one is having trouble writing an engine, then don't be afraid to continue writing games/apps and then refactor. The notion of "I must write an engine before I write a game" can be a trap for beginners with less experience.

If one is experienced and disciplined enough to write small unit tests for each feature and prove that their implementations work and their API's are intuitive throughout development, then I agree with your methods completely.

The bottom line is, just write code, fail early and often, write more code, and continue learning. This experience will only make you a better programmer overall.

I think that writing engines isn't a foolish act, given the fact that there are a bunch of engines out there that everyone could afford even for a commercial application. If someone feels good writing their own stuff and feels strong enough to comprehend the concepts behind that or the other technical feature - then let them do it.
I think that if someone is capable of writing a proper engine, then they are capable of writing a proper game too. Writing the game logic and stuff sometimes isn't much easier than writing a renderer for example. In order to make a great use of any engine, a person should know a great deal of 3D math, theory and concepts.
The experience gained while diving into the code is priceless in the long term. You depend only on yourself, your game has no "back box" dll's, you understand how it works , you can put it in your portfolio, you can show your employer that you can write an engine + editor that the previous candidate could only use.
Does anyone have a statistics of how many successful indie games are made using third party engines ?

Stitching means that your shaders are spread over many files, each file containing code for just one small feature/algorithm in the shader.

I am using the glsw shader wrangler for this, and I am happy with the simplicity and functionality I get.

OK, so you invent an animation format AWESOMEANIM, you get it hooked up and working, you can see a character being animated in a window... how do you know if this is any good for a game to use?

Because:

This implies that the programmer has no idea what kind of game he will be programming, or will be somehow working on-the-fly without some kind of design document about how the game should be implemented written down first. If you are writing a text adventure, you know you will not need camera control of any kind. If you are writing some kind of 2D or 3D game, you will certainly need some kind of "camera" in as abstract of a sense as I can put it. Don't need a particular camera feature for your game? #ifdef it out in your game header files or something.

In other words the only people who actually run off and code an engine without any clue what-so-ever as to how it will be used are people such as myself who intend to make a middleware company, yet even I have 2 ideas for upcoming games I want to make, meaning that even I am initially building my engine based on some idea of how it is going to be used.

Without experience you may not make the best animation system. Without experience you won’t make the best anything. But you know you’ll need one so you have to give it your best shot anyway, and there is no reason not to mentally organize it as being part of the engine.

The only time I write engine code without actually putting it into the engine library is when it is highly experimental for me and I basically consider that I am not sure how it is going to turn out so I don’t want to commit it to the engine yet. For example my first physics engine. The components and bits of math/intersection tests/collision detections/etc. were directly implemented into the engine because they are easy to prove correct, but the overall “engine” part of the physics engine was created between the game project and the engine project until proved to be working, efficient, and a satisfactory solid base (note I didn’t want to create a whole separate project for it so it was actually part of the game project but so isolated from the rest of the game code that all that would be necessary to move it to the engine would be to copy the folder and add the files to the engine).


Writing games and then trying to stockpile all the reusable bits—I don’t imagine that really works out most of the time. Your reusable bits go from project-to-project and get mangled and cluttered over time. It is easy for you to just keep copying the files to your new game project so you end up with a ton of copies of it and never really take the time to make a proper framework out of it. When you finally do you realize, “Oh, this file was supposed to be reusable but now I remember I was really excited to get results on X project and I threw in some illogical dependencies thinking they would just be for that project, then forget and it grew.”
“Hmm, this file relies heavily on my implementation of X feature, but for Y reasons that code needs to be rewritten so I can’t really move any of this ‘reusable’ file over to the engine.” (This case might actually improve your end results though assuming it forces you to rewrite that code.)

The only way to gain success is to be thinking fairly strictly about what is engine code and what is not, and if you are doing that anyway it is better to actually do it too.
Besides, if you aren’t able to identify what is or could be reusable, you wouldn’t be able to make “make games, not engines” work for you either.


L. Spiro

There are some amazingly helpful people on here who have some really quite brilliant advice but I think sometimes it becomes a bit repetitive where something subjective is in discussion and I can't think of anything more subjective than "games or engines?".

If someone wants to write engines, they will. If they've got to the point where they have an interest in engines, they've probably already made their mind up and nobody on here is going to make them think "wow, writing games not engines, now why didn't I think of that". If they want to write games on the other hand, the same thing applies, that's what they'll do.

I find that 9 times out of 10, the people who come on here wanting to write the next angry birds usually want to write a game, not an engine. That's fine, no need to tell them to write games not engines.

I am writing an engine and I get massive amounts of satisfaction from it, it's the most therapeutic thing I can think of doing. I thrive on finding out how things work and trying to do them myself. Will I end up with a game? Probably not but anyone who wants to write an engine, generally has an idea of the type of game it might be suitable for, 2d, 3d, etc. you don't need a full game spec to write an engine, it's really fun trying to think of how an animation system might be used, or how particle effects might be used. It's also great for coding practice from an architecture perspective - who cares if it makes a game or not. Same goes for creating games, who cares if it makes a great engine or not, as long as you're enjoying yourself - at the end of the day, isn't that what we're here for?

I say let's just agree that if someone wants to do something, they've already made up their mind and let's help them with what they're trying to do. If someone has no clue about games or engines and is asking advice on which is best to start on, then we should be asking questions like "what are you interested in? What are your goals?" and so on.

Debates like these are always healthy but surely this particular subject has been discussed too much....?

The OP asked about shader permutations and it sounds like he wants to write an engine and isn't just messing about - shall we concentrate on that?

I just wanted to very quickly add, I don't disagree with any responses on here, just that a useful-sounding first post has turned, once again, into "write x not y"

If someone wants to write engines, they will. If they've got to the point where they have an interest in engines, they've probably already made their mind up and nobody on here is going to make them think "wow, writing games not engines, now why didn't I think of that". If they want to write games on the other hand, the same thing applies, that's what they'll do.

In my experience most people who 'want to write an engine' do so because they believe that writing an engine is some kind of requirement before writing a game. Most of these people want to write games really but due to hearing the word 'engine' a lot think the wrong thing and go off in the wrong direction generally because they think they must have an engine and don't want to 'do it wrong'.

The OP asked about shader permutations and it sounds like he wants to write an engine and isn't just messing about - shall we concentrate on that?
[/quote]

The OP didn't ask about permutations at all.. he never mentioned them... he asked 'How should shader support be designed in the engine so it is actually easy to use while programming the game?' which says nothing about permutations at all.

Maybe shader permutations factors into this.. maybe it doesn't... of course without knowing what he wants to DO (beyond a nebulous 'make an engine' for some mythical 'game') and how things interact this is close to impossible to really advise on...