engine design

That sounds more like an operating system than a game engine. Which is possibly a good thing.

It sounds like you want to do thread management yourself though... uh, good luck ^^v

"That brings with it a whole new set of problems (the graphics task stopping in the middle of a frame. "

Make a special priority that cant be stopped even if time runs out.

Just my two cents.

Zahlman: Now that you mention it.. its complexity is starting to look kind of daunting. Nowhere near an OS, though. I guess it''s time to start programming this sucker.

I know that the graphics task shouldn''t be interrupted, but what about for things like game logic? I''m thinking of telling it how much time it has for that iteration, and then it would do some processing with complexity based on how much it can do at once. The problem with this method is that the function would have to record its state somehow. (Curioser and curioser.. learning about OS design might be helpful. Or it might be going overboard.)

I think you''re overengineering. You have to ask yourself if you really need the complexity that you are designing.

Do you really need a memory manager? If you do, then do you need it for the release version or only for debugging? I really feel it''s pointless coding a memory manager before you''ve even made a game with the engine, yet so many people do it. That is something that shouldn''t be messed with until later when you find out you need one (for performance reasons). If you aren''t dynamically allocating memory all over the place then a release mode mem manager is just wasted effort. A debug mode memory tracker on the other hand would be useful for catching leaks and determining just how much memory you use, but need not be anything complex.

Why all of the complexity with the tasks? Do you really need different priorities? What will the tasks be used for? Would it be simpler to just put them all in a queue and have them execute in any order? That could be accounted for in the design. Even simpler would be to allow only one active task at a time. And if you really think you need multiple tasks, why not just use two priorities: FOREGROUND and BACKGROUND? Foreground runs in the main thread, while each background tasks gets a worker thread. This will allow you use tasks to make use of things such as asynchronous file loaders.

The biggest thing to keep in mind when designing an engine is to forget you''re designing an engine. If you are planning to make a game with it, that is. While the Enginuity articles are great, I really feel what they describe is overkill for most purposes. The scale of the games you want to make should be a factor in determinining how complex the design should be. Design decisions should be based upon functionality, not upon what fits the OO paradigm best. Focus on what you *really* need, not what you *think* you need.

One thing people often overlook in software design is that it is extremely easy to overengineer. Any idiot with and education in object oriented design can do it, and many do every day (look at some of the open source Java web application tools to see what I mean). It''s something I also see here quite often. Try not to think about the engine you are making, but the first game you want to make with it. What does that game absolutely need? That''s what you put into the engine. Keep the design flexible enough that you can add more features as you need them (such as a memory manager!) later down the road, perhaps for the next game. But don''t overdesign - i.e. don''t make it so flexible that the design is more complex than it needs to be. That''s the hard part, but can make a big difference not just in performance, but maintainability and usability as well.

Take a look at Crystal Space to see an example of an open source engine that, in my opinion, is overengineered. One of the reasons is that the project is designed to support any sort of game under the sun. The result is a complex, bulky system of plugins and design patterns. Othere may disagree with me. But the reason I would never consider CS for a project is precisely because it is designed to support any ype of game. I don''t need that. I need an engine designed for the type game I want to make right now, not for every game I might ever make.

However this turns out for you, I wish you luck.

Hmm... I kind of disagree with the idea of "overengineering". If you add that level of complexity, then most likely you will use it somewhere. I find that a strong design in the beginning will allow your code base to adapt with you. Hell, one of the advantages of an object oriented design is flexibility and reusability. You could make three different engines for three different games, or you could make one for three. Don''t get me wrong, there''s nothing wrong with making three different engines if that''s what you want, but IMO there''s nothing wrong with making a complex piece of software if you want. Here''s my thinking. 1. a complex engine is reuseable in theory. 2. a complex engine can lead to less bugs ( comparing debugging one complex engine to three game specific ones) 3. a complex engine can be just as easy to use if it has a simple enough interface.

quote:

---

Original post by INVERSED
If you add that level of complexity, then most likely you will use it somewhere.

---

I won''t argue with that (even though it''s not always true - it''s entirely possible to wind up with features you''ll never use, but you just wanted to have it there in case you need it). The point is, do you *need* it? Just because you make use of it doesn''t mean it''s needed.

quote:

---

I find that a strong design in the beginning will allow your code base to adapt with you. Hell, one of the advantages of an object oriented design is flexibility and reusability.

---

Amen. But strong design does not need to equal overengineering.

quote:

---

You could make three different engines for three different games, or you could make one for three. Don''t get me wrong, there''s nothing wrong with making three different engines if that''s what you want, but IMO there''s nothing wrong with making a complex piece of software if you want.

---

I certainly wasn''t advocating three engines for three different games. Although I can see how you got that impression.

Here''s what I''m getting at - design for the task at hand, not for games you may never make. If my first game is an indoor only, first person shooter, then why would I need a terrain renderer in the engine? Sure, maybe I want to make an outdoor adventure game next, but I''m not there yet. If I add the terrain engine now, I''ve just increased the complexity of the software for no gain. I should be focusing only on the features required by the current game, but leaving the door open for further extensions later. So when I''m ready to start the next game, then I can add the terrain renderer.

There''s no need to write three different engines for three different games. That''s just wasteful. If you know all of your games will be of the same style (such as first person) then you don''t need to change anything. But even if you want to make one FPS, one RTS, and one outdoor adventure, it''s still possible to reuse the parts of the engine that are common to all three games and only rewrite the parts that aren''t (such as the renderer). I don''t want code for a BSP tree and PVS in my RTS. It increases the size of the code with features I don''t need (bad). And a generic renderer takes away some opportunities for optimizations more specific to rendering an RTS.

quote:

---

1. a complex engine is reuseable in theory.

---

Any code is reusable in theory. Complexity does not equal resuability. But this debate isn''t about complexity. It''s about overenigneering - which is unecessary complexity.

quote:

---

2. a complex engine can lead to less bugs ( comparing debugging one complex engine to three game specific ones)

---

I totally disagree with this. More complexity in design usually means more code. More code means higher likelyhood of bugs and increased maintenance costs. As for the part about the three engines, see above.

quote:

---

3. a complex engine can be just as easy to use if it has a simple enough interface.

---

Of course it can. But using it is not the issue. Building it is. Do you really want to waste time on features you really don''t need?

Again, I''m not at all saying an engine shouldn''t be complex. I''m saying it shouldn''t be overly complex. Overengineering is bad! It leads to suboptimal performance (from code bloat, unecessary code paths, and other unforeseen issues), maintenance nightmares, and never-ending debugging sessions. Good software design begins with knowing what you need, and ends with impelementing what you need - not what you think you need.

quote:

---

Original post by Aldacron
One thing people often overlook in software design is that it is extremely easy to overengineer. Any idiot with and education in object oriented design can do it, and many do every day (look at some of the open source Java web application tools to see what I mean). It''s something I also see here quite often. Try not to think about the engine you are making, but the first game you want to make with it. What does that game absolutely need? That''s what you put into the engine. Keep the design flexible enough that you can add more features as you need them (such as a memory manager!) later down the road, perhaps for the next game. But don''t overdesign - i.e. don''t make it so flexible that the design is more complex than it needs to be. That''s the hard part, but can make a big difference not just in performance, but maintainability and usability as well.

---

Thank you.

I guess I was a bit soft in my first post. (The potential good that I see mostly comes from the fact that a lot of interesting ideas have come out of the whole Linux thing, what with microkernel architectures and all.)

Doing simple things, knowing how to keep things flexible without preemptively adding a bunch of extra classes, and being able to refactor mercilessly (which includes recognizing that your design is messed up in some way, and also knowing how to fix it) are great virtues in a programmer.

I really hate when people defend over-bloated designs with the belief that they are "more object-oriented" because there are, well, more objects. You wouldn''t call (warning: potentially offensive political content coming up) China more "human-oriented" than the US, or the US more so than Canada, would you?

I totaly agree with Aldacron. You don''t have to fgo far to see overengineering. Just look for any postmortem at gamasutra. They all have "What Went Wrong" where they describe that early enumeration of all possible features that will be used in game results in dropping large number of them in order to finish game on schedule.

While I appreciate Enguinity series, I think that kernel design is not that neccessary, simple switch block is enough. Also, for memory leaks use crtdebug library that comes standard with VC++. It is not easy to actually predict which features are going to be used in engine at design stage. Instead of that, consider another solution : make something workable. You are going to be more enthusiastic if you have something working. I''ve spent 4 months trying to fit my design with Enguinity(I thought then that it is a good design) when I realized that most of those features are not going to be used so I dropped whole idea altogether.

Remember : KISS.

I think the keyword here is "Focus".

Focus on what you require and then implement it.

For example, Doom3 was designed mainly for indoor environments and is likely to be extremely optimised for doing indoor scenes, using things such as Portals.

Another example would be to implement skeletal animation for a RTS. Sure it is the latest way to animate characters these days, but before implementing it, you should consider if it is viable for a RTS. I don''t think any RTS today would be using Skeletal Animation at all. It just doesnt make sense to calculate the different vertices of a model again and again for every similar unit onscreen at any particular frame. It would be far more efficient to just use the traditional frame by frame storage of animation(similar to MD2).

Oh my, such controversy I have inspired. Anyway, let me try and explain why I would ever want an engine this complex: my entire project is being done as a learning experience. I''ve programmed simple games (i.e. pong, breakout clones) using much simpler engine designs. In fact, I wouldn''t even call them engines, I''d just call them games, because everything was hardcoded. My goal for this engine is to have an engine codebase capable of supporting an FPS like UT, while still staying game-independent enough to work a Pong clone on.

I agree that if I''m just going to do another Pong, that I should keep the engine simple, and maybe extend it later when I actually want to do something more complicated. The memory management for system RAM is a way to learn about memory management (of course I don''t _need_ to memory manage), but I''m writing it to get a grasp on how it works, and maybe implement it for a video RAM memory manager in case I ever have that much geometry.

Hmm... I can see the argument against "overengineered" designs. However, I''m in a situation where I''m designing an engine to be as flexible as practicaaly possible. While I can see what you''re saying about things like spending time on unnecessary features, at the same time, I see it as being just as easy to not work on a given feature (say terrain rendering), but leaving enough space in the architecture for such a thing to exist. I agree that it is easy to get carried away with software design and all these neat design patterns you can use, but I think there can be a happy medium between bare minimum and code bloat. For instance, why is a kernel that bad of an idea? It''s simple to implement, and then may be extended to have profiling and multi-threading capabilities. So if half way down the line you decide that you need process XYZ to be multi threaded, you already have it setup and timed properly. It''s just been my experience when coding that sometimes things that look trivial now can be a big help down the line. Also, I believe in the idea that any given object should do it''s job and only it''s job, and do it''s job well, which can sometimes lead to a larger number of objects achieving a goal than in a less is more setup. Maybe that''s just my preference though. Anyone here working on any comercial projects that ccvouh for either philosophy?

Hmm, so much to respond to...

quote:

---

Original post by psykr
I know that the graphics task shouldn't be interrupted, but what about for things like game logic? I'm thinking of telling it how much time it has for that iteration, and then it would do some processing with complexity based on how much it can do at once. The problem with this method is that the function would have to record its state somehow. (Curioser and curioser.. learning about OS design might be helpful. Or it might be going overboard.)

---

One approach you could take would be to split algorithms into logical blocks and then stop processing blocks after a certain amount of time has elapsed - it's a bit like suspending the thread, except that (a) it won't ever stop in the middle of something which should be allowed to finish executing, achieving 'atomic' operations, and (b) you probably won't be resuming that thread at a later point in time.

Say you want to process AI for all the entities in your game world, and that this processing consists of having each AI 'think' about entities that are within a certain radius. You could design the think function so that it would only test entities that lay within a ring (specified by an inner and outer radius) - then you make a set of calls for each entity, testing first 0-10 metres radius, then 10-20, then 20-30, and so on.

If you were to run the 0-10 think for each entity first, and when they're all done you repeat with the 10-20, and so on, you can check how much time you've got left between each stage and stop if you're reaching your limit. So, entities which are further away are more likely to be 'skipped' by the AI.

The generalized approach is to turn tasks into discrete subtasks, which you then prioritize, and execute in order of priority. To make the system more flexible, you can set a 'maximum skip count' for tasks, which forces the system to run a subtask (no matter how much time you have) at least once in a while.

If you're basing it on Enginuity, then you'd probably want to write such a feature into the kernel itself. A maximum skip count of 0 forces a task to execute every loop, while a maximum skip count of 3 allows a task to be skipped 3 times in a row before it must be run.

Oh, and with regards to it looking like an OS - what is there, besides games, which executes such a variety of tasks in realtime?

quote:

---

Original post by Aldacron
Do you really need a memory manager? If you do, then do you need it for the release version or only for debugging? I really feel it's pointless coding a memory manager before you've even made a game with the engine, yet so many people do it. That is something that shouldn't be messed with until later when you find out you need one (for performance reasons). If you aren't dynamically allocating memory all over the place then a release mode mem manager is just wasted effort. A debug mode memory tracker on the other hand would be useful for catching leaks and determining just how much memory you use, but need not be anything complex.

---

I've been writing design documents for my commercial engine recently. The memory manager in that doesn't do nearly as much as Enginuity's one - no reference counting or built-in smart pointer support - but I discovered early on that I wouldn't be able to use the standard new/delete operators for allocating objects.

You can't properly examine performance until the game performs something worth examining... Imagine if I'd taken your approach, and only thought about the memory manager when finding that something in my engine is leaking memory. Imagine if I'd found out that I couldn't use new/delete after a few thousand lines of code had been written. Replacing new calls with my ALLOC() macro would have taken a fair bit of time.

quote:

---

Original post by Aldacron
Keep the design flexible enough that you can add more features as you need them (such as a memory manager!) later down the road, perhaps for the next game. But don't overdesign - i.e. don't make it so flexible that the design is more complex than it needs to be.

---

That's the crux of the matter - balancing the two. But the design isn't a locked document that must be written before coding starts; there's nothing wrong with refactoring a design which is inappropriately complex into something which is just right. And given the choice between refactoring to add flexibility and refactoring to take it away, I find the latter to be much easier in pretty much every situation. That's why I'd say that it's better to go all out with the design and featureset at the beginning, and then scale things back later when it's obvious that a switch() will be a sufficient replacement for polymorphism, and so on.

Yes, it inevitably ends up in writing more code that you need to. But with a library like an engine, one of the main points is reusability; surely then the most reusable library would be one which contains a massive featureset of which the client then only picks out a subset?

You might not need prioritized tasks today, but maybe you'll need them two projects from now. More importantly, maybe you'll be presented with a situation in the next project where they could provide an alternate solution to a problem. As such it's giving you extra options; you could profile both and then pick the faster one. If you hadn't written that functionality originally, it wouldn't have been written until the second project from now; it may have occured to you as an alternate solution in the first project, but if you already had a working solution, you'd be unlikely to go ahead and code it.

It's the pyramid structure I described in the first Enginuity. All your clients use a common base (kernel, file io, etc). Then they select one from a set of 'level 2' layers. That leads them to a possible set of 'level 3' layers and so on. 'Foundation'::'Client-Server architecture'::'Octree based world,' etc.

quote:

---

Original post by Aldacron
Here's what I'm getting at - design for the task at hand, not for games you may never make. If my first game is an indoor only, first person shooter, then why would I need a terrain renderer in the engine? Sure, maybe I want to make an outdoor adventure game next, but I'm not there yet. If I add the terrain engine now, I've just increased the complexity of the software for no gain. I should be focusing only on the features required by the current game, but leaving the door open for further extensions later. So when I'm ready to start the next game, then I can add the terrain renderer.

---

Hmm... if that's your stance I may have misinterpreted you somewhat. I thought you were saying that there was no point even "leaving the door open."

However, it's sometimes not possible to 'leave the door open' without building a whole new room the other side for it to open into. This is especially true with something as low-level as a memory manager or game kernel.

Richard "Superpig" Fine
- saving pigs from untimely fates, and when he's not doing that, runs The Binary Refinery.
Enginuity1 | Enginuity2 | Enginuity3 | Enginuity4 | Enginuity5
ry. .ibu cy. .y'ybu. .abu ry. dy. "sy. .ubu py. .ebu ry. py. .ibu gy." fy. .ibu ny. .ebu
"Don't document your code; code your documentation." -me
"I'm not a complete idiot; parts of me are missing." -}}i{{

[edited by - Superpig on January 11, 2004 7:18:13 PM]

quote:

---

Original post by superpig
Imagine if I'd found out that I couldn't use new/delete after a few thousand lines of code had been written. Replacing new calls with my ALLOC() macro would have taken a fair bit of time.

---

Talking about memory managers specifically, it's very easy to plug one in even at the end of a project as long as you account for the possibility up front. For example, in my C projects I always #define myMalloc and it's brethren and use those calls throughout the code. Later, if I need to track the memory usage stats (which I usually do for safety), or actually manage a memory pool (which is rare), then I just change the defines in one header. The same can be done in C++ by implementing a custom new/delete. Really, that doesn't even need to be done up front. The memory manager in GPGems 2 can be dropped in at any point in a project's dev cycle. And the mem manager in Torque (which is a beast of a red/black tree) was originally coded and dropped in the weekend before Tribes went gold, to help solve some memory issues.L So it's something that can certainly be delayed.

quote:

---

That's the crux of the matter - balancing the two. But the design isn't a locked document that must be written before coding starts; there's nothing wrong with refactoring a design which is inappropriately complex into something which is just right. And given the choice between refactoring to add flexibility and refactoring to take it away, I find the latter to be much easier in pretty much every situation. That's why I'd say that it's better to go all out with the design and featureset at the beginning, and then scale things back later when it's obvious that a switch() will be a sufficient replacement for polymorphism, and so on.

---

I agree, the design is not set in stone. Perhaps I'm just jaded, but my experiences have made me so. Java web programmers love to abstract, dynamically configure and instantiate, and implement fantastically flexible MVC designs for their apps. They are so good at it, that they produce a finished product that meets all of their expectations in flexibility and reusability (the developers perspective), but often leaves a lot to be desired in performance and usability (from the user's perspective).

It's not the language. Web apps that are not overengineered work wonderfully well. So I tend to think the opposite of what you say here. I think it's much easier to add than to take away. And I'm not talking just about features, but actually making changes in the architecture. You don't get overengineering from feature bloat, but from object interactions designed to ease the job of the programmer. For large shops (or for someone developing an engine to license), other programmers may well be the target user. That's fine. But for small teams, hobbyists, indies... I really believe time is much better spent deciding up front to do only as much as is necessary.

And while it may be a tremedous amount of work to overhaul the architecture of an engine at a later date, it may not be necessary to do so with a good deal of forethought into how you can plug components in and out of the engine.

quote:

---

Yes, it inevitably ends up in writing more code that you need to. But with a library like an engine, one of the main points is reusability; surely then the most reusable library would be one which contains a massive featureset of which the client then only picks out a subset?

---

Sure, a huge featureset infers greater resuability. But again, the designer has to look at his needs. This goes back to considering the size of the team. Does a one man show really need all of those features up front? No way. If he wants to make money, he needs to get games out the door and on to real arcade, not spend a year developing all of the features he *thinks* he needs. A company like EA, on the otherhand, would certainly benefit from such an engine.

Another consideration is what sort of games you plan to make. I haven't registered my company yet, but I already have finished design documents for the first 3 games I want to sell as an indie. The engine I'm developing will support all three games. The first one is single player only, so I won't be putting any net support into the engine until I start the second game. Also, the first game will not have a need for any sort of object/memory pooling. The third game most certainly will, but the second one I'm not sure about yet. So there will be no pooling implemented until the second game comes if I need it, and not until the third game if I don't. I don't want to waste time on features I don't need now. I can't afford it.

quote:

---

Hmm... if that's your stance I may have misinterpreted you somewhat. I thought you were saying that there was no point even "leaving the door open."

---

Always leave the door open! In a nutshell all I'm saying is design for your needs. For most of us on these forums, we don't need to code an engine that 100 other coders in 10 different departments will be using. If, ultimately, putting together a behemoth which supports every sort of game from Pong to Doom 3 to Steve's Super Soccer, benefits you and your project then it must be the right path. People just need to learn to recoginize when they are designing what they really need, and when they are designing what they think they need. There really is a big difference, and making the wrong choice (either one) can cost you time and, if you are in in commercially, money.

[edited by - aldacron on January 12, 2004 8:01:06 AM]

OK, I think we''re on the same page now, in that I understand and agree with what you''re saying. I think I''ve been inconsistant in that I''m talking about commercial engines (where the client is the 10 other departments that you mention) while happily ignoring the realities of commercial development, i.e. the more you put into the engine up-front, the longer it''s going to take to develop and thus the more time you''re spending not writing games which are going to make you money.

Richard "Superpig" Fine
- saving pigs from untimely fates, and when he''s not doing that, runs The Binary Refinery.
Enginuity1 | Enginuity2 | Enginuity3 | Enginuity4 | Enginuity5
ry. .ibu cy. .y''ybu. .abu ry. dy. "sy. .ubu py. .ebu ry. py. .ibu gy." fy. .ibu ny. .ebu
"Don''t document your code; code your documentation." -me
"I''m not a complete idiot; parts of me are missing." -}}i{{