Extendable engine architecture

First of all, some paragraph breaks are always appreciated

Quote:

I'm essentially trying to think of a way of writing an engine from scratch in such a way that you'd never have to write an engine from scratch *EVER* again. I think such an approach is possible, but i'm not sure if such an approach is practical.

Not going to happen. It reminds me of something another board member said. To paraphrase: "There are those on this board that are making games, and there are those that are making engines that are so flexible and so generic that no one will ever use them." You will always have to tailor the game you make to some degree. You can make generic feature-full "engines" all day, but eventually someone will have to make a game, which of course requires something specific.

Your other problem is that you are thinking in "3D FPS" terms. That's woopdeedoo great, but your Ultimate 3D FPS Engine will do nothing for my strategy game or my puzzle game or anything that isn't a 3D FPS.

leiavoia, I don't agree, true i used only 3D FPS examples, but the engine i'm currently writing is aimed towards an RTS project. It is an engine with a purpose, i'm mearly trying to think of ways of removing a great deal of the work i'm currently doing in future projects.

The difference between a 3D FPS and a 3D RTS is at a high-level of the engine, not a low-level one. Both need to push polyons on to the screen, and the graphics engine shouldn't care if its doing the polygon pushing for an RTS or an FPS. The polygons may be generated, culled, or dealt with differently in a different area of the engine. Why rewrite the graphics render between an FPS and an RTS if it could do what you want if the engine architecture was designed with flexibillity in mind?

Also i'm not suggesting writing an engine that has 'all the features under the sun' more an engine that can swap in and out features easilly and quickly WITHOUT rewriting anything but the specific code for dealing with the said feature.

The RTS i'm currently designing is based around planetary assualt, using space fleets and ground troops, and as designed would require two engines, one a Homeworld style space sim, the other a more traditional 3D RTS engine. I'm trying to think of ways of using almost identical frameworks, with minor changes to AI, pathfinding, UI, etc to produce two engines with minimal work.

Quote:

Both need to push polyons on to the screen, and the graphics engine shouldn't care if its doing the polygon pushing for an RTS or an FPS. The polygons may be generated, culled, or dealt with differently in a different area of the engine. Why rewrite the graphics render between an FPS and an RTS if it could do what you want if the engine architecture was designed with flexibillity in mind?

That's absolutly right, that's what DirectX and OpenGL are for.

If you go a little step upwards:
The efficiecy you are rendering depends very much on the computer architecture.
So if the architecture changes you must change your algorithms that means you must rewrite the engine at it's core!

For example terrain rendering:
Some years ago it was popular to use clever LOD-algorithms like ROAM to render huge landscapes in realtime. Their approach was to send as few polygons as possible through the graphics pipeline. Calculating this did load the CPU but the bottleneck was the graphics processor so doing that made sense.
Today the graphics pipeline is much more powerfull. So the algorithms changed. Their approach is that is's better to send some unneccessary polygons than to waste CPU speed.

You see what I mean?

Constantin

The industry is heading much more towards a middle-ware mindset as the production gets more an more expensive. Of course there will ALWAYS be some modifications made, unless the game itself is so stock-standard that its completely unoriginal. I don't necesarily believe that we'll one day be able to build a game entirely from modules like playing with lego bricks. But we've been seeing more and more middle-ware becoming standard. Things like Havoc, renderware and entire engines like Source, Doom3 and UnrealEngine3. The simple fact is that developing every bit of technology yourself is becoming unfeasable for large productions. Not only would it cost more up-front to build this kind of technology, but it would put a huge hit on the schedule. I'd imagine that some of the Big Guys like EA have a sort of internal middleware, that is, some reusable technologies that are used company wide, but EA is huge and can afford to have a team devoted to developing such things. Most companies don't have those resources and thats where middle-ware components fit.

I think your idea is a little ambitious, in addition to that, no one has a crystal ball, so an old interface is highly unlikely to be able to expose new internals. Not that its impossible, in fact its EXACTLY how DirectX works, you can see what I mean if you're familiar with DX or COM. Basically you can request the newest interface, which is (supposed to be) more or less compatible with previous interfaces but extended, in addition the old interfaces are always available just in case compatibility is broken. However, DX is at a fairly low level of operation, As you get higher up the functionality tree its going to become exponentially harder to maintain a similar system.

My advice is to simply focus on writing good, reusable code. Actually spend some time to engineer your game and seperate out logical components. I have some components that I reuse for things like 2D and 3D graphics, input, system functions, 3D maths and other things all in my own library. Be sure to document it too, Doxygen is wonderful and you never know when you might have to explain it to someone else.

I see where your coming from yes.
But why would such a change to the terrain engine effect anything but the terrain engine? Why rewrite the entire engine because the new hardware architecture demands different approaches of a single aspect. (I understand that more than one aspect will be changed by such a change :). )

I suppose i'm suggesting more a framework with basic functionallity than a fully fledged engine, so that the glue that holds the engine together can be changed easily and cheaply without major engine retooling. So a programming team can get used to the behaviour of the engine, and know what to expect when making changes, does that make sense?

Ravyne, The COM model was what I was thinking of in a way, I am familiar with the system and I understand it to a degree (tho still find myself writing code with my 'fingers crossed' occasionally).

I have no doubt development houses use reusable code, they'd be stupid not to, all companies have such a thing as a 'toolkit' whether they produce code for 'outside consumption' or internal use only often used procedures functions code etc are universally stored in some kind of code toolkit in my experience. Take a look at the ID Doom3 SDK, it has an IDLIB project included which i assume (i haven't looked closely) is atleast part of an in house ID code toolkit, probably in the form of a static link library.

The basis of my idea was to have a rigidly defined way that the components of the engine communicate, so that any one or more components of the engine could be changed as long as they communicated in an expected way.

The way engines seem to be written is that you write a new engine and drop in old code to give functionallity. I'm suggesting writing an engine so that you drop in new code to add extra functionallity.

Eg. The brakes, chassis bodywork etc on your car are great, but the motor isn't powerful enough. Do you build a new car with a bigger motor or replace the motor in your existing car. Even if your new motor requires a stronger chassis and more powerfull brakes, is it better to enhance the chassis, replace the brakes and the motor or build a whole new car?

Thats basically what I'm saying with the reusable modules. A terrain engine, if properly enginered, could be a logical component. However, you must be carefull of how it is designed, for example, an AI module may require access to terrain data for pathfinding, the physics module for collisions, and the audio module for various environmental filter effects. Each of these modules expect the data to be in a certain format regardless of who has responsibility of its storage. How do you propose to extend the terrain data in a logical manner while still maintaining compatibility with the other modules? If you were to, extend the data with another container (BSP, Octree, what have you) you'll likely kill your performance, and if you change the data itself, you may have to update ALL the interdependant modules, which is essentially working from scratch, which is what you wanted to avoid in the first place. Again, all this is entirely possible, with a well engineered system but the design process would be a massively complex operation.

I'm just saying that middleware/reusable code will give 90% of the benefit at a fraction of the cost for such a highly extensible system and you'll never be able to avoid custom code entirely for any game worth making.

Quote:

Original post by Citizen Pain Eg. The brakes, chassis bodywork etc on your car are great, but the motor isn't powerful enough. Do you build a new car with a bigger motor or replace the motor in your existing car. Even if your new motor requires a stronger chassis and more powerfull brakes, is it better to enhance the chassis, replace the brakes and the motor or build a whole new car?

Thats precisely what I'm getting at. At some point, when one change leads to a series of changes it becomes more timely, less expensive and much less of a headach to buy/build a whole new car, the results of which will likely be better than the shoe-horning that has been done on the old one. Of course, you can be sure to get at least a few generations out of the old car, which is absolutely worth it.


You should take a look at the HL2 SDK, you can get it off of steam if you own HL2 (Which you should anyways, Best Game Ever!) They put on a presentation about Modding and the Source Engine in general at school that I attended. They have some really interesting stuff, such as a flexible inter-entity messageing system. If you want to see some well enginered code its definately the place to look. Its downright beautiful.

Ravyne

The custom code point accepted, preaching to converted etc :).

Using the Terrain engine as example, why? Collison detection needs to know whether something hits something, not really a terrain issue in my mind so we can throw that away, (you may disagree, perhaps i missed the point?).

If you have to update dependent modules because of an internal change to how the terrain is rendered/stored then you've not programmed in an OO manner, maybe you've lost me, but I can't see how simple encapsulation would kill your performance, its the essence of OO.

All your examples, pathfinding, collison detection and the sound engine should all request data and be given said data in the format expected, again the very essence of encapsulation. Also if the basics of encapsulation were to cause a major performance hit then the design for the module in question is, too my mind, unforgivably poor.

My idea is not to remove custom code, but to make it trival to 'add in'.

Could you elaborate on what you mean?

I desperatly want HL2, but my games machine is not connected to the net, nor will it be if i have my way, i prefer to download patches etc and ensure their okay before letting them lose on 'my pride and joy' :p.

Since you need a net connection to play HL2 I'm outta the loop so to speak, extremly annoying since i've been waiting for HL2 since i completed HL :(.

Quote:

Original post by Ravyne Thats precisely what I'm getting at. At some point, when one change leads to a series of changes it becomes more timely, less expensive and much less of a headach to buy/build a whole new car, the results of which will likely be better than the shoe-horning that has been done on the old one. Of course, you can be sure to get at least a few generations out of the old car, which is absolutely worth it. You should take a look at the HL2 SDK, you can get it off of steam if you own HL2 (Which you should anyways, Best Game Ever!) They put on a presentation about Modding and the Source Engine in general at school that I attended. They have some really interesting stuff, such as a flexible inter-entity messageing system. If you want to see some well enginered code its definately the place to look. Its downright beautiful.

I'm going to try and put my thoughts a little more succictly.

I accept that an engine that will never need to be thrown away and replaced is a pipe dream, to use an example compatible with my opening post:
Although i believe it could have been possible to write the quake engine to be extended to the point it could produce Doom3, I also believe the Doom engine could catorgorically NOT produce Doom3. Why? Because Doom is essencially a 2D FPS, which is so far removed from 3D FPS's at a fundemental level that no engine no matter how flexible could be extended in this way.

However, many of the fundementals for Quake are identical to the fundementals for Doom3, likewise many of the fundementals for ANY 3D game are identical to the fundementals for any other 3D game.

I'm suggesting instead of having lots of 'nuts and bolts' that you throw in to your new game engine 'chassis' to hold it together, why not have a fully working chassis that can plug in the new bits, and replace old with minimal effort?

Instead of adding old to new (tried and tested to new and experimental), add new to old (new and experimental to tried and tested).

All cars have the same components, its the components themselves that determine the car.

Quote:

Original post by Citizen Pain Ravyne The custom code point accepted, preaching to converted etc :). Using the Terrain engine as example, why? Collison detection needs to know whether something hits something, not really a terrain issue in my mind so we can throw that away, (you may disagree, perhaps i missed the point?).

You need to know what the terrain is like to know if/where/how you've collided with it. Its a data locality issue, if the terrain engine "owns" the terrain data, the collision must request/gain access to it through the terrain engine.

Quote:

If you have to update dependent modules because of an internal change to how the terrain is rendered/stored then you've not programmed in an OO manner, maybe you've lost me, but I can't see how simple encapsulation would kill your performance, its the essence of OO.

You're right that proper OO practices should allow such things, but enforcing such a pure and strict OO methodology certainly will cause a performance hit. It may not be that huge of a hit in all honesty, it depends on what and how much of it goes on.

Quote:

All your examples, pathfinding, collison detection and the sound engine should all request data and be given said data in the format expected, again the very essence of encapsulation. Also if the basics of encapsulation were to cause a major performance hit then the design for the module in question is, too my mind, unforgivably poor.

Then, it seems to me that you'll need yet another subsystem, a terrain manager, to arbitrate data to each of these systems. It would have to shuffle data about and be responsible for interpretting data to and from each of the various formats, which will be a source of additional overhead. As for module design its simply impossible to design a system that will allow for all future posibilities without having a very fine granularity. And when you start layering virtual interfaces upon virtual interfaces and pervading the internals with virtual function calls the performance hit will really add up. Having such a robust system naturally calls for so many indirect calls and data accesses that its going to cause all sorts of overhead as well as miss-jumps, which are death on modern, deeply pipelined CPUs.

Quote:

My idea is not to remove custom code, but to make it trival to 'add in'. Could you elaborate on what you mean?

Your idea is fine, and certainly doable. I just believe you're focussing on the wrong end of the equation. Its not that you can't have such "drop-in" compatibility, its just that it doesn't provide so much extra benefit to traditional code reuse that it justifies the cost and effort, both in terms of development of the architecture and run-time overhead. Its simply diminishing returns. As I said before, much of the benefit can come much cheaper by simply designing carefully architected subsystems. Alot of stuff gets reused already, HL, Quake 2 and 3 all contained code from quake 1 for isntance, alot of it was either external libs or simply cut and paste. Sure some things were tweaked or changed altogether, but alot of the graphical enhancements simply came by virtue of having a higher minimum spec to throw more polygons at.

At the time there wasn't any major paradigm shift in the way things happened, for instance with the graphics pipe. All of those games rely on a fixed-function 3D pipe for instance(Q3 being a possible exception, not sure.) But when things like vertex and pixel shaders became common, it would have required a major change to the codebase to support. The next big changes coming are a unified shader pipeline and hardware tesselation(useful for procedurally generated terrain, among other things) coming with DX10. You simply cannot forsee such major shifts, ensuring a system that will cope with such new means is a difficult task. While I believe that Polygon-based hardware will be the norm for quite some time, what would happen if in 5 years polygon renderers are replaced by hardware accelerated ray-tracers? Such things exist, but not in a consumer product, much as 3D acceleration once existed only in scientific, cad and graphics applications. Progress happens quickly and often takes unexpected paths over even short amounts of time.

Basically I don't think your idea is impossible or even bad in general, but I think that your efforts could be better spent. No design will ever be so perfect that it is essentially timeless.