Two questions, for lead programmers ???

Our core low-level engine team has varied over time from 1 to about 4. Usually only 1 or 2 of these are graphics related. For atleast half of my 6 years, I've been the sole person on the engine team, mainly going graphics. The team increased in size for few months when we did the DX8/XBox to DX9 switch. The most important part of that switch was attempting to move anything API specific out into seperate modules, leaving nearly all of the remaining code to use useful for any platform.

We have a somewhat API agnostic interface, and underneath we can implement D3D8, D3D9, D3D10, OpenGL, or anything else that sticks fairly closely to some simple concepts. We also allow the game and other parts of the engine to access the underlying API to include platform specific features. When switching APIs there are only a few core D3D specific things. Textures, Surfaces, Vertex/Index storage, Drawing, and effects. These concepts will exist in every API, but only the implmentation details change. Our animation system, collisions, cameras, math libraries, etc, don't care about D3D. Even mesh loading doesn't care. It loads data and uses the platform agnostic interface to request somewhere to put the mesh. The renderer calls an agnostic API to activate whatever effect is needed, and calls another agnostic API to actually draw from the storage.

Rewriting code can be wasteful, or very good. If the code is slow, it may need a rewrite, or even a re-engineering, using another technique to solve the same problem. If the code has become so convoluted that it's hard to follow, for debugging or adding new features, perhaps a cleanup is in order. Try to keep in mind all those features you've been meaning to add and ensure the new design accomodates both the old and new features.

Thanks for your reply!

Did your core team also developed your tools and leveleditors ?

Quote:

Original post by Namethatnobodyelsetook We also allow the game and other parts of the engine to access the underlying API to include platform specific features.

Hm, isn't allowing other parts of the engine to access the API directly going against encapsulation and abstraction of the API. Isn't some kind of API agnostic wrapping needed even in that case.(at least a really simple one)

Ideally API change should include just a rewrite of that API agnostic module. No other part of the code should need to be changed.

Quote:

Original post by Radan Ideally API change should include just a rewrite of that API agnostic module. No other part of the code should need to be changed.

I take it you have never worked on a console? :)

Editors and tools were other people, not the core low-level team. The amount of platform specific things in the editor is very limited, and platform specific code is mainly isolated to a few CPP files.

The reason to allow a game to get past the abstraction is because, as an abstraction, it's taking some detail away, somewhere. (Read more). There will always be something super cool that one platform or API has that isn't available elsewhere. Should I arbitrarily decide games can't use that feature because it's not available on all hardware? Should I write an abstraction for something which may never be seen on another platform/API?

The code MUST be tailored to the target API/platform to some degree. I try to keep 95% of that hidden behind the engine API, but certain things make that impossible. There are common rendering ideas that if used on an XBox360 would kill performance. There are other things possible on the 360 that a PC would have trouble with. A game is going to have to understand how to load-balance it's work on the Cell on the PS3. I can't hide these facts. The majority of code doesn't have to change though, and that's what we aim for. Platform neutral whenever possible.

Quote:

Original post by Namethatnobodyelsetook The code MUST be tailored to the target API/platform to some degree. I try to keep 95% of that hidden behind the engine API, but certain things make that impossible. There are common rendering ideas that if used on an XBox360 would kill performance. There are other things possible on the 360 that a PC would have trouble with. A game is going to have to understand how to load-balance it's work on the Cell on the PS3. I can't hide these facts. The majority of code doesn't have to change though, and that's what we aim for. Platform neutral whenever possible.

Nice to hear I'm not the only one doing this - our company has very specific holes punched through the API to get access at platform features - the consoles are just too powerful/wacky/useful to give up on them.

Quote:

Original post by NamethatnobodyelsetookThe code MUST be tailored to the target API/platform to some degree. I try to keep 95% of that hidden behind the engine API, but certain things make that impossible. There are common rendering ideas that if used on an XBox360 would kill performance. There are other things possible on the 360 that a PC would have trouble with. A game is going to have to understand how to load-balance it's work on the Cell on the PS3. I can't hide these facts. The majority of code doesn't have to change though, and that's what we aim for. Platform neutral whenever possible.

Sorry 'bout the thread hijack, but this got me wondering. Exactly how do you deal with varying numbers of processors or cores in keeping platform neutrality? Obviously you could schedule more tasks on a single hardware thread if you have fewer processors available like on a typical PC, taking advantage of the OS scheduler. But still, I'd imagine you might need to write bigger or smaller tasks depending on whether you're dealing with 6 or 2 hardware threads. At some point, this could cut deep into your engine's API.

I don't know if it's really an issue, but it sounds like it might be a big pitfall in 'nextgen' cross-platform development. Has anyone already encountered problems with this? If so, is it a serious issue and what would be a good way to tackle it?

Yeah, I never worked on the console. I'm stuck with the PC.

Actually, punching holes to the API makes sense. It would be nearly impossible to abstract everything away.

How do you generally go about "punching holes"?
For example, in a higher level class that needs access to the API, would you just add "#include " or would you add a passtrough inlined function in the abstraction API that actually just passed the data trough to the actual API function?
First is simpler but the second one should be easier to maintain as the API code is still kept in the same namespace/class even though it actually isn't abstracted.

Do you have a specific system to mark the places in code where API abstraction is bypassed for easier maintance or API switching?

Quote:

Original post by remigius Sorry 'bout the thread hijack, but this got me wondering. Exactly how do you deal with varying numbers of processors or cores in keeping platform neutrality? Obviously you could schedule more tasks on a single hardware thread if you have fewer processors available like on a typical PC, taking advantage of the OS scheduler. But still, I'd imagine you might need to write bigger or smaller tasks depending on whether you're dealing with 6 or 2 hardware threads. At some point, this could cut deep into your engine's API. I don't know if it's really an issue, but it sounds like it might be a big pitfall in 'nextgen' cross-platform development. Has anyone already encountered problems with this? If so, is it a serious issue and what would be a good way to tackle it?

This is an issue for sure and the general solution is to use your own lightweight job management.

“taking advantage of the OS scheduler” is a bad idea as it requires that you use more threads than cores and threads are heavy kernel objects. If you force a core to switch between multiple threads all the time you will lost CPU power.
But a user mode job manager generates some overhead per job too. This is a dilemma as if you made jobs large there is a risk that you would not have enough jobs for all cores. If you made them to small the overhead would be larger than the additional CPU power you gain. There are even more problems especial when it comes to window handling and graphics APIs.
Consoles can be even trickier but at least you have a fixed hardware in exchange.

Thanks for the insights Ralf.

The situation you described is indeed what I was getting at. It seems that you're either stuck with too many small jobs for the PC, or too few big jobs for consoles. I think I understand what you mean with the usermode lightweight job manager. I guess I'll have to give it a go in XNA as soon as the XBox360 wireless lan adapters finally ship again over here ;)