Designing a 3D engine.

There are very few books and tutorials about this matter... Most game programming tutorials will mainly concentrate about superficial matters such as rendering methods for very specific things (ie: particle system tutorial, terrain rendering tutorial).

You basically have to decide it all yourself. There are references, such as the Quake engine which is now open source. Some websites will also provide you information about important decisions you will need to make.

Planning it all from the start is a good idea. You should plan early on what techniques you will use for everything. You should be able to tell how your map files will store data, how the engine will internally store it and render it (basically). You should also examinate the pros and cons and pick a texture format for example (please dont design another custom one).

Of course all this planning does not mean design cant change later on... Because it will need to change, either you want it or not.

Yup, plan before you code anything. That''ll save you from rewriting code a hundred times if you aren''t satisfied with the way things work. While writing my engine, I didn''t plan some of the things and that got me into a couple of problems. One of them was deciding a world file format. Decide on your file formats (for world geometry, dynamic characters, etc...) before you get into the gory details of programming the engine.

quote:

---

Original post by poly-gone
Yup, plan before you code anything. That''ll save you from rewriting code a hundred times if you aren''t satisfied with the way things work. While writing my engine, I didn''t plan some of the things and that got me into a couple of problems. One of them was deciding a world file format. Decide on your file formats (for world geometry, dynamic characters, etc...) before you get into the gory details of programming the engine.

---

I think though that with the file formats, they should match closely the data structures used within the engine internally so that you can almost do a straight memcpy from the file to the datastructs. Obviously in some cases some processing will be required to allocate memory and such like. I therefore think that in some cases it''s better the other way around. Implement the file loading last, after sorting out your world data-structure so that you can minimise the loading time.

An example of this is the Quake3 BSP file format. I''ve seen many people(including myself) load just what they need and copy the info from each struct, this results in quite a long loading time. If you can use the datastructures in the way they are stored in the file, then it is simply a memcpy for each chunk of data, and you''er loaded much quicker.

Take a look at the enginuity series in the featured articles. In particular check out the first page of the second noe. This includes a block diagram which shows the engine structure for a client server modelled engine.

Actually the loading time is not really due to the level data itself. If you take a good old game like Quake 2, and run it on a modern PC, you will notice that most levels, even the largest load in a second or less. The problem is that Quake 3 textures are compressed as jpeg, and that Quake 3 stores all it's data in compressed zip files... This is no load time optimization.

Modern machines are fast, and file loading is optimized... Your operating system will do some read buffering to increase speed. Personally I chose to use a text format for my levels because it was easy to modify without a level editor or test things that are not yet implemented in the editor. If I ever feel like it is too slow, my design will allow me to easily switch to a binary level format in a matter of hours. I believe Id Software tested doom 3 using a text map format, I dont know if they will change this in the final version, but thats how it was in the leaked beta .

What I suggest is to make some pseudo code, and pseudo data structures showing basically how things will be layed out. You probably wont be able to do a straight copy from the file to the memory, unless you really manage to make every variable a multiple of 32 bits in size.

Basically, a simple map file could look like this:

- File header (contains number of polygons, number of entities)
- Generic map information (map name, author, date of release, map specific settings)
- Polygon data (either a list of all polygons, or a more elaborate structure, like a pre-compiled BSP tree)
- Entities (most likely a list of entity descriptions, Quake did this in straight text)

Your level could also include things like lightmaps in your level data (if you plan on pre-compiling static lighting). Things like deciding for or against static lighting are the kind of choices you should make at the start.

If this is your first 3D engine... I would suggest perhaps following the road Quake 2/Half life took, that is:

- Binary map file
- Pre generated BSP tree in the map file
- PVS data for each tree node
- Pre compiled lightmaps in the map file
- Entity data layed out as text in the map file
- Uncompressed textures, such as BMP or TGA files
- Frustum/PVS based culling method

This is a basic design that has proven working and fast. When you have completed something that works well, then you can start thinking about more innovative things (such as 100% dynamic lighting, eliminating PVS, occlusion culling). This would also force you to make some tools, such as a basic map editor, a BSP tree/PVS generator and a radiosity tool to compile your lighting information (raytracing is very cool :D).

[edited by - Max_Payne on August 29, 2003 1:53:06 PM]

quote:

---

Original post by Max_Payne
Actually the loading time is not really due to the level data itself. If you take a good old game like Quake 2, and run it on a modern PC, you will notice that most levels, even the largest load in a second or less. The problem is that Quake 3 textures are compressed as jpeg, and that Quake 3 stores all it''s data in compressed zip files... This is no load time optimization.

---

I didn''t mean in the actual Quake games, I meant in loaders that people had made. I believe that the quake formats are designed so that you are able to just memcpy across the chunks into the respective arrays without any processing. The load times I was talking about are when people start having to do work on each individual face, vertex and node. This is what takes some time and is not really acceptable. The reason why you''d want to be able to just memcpy your data across is this makes streaming level data very easy. You can just copy the appropriate array from teh file into the local cache and that''s it. No processing of the data is the target.

Why does the variables have to be multiple''s of 32 bits? Even if you did require data to be 32 bit aligned you can easily do this with padding bits in your structures.

Just make sure that you plan the whole thing out on paper before you start developing. Then after you think you''re done planning, walk away from it for about 2 days, then come back and glance it over. If you''re like me you''ll probably find some mistakes in the design. Before implementing anything in your engine, plan out how you will do it, and what algorithms you plan on using.

It also helps to draw out a diagram of how the systems in your engine are related to each other.

You probably have one already, but make sure you incorporate a nice stable logging system into your engine. It is a god-send for those little obscure bugs that surface later on.

quote:

---

Why does the variables have to be multiple''s of 32 bits? Even if you did require data to be 32 bit aligned you can easily do this with padding bits in your structures.

---

Padding bytes are stupid, and I will let you figure out why.

My Q3 levels load fast. I simply allocate arrays and fread straight into them, no preprocessing (except for switching up vertex axes...that still rubs me the wrong way and I wish I could simply not do it, but the weird axes bother me more than the extra second of loading time). On the other hand, I keep my textures as TGAs (RLE or not, depending) and just let them float free instead of being locked in a pak/zip.

Planning is the key. It doesn''t matter so much how the engine is built (well it does, but usually anything that makes sense and scales well works) but that you plan it and make sure the architecture is consistent.

It''s true that there are few resources that deal exclusively with Game Engine architecture. One fantastic book however is David Eberly''s "3d Game Engine design". I strongly recommend you check it out. Also, go to Magic Software, the companion site to David''s book. There''s a pretty complete game engine with full source available that will probably help you a lot (when combining it with his explanations in the text. Also, download the free OGRE engine. It''s a fully Object-oriented C++ based graphics engine, with a nice architecture, fully commented code, and full source code available.

Good luck! Don''t get discouraged either - once you start, you begin to realise what a mammoth undertaking an engine is. Just try and break things down into bite-sized chunks and TEST each of your new classes/modules thoroughly before you get excited and move onto the next step.

Cheers,
DChavez out.

quote:

---

Original post by Max_Payne
Padding bytes are stupid, and I will let you figure out why.

---

I might be missing something, I don''t see why it''s stupid... let''s say you want to keep it 32-bit aligned, and you have 3 variables with possible values of 0-255. Should I allocate 3 32-bit integers, and use 12 bytes? Or should I use 3 bytes and pad it with one, using a total of 4 bytes and still keeping it 32-bit aligned?

quote:

---

Original post by Max_Payne

quote:

---

Why does the variables have to be multiple''s of 32 bits? Even if you did require data to be 32 bit aligned you can easily do this with padding bits in your structures.

---

Padding bytes are stupid, and I will let you figure out why.

---

Padding bytes does have a legitimate use. I know in DirectX that retrieving data from vertex buffers is most efficient at 24/32 bits. If you have a vertex format that is 28 bits you could get a significant boost by adding an extra float/int in there to make it 32.

quote:

---

Original post by Raloth

quote:

---

Original post by Max_Payne

quote:

---

Why does the variables have to be multiple''s of 32 bits? Even if you did require data to be 32 bit aligned you can easily do this with padding bits in your structures.

---

Padding bytes are stupid, and I will let you figure out why.

---

Padding bytes does have a legitimate use. I know in DirectX that retrieving data from vertex buffers is most efficient at 24/32 bits. If you have a vertex format that is 28 bits you could get a significant boost by adding an extra float/int in there to make it 32.

---

uhm a float is 32 bit. 28 bits + 32 bits = 60. I think you mean bytes

My Site

If you are doing an infinity project as an hobby premature optimization is the root of all good.

In other projects it is pretty much the reverse.

http://www.extremetech.com/article2/0,3973,594,00.asp
no actually but alot of explanation about the engine design and buzzwords associated with building one.

http://www.gamedev.net/hosted/3dgraphics/
has the basics of a software 3D engine.

http://www.fastgraph.com/tutorial/fgtut03.html
explains 3D concepts using their API but helpful.

[edited by - Alpha_ProgDes on September 4, 2003 9:46:19 AM]

It''s probably a good idea to have the lights reside in your scene graph tree. This will allow to be flexible in terms of deciding which child nodes get affected by lighting calculations. otherwise you might end up recalculating lighting for everything in your world each frame, which you may not want to do.

DChavez out.

quote:

---

uhm a float is 32 bit. 28 bits + 32 bits = 60. I think you mean bytes

---

Hehe, yeah I''m sure he did. A typical 2D vertex, XYZW(16) + RGBA(4) + UV(8), is 28 bytes. Adding an extra int would pad it to 32 bytes, which is apparently good. My vertex format is 36 bytes: XYZ(12) + Normal(12) + RGBA(4) + UV(8). That''s a nuisance but oh well.

~CGameProgrammer( );