creating a custom 3d format

Haven't stumbled across any really thorough ones, and I don't think the whole process is something that would translate to a good "tutorial", as they only really serve as short to-the-point introductions to solving a small task. What is involved in creating a custom format depends greatly on what you intend to use it for.

The process is usually more or like the following (I'm intentionally trying to break it down to small items to show what needs to be done):

- You decide what data you need in the format. This depends of course on which art authoring software you use and what are the capabilities of your render system. You obviously want geometry (vertex+index data? different primitive types? special attributes like tangent frames, per-vertex colors, several UV sets?) and animation (bone structures? animation sequences? morphs?), but is that enough?
Do you need the ability to refer to other assets in the format file? (Geometry -> Materials -> Shaders, Textures).
Are those references to external files (Geom file refers to .jpg, .fx etc.) or do you need to embed all other data inside your custom format as well?
Do you need to leave some room for game-specific data? (geometry physics parameters, weight, etc.)
Do you need to store the whole game world in a custom format? (World scene file that contains all the geometry, scripts, triggers and other game information)

- You decide how to store the data in the file. If you're totally ignorant about space usage or if this is not the format for end-use ingame loading, you
might just use ascii (see www.collada.org/ for example).
Does your data file contain just a single resource, or do you want it to be more like an archive containing lots of files of several types?
Do you need to support fast loading? If so, you need to serialize the data in a way so that it's already in the right format and ready-to-render when you stream it in to the game. Or
Do you need to save on disk space? If you're limited on HDD usage, or if the target platform's disk access speed is slow, you might want to compress the data in some way (lots of lossy and lossless methods).
Do you need to detect data corruption or protect against anyone decompiling your data format? You might consider using checksums and/or crypting it.
Do you need to support several platforms and specialize your format for each? Then you're not looking at a single format file, but a family of different formats specialized for each platform.

- You decide which programs need to export the data. Familiarize yourself with the exporting APIs of those programs (3DSMax IGame export interface for example) and figure out how to write a plugin for the program and how to pull the data you want using those interfaces.
Do you need to support multiple authoring software? Or do you need your game software have an export/build mode as well? You need to make sure that they all export data in the same way.
As for the serializing part itself, the task is just about writing bytes into a file in a certain order that you've specified. Just make sure that it can be unambiguously read back to recover the original data.

- You decide which programs need to import the data. Your game obviously does, but do you need to get the data reloaded back to the art tool? Or do you need to pass the data between several art tools (3DSMax -> Blender)? Then you need to familiarize yourself with the import plugin APIs of each software.

For your game, the deserializing part is essentially doing the reverse of what the serializing code does. Read bytes from a file according to the schema you've laid out to yourself, there's nothing overly difficult or magic to it. The challenge comes from the huge amount of engineering that is needed to solve some of the issues I listed above.

To make a simple geometry format, you can just dump vertex and index data to a file and read it in your program, which isn't that difficult serialization-wise. The bigger hurdle is learning how to use the export/import interfaces of particular art tools to get out the data you want. Most programs have a good reference documentation, but it can be quite difficult at first to look at it. There are some bits you can google, e.g. for IGame: Exporting Animation with the IGame Exporter

In the end, I guess it comes down to doing a cost-benefit analysis whether it'd be just quicker and sufficient to use .x or collada or .3ds or something else.

Since I made a 3D file format and an exporter for Maya, I have a bunch of Maya tutorials laying around.
I really doubt you need a tutorial to tell you how to structure the file or what should be in it; you can decide that on your own and there is no right or wrong answer. The real work is just in making the exporter.


http://www.robthebloke.org/research/index.htm
http://florian.loitsch.com/gpExport
http://www.gamedev.net/reference/articles/article1906.asp
http://hohehohe2.hp.infoseek.co.jp/openMaya-eng/0mokuji.html


L. Spiro

Hi,

I was on the same quest a few months ago.
And I didn't find an "all in one" tutorial.
I asked here for some specific advice, but haven't got much (actually none).
But there is info out there if you look for it.
After learning a lot about models and animations, I finally got it where I wanted.

It's important to understand well the theory behind all this, because otherwise you will end up with incorrect skeletons or bad animations. And most of the time it's very hard to know what's wrong. It looks "almost" right, perhaps a bad 90deg rotation, or a flipped bone, but you just can't nail the real problem.

Anyway, here are some tips on supporting models and animations:

1. Don't expect all models or animations assets (found on net) to be correctly exported. Always have a few other viewers to check them. Especially the skeleton and the animations, if you can see them.

2. Have a good math module, with correct conversions from quaternion to matrix etc. Compare your functions with other math libraries.

3. Make sure you know how your math and render matrices are stored and what's their multiply order. I'm talking about OpenGL vs DirectX vs 3DStudio Max coordinate systems.

4. Make sure you know the vocabulary: local, global, offset (InverseSkinningPose) matrices, skinning pose, initial pose (these can be different), source and instanced models.

5. Make sure you know how vertexes are transformed during skinning and rigid animations. And by what matrices. Vertexes can be in bone space or in model space. Also you can have the whole thing in a same mesh (best for gpu skinning) or you can split it in rigid sub-meshes.

6. Decide what you want to keep in the bones: matrices or quaternions, etc.

7. Consider the fact that you might want to attach some other models to some bones, or to attach the whole model in a scene graph. I have a class for Node and derived the Bone from it. The skeleton keeps a list of bones, linked hierarchically using child and sibling Bone pointers.

8. Decide how you want to split your model format. Do you want the sub-meshes in separate files? Do you want separate animation files or a separate skeleton file? This is very important for reusing the assets on more instances and for combining them (like having soldiers with the same skeleton and animations, but different meshes or materials). Make sure you know how your models management will be implemented to support such features.

This was the toughest part for me and in the end I decided to use the model format as a container for all mentioned elements: one skeleton, more anims, more meshes with material names. The management (might be very different from game to game) will be implemented per game and will be able to spawn instances that combine these elements.

9. Use the Assimp library for importing assets and then save in your own format. Don't bother to write model format converters. And not even an exporter plugin. Google for Open Asset Import Library. For me it was very useful and I got the models loaded with not much effort.

10. Plan for a "model compiler tool" to import, adjust or combine assets and save them in your format. Mine can get the mesh and skeleton from one .x model for example, and add animation from other .x models if this is how I get them. Also it can scale the geometry and adjust animations speed.

11. Speaking of models, get some great looking ones. I ripped some nice character models from a popular adventure game to test with. This makes the whole thing look great and motivating. Much better than cubes or DirectX tigers.

12. As a final and most important advice, don't try to support everything! Decide first on what you really need in your game(s). Things can get very complicated (and hard on cpu) with some features like multiple blending and additive animations, face morphing, etc. If these are not the main feature in your game, try to avoid them as much as possible.

In the end, it really worth having your own model and animations support. That is if you can manage to implement it well...

If you need more ideas, let me know.
Good luck with it!
Alex