Dear All,

I'd like to introduce you a new model format. It is simple, well-defined and well-documented, and has the best data density of all formats. Suzanne with normals, UVs and materials require less than 12K using this format.

I'm not good with names, so it's just simply called Model 3D and has the extension ".m3d". I'd like to get some feedback and hear your opinion about it.

Provided implementations

For compatibility, it is implemented with Assimp. Unfortunately according to my findings Assimp fails to live up to it's promise "Loads 40+ 3D file formats into one unified and clean data structure.", as its data structure is neither unified nor clean. But I managed it, so here it is, libassimp can load M3D files with vertex colors, materials, textures, and skeletal animations.

A new model format is no good if you can't use it with your favorite modeler software, so I have provided Blender plugin to export objects and their animations into M3D. The exporter is fully functional and supports all features of the M3D format, however the importer is WIP. You can export vertex lists with colors, normals, UVs, PBR materials (PrincipledBSDF), textures, armature, and several model actions too using the Blender Animation's timeline markers feature.

And finally, it's native SDK is an stb-style, totally dependency-free, single header file ANSI C89/C++11 SDK. It can load and save files, supports the binary M3D format (and if included with the M3D_ASCII define then it's ASCII counterpart, A3D too). It is MIT licensed, written in ANSI C and with the provided C++ wrapper class it should be straightforward to integrate into any new or existing projects. It compiles to about 70K, which includes the built-in PNG texture decompressor.

Features of the format

A few words on why does it worth it to use this format:

• it is Open Source and free. Not only its SDK, but the format itself is MIT licensed and expandable.
• it can store big models in very little files. Blender's cube can be saved in about 100 bytes. An usual Blitz3D animated model (.b3d) in .m3d saves 70%-80% storage space.
• exactly one model per file, but with all properties: materials, skeleton, textures, skeletal animations. No complex scene parsing involved. (Lightning and camera setups are NOT properties of the model!)
• but if needed, engine specific data can be embedded in a way that it won't break the model compatibility with other software.
• the same model can be read by different software with different levels. The same file is compatible with more applications, for example: one can only load the static mesh from it, while another can also utilize the bone information and the animations.
• it has a human readable ASCII variant, which can be easily loaded with a slightly modified Wavefront OBJ loader (however the SDK can handle).
• it's SDK is simple, fast and painless to use (according to the K.I.S.S. has only 5 functions altogether).
• it's in-memory format is efficient and fast (uses indices in contrast to recursive node parsing matching strings), and well-specified (for example it makes it clear that right-handed coordinate system is used with the coordinates, and how the coordinate system is oriented. No surprises when you load a model from M3D).

I've provided a command line utility (m3dconv) which loads any Assimp-supported format and converts it into Model 3D. A simple, portable GL-based viewer (m3dview) is also included in the repository to demonstrate how to display the animated models once you loaded them (whole thing less than 500 SLoC :-) ). You can compile the latter with GLFW, GLUT and SDL2 (the interface is autodetected, just run "make").

Please let me know what you think about this project and the file format,

Cheers,
bzt

13 hours ago, bzt said:

A new model format is no good if you can't use it with your favorite modeler software, so I have provided Blender plugin

I won't call Blender to be favorite but if you really want to push your stuff, Maya, Houdini, 3D Max and ZBrush are must haves for you because those are the industry leading ones for games.

 

13 hours ago, bzt said:

it has a human readable ASCII variant

How fast is parsing performed? High compressionratio most of the time leads to slow processing performance so I'm corious if it can be as fast as FBX or OBJ (as you mentioned).

 

13 hours ago, bzt said:

it's SDK is simple, fast and painless to use

How complex is the code for processing the file and what about flexibility? Do I have to integrate Assimp into our engine just to use your file format or do you also provide code that can easily be grabbed and placed in our source without the need to make huge adaptions for us to work?

Hi,

Thank you for your reply!

5 hours ago, Shaarigan said:

I won't call Blender to be favorite but if you really want to push your stuff, Maya, Houdini, 3D Max and ZBrush are must haves for you because those are the industry leading ones for games.

Well, are they Open Source? Until then you can use the m3dconv utility to convert their formats into M3D in your build system automatically.

 

5 hours ago, Shaarigan said:

How fast is parsing performed? High compressionratio most of the time leads to slow processing performance so I'm corious if it can be as fast as FBX or OBJ (as you mentioned). 

Actually a LOT faster than any FBX or OBJ loaders out there. However the ASCII variant A3D is for debugging purposes. For your game, you should use the binary variant, M3D which loads as fast as possible. There are no unecessarry string conversions, everything is read right from binary and all chunk sizes are known in advance (so it allocates memory as few times as possible). Profiling showed that the slowest part is performing the post-processing phase, calculating the transformation matrices for the bones, but a) that's still pretty fast, b) you can avoid that if you include the SDK with the M3D_NOANIMATION define. That way only the file format decoder will run.

For small file sizes, the binary format is compressed. To further speed up your game loading for the price of bigger storage, you can avoid that compression (see m3dconv), and store only the binary chunks (which are still pretty small compared to OBJ, FBX, .glF or any other XML or JSON based formats.) The m3d_load() autodetects if the binary is compressed or not, and what's more, if it's not then it does not allocate extra memory and uses pointers directly into the uncompressed image.

5 hours ago, Shaarigan said:

How complex is the code for processing the file and what about flexibility?

Both questions are answered on the project's main README.md file. You can load a colored static mesh in 80 SloC if you don't want to use the SDK (which you should, it's a single header file with only one function for loading). The format is also flexible, supports engine-specific chunks (you have to parse those on your own of course).

5 hours ago, Shaarigan said:

Do I have to integrate Assimp into our engine

Most definitely NOT. You only have to include an stb-style single header library if you don't want to parse the files on your own. The Assimp support is only there in case if you already have integrated Assimp, but that's all.

5 hours ago, Shaarigan said:

do you also provide code that can easily be grabbed and placed in our source without the need to make huge adaptions for us to work?

I'm starting to have a feeling that you haven't read my post, nor have you checked the repo.

Of course I provide code:

• an example how to parse the files without the SDK (80 SLoC)
• tests directory contains examples
• m3dconv also uses the API and provides example
• m3dview was written to showcase how to use the API and display an animated model (500 SLoC)
• plus there's the API usage manual, full with examples.

Cheers,
bzt

A few numbers to satisfy your curiousity:

The WusonBlitz.b3d file (provided in the Assimp repo test/models/B3D) is a binary format, 87k. Converted into M3D it's only 29k. Loading the same file using

Assimp: 581,017 bytes in 4,088 blocks
M3D SDK (m3d.h): 229,534 bytes in 4 blocks

As you can see, Assimp requires more than twice the RAM, and it uses more than 4 thousand blocks. That means at least more than 4 thousand memory allocation calls. The M3D SDK on the other hand not just loads the same model into less RAM, but it only calls memory allocation 4 times only. Also the native SDK's memory is less fragmented, meaning less memory wasted. (FYI: fewer memory allocation calls => faster loading time.)

Also M3D SDK stores the model in a more compact way and provides faster access (indices instead of recursive node-walking with string comparititons). The native in-memory format resembles VBOs and EBOs so you can display them with fewer conversions (actually a non-animated mesh can be directly fed into a VBO/EBO).  (FYI: simpler structure with direct indices instead of look-ups loops => faster rendering time.)

All the differences in the Assimp's and M3D's philosophy is explained in the repo, assimp/README.md and the M3D SDK API usage manual is here with both C and C++ examples.

Cheers,
bzt

17 hours ago, bzt said:

Well, are they Open Source?

Defintely not but they provide plugins and are the industry standard. If I have to parse their formats into something you provide, I can on the same track parse it directly into something our engine supports by native so I don't see any benefits of your file format here especially in a real production case.

There might be benefits for indie games that really have to take watch of the space they have for assets on an online share for example but those people won't use a file format not supported by their major engines Unity and Unreal so, im corious what your statement is ?

 

17 hours ago, bzt said:

Actually a LOT faster than any FBX or OBJ loaders out there

Did you see/ test against this very popular implementation on GitHub? I implemented similar code in our engine which makes use of our streaming API and came up with round about 800 ms for the file the person provides on GitHub on a Win7 maschine Intel i7 hexa core x64 OS. Would be corious if you can provide some benchmarks

 

17 hours ago, bzt said:

if you don't want to use the SDK (which you should, it's a single header file with only one function for loading)

As you make use of the STL, especially in the model wrapper, this is against our engine policy, so no I would prefer to not use the SDK, the same for the sprintf's I saw in your code. This makes it difficult to integrate it in our existing systems.

So if I would decide to integrate your file format in our workflow, I would need to rewrite it from scratch to play well with our existing systems and API. This should not value your work (I feel I have to write it into each of my posts when providing a personal opinion today), just a notice from my all-day industry perspective ?

18 hours ago, bzt said:

I'm starting to have a feeling that you haven't read my post, nor have you checked the repo

I have carefully read the post because it might propably be of interest but you wrote about Assimp and the SDK. I don't had checked the repo and still didn't browse through it entirely. If you post it in a forum, you have in my opinion to somehow work with questions ?‍♂️

5 hours ago, Shaarigan said:

Defintely not but they provide plugins and are the industry standard. If I have to parse their formats into something you provide, I can on the same track parse it directly into something our engine supports

No. Seriously, please read at least the README.md on my repo! Seriously no offense, but you expect me to answer your questions, and I will; but in return please allow me to expect from you to at least read the readme file on what this is about.

First, you don't have to parse their formats into something I provide, because there's already a tool to do that for you.
Second, you don't have to parse anything into your engine, you just include one header file and there you go. Are you familiar with stb-style headers? This is what my m3d.h provides for C and C++.

5 hours ago, Shaarigan said:

Did you see/ test against this very popular implementation on GitHub? I implemented similar code in our engine which makes use of our streaming API and came up with round about 800 ms

Yes of course. 800ms that's laughable ? I also got a similar value repeating the test 32 times. Here's a comparition for you.

Model Format File Size Library Required Time WusonOBJ.obj text OBJ 258k tinyobjloader real 0m0.748s WusonOBJ.obj text OBJ 258k libassimp real 0m2.716s WusonBlitz.b3d binary Blitz 87k libassimp real 0m0.414s WusonBlitz.m3d uncompressed M3D 33k libassimp real 0m6.906s WusonBlitz.a3d ASCII Model 3D 141k M3D SDK real 0m0.258s WusonBlitz.m3d uncompressed M3D 33k M3D SDK real 0m0.073s WusonCompr.m3d deflated M3D 42k M3D SDK real 0m0.162s

As you can see my SDK is 10 times faster than your beloved tinyobjloader. Even with deflated chunks it is still 4 times faster. But let's be fair, compare only text formats, so tinyobjloader vs A3D: my SDK is still 2.5 times faster (but this is irrelevant because you should not use the debug format in your engine). Feel free to repeat the test on your machine.

Believe me, if I say my implementation is a LOT faster, then I know what I'm talking about, I've done my homework and I had performed my tests ?

I kindly ask you to read the README.md, and comment only when you have a clue what my repository is about. Thank you.

Cheers,

bzt

5 hours ago, Shaarigan said:

As you make use of the STL, especially in the model wrapper

Again, NO, I do not use STL. You would know that should you have read the API documentation. You can configure the SDK not to use zlib inflate at all, but then you won't have textures either, because that's needed for PNGs too. (You'll have to use your own image decoder in this case.)

5 hours ago, Shaarigan said:

I have carefully read the post

Then why are you asking questions like "do you provide code examples"? Just askin'.

Cheers,
bzt

Oh and two more things:

5 hours ago, Shaarigan said:

the same for the sprintf's I saw in your code

That's only needed for the A3D writer. So IF you specify M3D_EXPORTER and IF you specify M3D_ASCII. By default only the binary M3D importer gets included. Why on earth would you want a model dumper in your engine????

5 hours ago, Shaarigan said:

So if I would decide to integrate your file format in our workflow, I would need to rewrite it from scratch to play well with our existing systems and API.

If ANSI C89 and C++11 standards WITHOUT ANY DEPENDENCY is a problem for your existing systems, then the problem is in your existing systems. No offense, that's the truth.

But if you want to rewrite it from scratch, feel free to do so, go ahead, check out the repository's main page README.md, which has a 80 SLoC example how to read a colored static mesh. The file format is well-defined and well-documented in every little detail, up to the last bit, see m3d_format.md.

Cheers,
bzt

Hi,

As I said, the code does not use STL, std::string and std::vector are merely containers, only initialized in "return" statements (you won't find any push_back calls in my code). But to ease your mind, I made the M3D::Model wrapper class optional, now it's only included if M3D_CPPWRAPPER is defined. Without you're free to implement your own wrapper, but there's still no need to start from scratch, just call the C API m3d_load() and convert the struct arrays to whatever C++ object you want.

About sprintf, I see no reason why do you ever want to export a model in ASCII format from an engine. Or why would you ever want to use the ASCII format at all other than debugging. But let's say for a moment, that you need it. Could you elaborate what is your issue with the use of sprintf? If you are concerned about buffer overflows, check the code again. All buffers are accurately measured and allocated before any sprintf use. If I made a mistake in that, I'd like to know so that I can fix it.

Cheers,
bzt

If you want your file format to get adopted by someone, you should give a good reason to take the effort to support it. First of all, you have to think about when it's used in the entire development cycle.

• During Content Creation
• Between Creation and Integration
• in the built game

Not every engine/game/developer is differentiating strictly between these steps, e. g. because the artists store and export using the same file format, or because the engine/build system doesn't process the assets.

In general, during content creation, the file format has to support everything the tool used for creation is supporting, without altering the data to much. (If you create models by placing primitives, this format has to support these primitives.)
After creation, the format should still support a lot of features (e. g. include not only the model, but also material information, a rig, animations, ...).
The format used in the built game has on the other hand to target performance, either small size or (what most will prefer nowadays) fast load times. This format might also have to rely inside a container file format, so the loader has to be able to handle the memory as a source, and not just files. Maybe there could be a special "batch mode" if there are multiple consecutive modles in memory/in a container file.

Since you emphasize a lot on the fast load time, maybe the last is most suitable for your file format. If that's the case, only the engine developers are your target audience.

 

Another use case might be any model file that's handled using a Version Control System (SVN, Git, ...). For all of these, a text format is easier to handle (e. g. detecting conflicts), but on the other hand, this would also require the file content to be predictable (e. g. if I change something over here, this section in the file changes and nothing else). Since I don't think this would be the case for 3D Models in general, I don't think it's worth spending to much time thinking about it (and didn't include it above).

17 hours ago, Sacaldur said:

The format used in the built game has on the other hand to target performance, either small size or (what most will prefer nowadays) fast load times.

It is fast load times in nearly any case, especially when developing a console game. Companies Sony and Microsoft have strict guidelines how long a game is allowed to load until it first shows up to the player or become rejected from the platform