One thing I am struggling with in my custom planet simulator/game engine is how do I get assets into the procedural generator in a generic way that supports everything I need to do. For instance, I need to load a library of textures that should be applied to different types of terrain, or a library of meshes to spawn on the planet surface.

I currently have a fairly sophisticated asset serialization system that can pack all assets into one or more container files, where each asset is uniquely identified by a 128-bit UUID. This works really well for maintaining references between assets within the editor. To load one of these assets by UUID from code, I would have to hard-code the UUID for whatever is needed into the C++ code, which doesn't seem that maintainable to me, since the UUIDs are automatically generated and not really exposed in the editor. I can also reference assets by a human-readable name or URL, but this has problems with name collisions, and strings are much slower than fixed-length UUIDs, so I would like to avoid it.

There has got to be a better way. In Unity, they have the luxury of a scripting language which can automatically map assets to component member variables (via GUID I believe), but this is not really possible in C++.

The best thing I can think of is to have some layer of indirection between the code and data, i.e. rather than referencing assets directly, some other configuration data is loaded that says which assets are needed by a particular piece of code. This way the code doesn't need to know exactly about what assets it needs, it only needs to know about the fields in the config file which provide the UUIDs of the actual assets. Then I can make a GUI to edit the configuration file, which could handle mapping UUIDs to the asset slots in a nice maintainable way.

Does anyone have any ideas for how to solve this problem in a better way?

Engine code with extensive and fragile dependencies from data in asset container files seems a major architectural faux pas. There should be representations of “levels”, “worlds”, “scenes” or the like that depend on asset libraries (using the identifiers in the library to access the corresponding assets: “this object is an instance of model A554 and it has texture atlas C682”) and the engine should load these generic levels, follow references (find model A554, find texture atlas C682), and never care about what that stuff really is (whatever matters can be expressed with metadata, e,g, flagging texture C682 as a “medium texture atlas” rather than a “skybox”).

If you feel the need for shortcuts and hardcoded features depending on hardcoded assets (e.g. a sequence of splash screens that you load from textures), you can put them in unchanging default archives that are always available and robust against modding (your editing tools are able to deny deletion of the hardcoded assets or replacement with something incompatible).

You might also be changing asset IDs unnecessarily, generating new ones (and breaking dependencies) when you save assets instead of keepin the original ID.

Simple solution: use namespace-based UUIDs (UUID version 5), and convert from human-readable name to UUID in a constexpr function.

Aressera said:
I would have to hard-code the UUID for whatever is needed into the C++ code, which doesn't seem that maintainable to me, since the UUIDs are automatically generated and not really exposed in the editor.

You do control the editor, right? In this case, first step would be to actually expose this UUID in the editor. This is one of my many gripes with engines like Unity, that you have no idea about lots of the intricate properties. I personally have a field for all UIDs in each inspector, in case you might need it eigther for some manual lookups, or just to compare to the asset-files.

Aressera said:
I can also reference assets by a human-readable name or URL, but this has problems with name collisions, and strings are much slower than fixed-length UUIDs, so I would like to avoid it.

As with name-collision: You can avoid this by using the full path. For example, I allow referencing assets via a path-string, which if the asset is in Assets/Textures/Effect.png, it can be referenced via Assets/Textures/Effect.
As for performance: Yes, string is slower than fixed-length structures like UUIDs, but not by that “much”, especially if you take into consideration that you have perform the actual lookup, which will probably eigther be a dictionary (where the < operator doesn't have to check the full string) or a hash-table (where hashes are calculated once and only for same hashes, the string has to be compared). Also depending on when and how often you lookup the assets (I'm assuming you are not looking up the same texture each frame in a loop) it probably doesn't weight in much compared to all the other things that happen at load-time.

Aressera said:
There has got to be a better way. In Unity, they have the luxury of a scripting language which can automatically map assets to component member variables (via GUID I believe), but this is not really possible in C++.

If you mean Unitys property-binding-system; where you can create a field of a certain asset-type and then link the asset. Surely this can be done in C++. I have the same system in my c++-game engine. Now obviously to be as flexible as Unity it requires a very large type-system to be created manually, but just for linking assets, I imagine you can get away with a lot less work. You'd need a way to specify which fields to bind; some mapping of which type of asset you want; and some mechanism in the editor for picking. The serialization could then be done via your UUIDs.

LorenzoGatti said:
There should be representations of “levels”, “worlds”, “scenes” or the like that depend on asset libraries

I already have this capability, but it's not feasible in the context of procedural generation where nearly everything is created at runtime except for textures and clutter meshes. The rest of what you propose seems similar to what I mentioned in the OP, where you have some kind of data that tells the engine what assets to load. This still doesn't help with the question of how do you reference the scene or texture library to load? By referencing the assets in this way, it reduces the number of places where I need to reference assets from code, but doesn't eliminate it. You can have scenes that load scenes, but in the end there is a root-level scene that loads everything else which needs to be referenced somehow.

Juliean said:
you mean Unitys property-binding-system; where you can create a field of a certain asset-type and then link the asset. Surely this can be done in C++. I have the same system in my c++-game engine. Now obviously to be as flexible as Unity it requires a very large type-system to be created manually, but just for linking assets, I imagine you can get away with a lot less work. You'd need a way to specify which fields to bind; some mapping of which type of asset you want; and some mechanism in the editor for picking. The serialization could then be done via your UUIDs.

I have something similar already, but due to the lack of reflection it requires a lot of boilerplate code to define the GUI for each type of asset. This works fine for content that is authored in the editor, but not for procedurally-generated stuff that is created from scratch at runtime. In this case I need to be able to create generated meshes and attach predefined materials referenced in some way. I can roll up all materials into a MaterialLibrary resource or something like that, but in the end I'll have to do resourceManager→load<MaterialLibrary>(“terrestrialPlanet”) or resourceManager→load<MaterialLibrary>(Hardcoded UUID) somewhere in the code in order to find the assets I need to generate something.

I can roll up all materials into a MaterialLibrary resource or something like that, but in the end I'll have to do resourceManager→load<MaterialLibrary>(“terrestrialPlanet”) or resourceManager→load<MaterialLibrary>(Hardcoded UUID) somewhere in the code in order to find the assets I need to generate something.

I think the opposite direction of dependency is more common: some resource manager sits between the two distinct phases of specifying a set of archives and resources, with precedence rules, and loading resources from the resulting aggregate by name without bothering with their provenance.

You can reduce the extent of the dependencies of your procedural generation from data archives by introducing abstractions, and reduce instability by ensuring that a few well known assets are "always" available.

Suppose you have a function that generates a sand desert environment consisting of a relatively smooth terrain mesh painted with a mixture of texture Q12 (fine pale sand) and Q13 (fine pale sand with some gravel), and decorated with a few instances of "clutter meshes" W34 (dead camel) and W35 (camel skull). It is fragile because it references assets by ID: changes affect both engine code and assets.

Instead you could have a function that generates a more generic open environment and uses assets by looking for combinations of tags in a set of available assets: some externally specified (for example "desert") and some implicit (e.g. terrain textures for solid and loose terrain to be used according to erosion patterns, rarity and size of clutter meshes affecting their placement).

You could then reproduce the previous desert by tagging assets appropriately in the editor: all of the above as "desert", Q12 as "solid terrain texture", Q13 as "loose terrain texture", W34 and W35 as "clutter mesh", and subordinately W34 as "rare" and "big", W35 as "rare" and "medium".

Of course, other assets could be tagged "desert" and the function could use them as alternatives.

Instead of depending on a bunch of asset IDs, your procedural generation would depend only on the presence of enough assets with certain tags that, unlike IDs, can be assumed to be quite stable over the lifetime of the project (sometimes added as sophistication increases, occasionally reorganized).

The next level would be, of course, moving procedural generation rules from engine code to scripts in asset/level archives, removing numerical parameters and tag names from the code and shifting from specific generation purposes (e.g. make an open environment for an offroad race) to general building blocks (e.g. make a bumpy terrain mesh and place clutter meshes on a terrain mesh).

But it wouldn't necessarily be justified. Even hardcoded asset IDs can be fine if you ensure that the resource is available (for example, because it comes from the game's built in assets and mods can only replace it without removing it) and valid (by checking properties and metadata upon loading).

I have something similar already, but due to the lack of reflection it requires a lot of boilerplate code to define the GUI for each type of asset.

Please elaborate. Different asset types shouldn't be very different.

Aressera said:
I have something similar already, but due to the lack of reflection it requires a lot of boilerplate code to define the GUI for each type of asset. This works fine for content that is authored in the editor, but not for procedurally-generated stuff that is created from scratch at runtime. In this case I need to be able to create generated meshes and attach predefined materials referenced in some way. I can roll up all materials into a MaterialLibrary resource or something like that, but in the end I'll have to do resourceManager→load(“terrestrialPlanet”) or resourceManager→load(Hardcoded UUID) somewhere in the code in order to find the assets I need to generate something.

Why is that? Maybe you didn't invest enough time to reduce that boilerplate. Speaking from my own experience, I did start with having to define about 300 LoC for each different “type” I implemented, back before I introduced my own sorta-reflection system. The “drawback” now to the system is that I have to manually specify a declaration:

ComponentDeclaration Sprite::GenerateDeclaration(void)
{
	return 
	{
		"Sprite",
		{
			{ "Texture", bindAttribute(&Sprite::texture) },
			{ "Color", bindAttribute(Sprite::color) }
		}
	};
}

And GUIs/serialization etc… is all done automatically based on those declarations. Which is still not as slick as something like Unreal or Unity do, but pretty close.

Not even sure it helps you with your question at that point, but just thought I'd throw that out to give you an idea on maybe how to remove some of that boilerplate.