Resource Streaming

Quote:

How does reference counting (smart pointers, weak references and such) fit in such a system? What is the need for them? I especially appreciate a simple example on this.

Reference counting is ideal in this, as in just about any situation you can think of. Put simply, reference counting means you can just-about guarantee that nothing will be freed until there is nothing left that points at it. This means less crashes.

The only tricky bit that streaming resources into and out of memory adds is whether to point directly at the resource, or to store a reference-counted handle for a resource. The reason for this is that realistically, you're going to need to package your assets, and you will probably need a contiguous block of memory to load your assets into. If individual textures refuse to free themselves, you may find memory fragmentation soon becomes a problem. It may be easier to store a reference counted handle, and duplicate assets between packages, and then so long as there is at least one package with the necessary asset in memory, the game objects will be able to find it.

A simple example of reference counting (I'm not sure that's quite what you're asking for as you sound like you know about it already, but here goes):

class Texture : public IRefCountImpl
{
// ...
};

SmartPtr<Texture> ptrToTexture = new Texture();

That's it. No need to explicitly call delete, no need to check if anything else still points to it. You can guaranteed that as soon as ptrToTexture is destroyed or goes out of scope (or is reassigned, or whatever) the texture memory will be freed. See also tr1::shared_ptr, and RAII.

Quote:

What subsystem of the engine decides to unload/reload resources that go out of/come into view? Is the scenegraph responsible for such calls to resource manager to do the job for us?

Put simply: a resource manager. Another entity that updates along with all the rest of your game logic. The scenegraph (or equivalent) asks the resource manager for resources, and gets them back.

Typically, a resource manager has an offline component too, that bundles assets into package files. A mature asset management system will be able to interface with your game data, and will know that level one, for example (or, in a streaming game, a certain area of the map) will need an Orc model, and all his animations, textures and sounds, plus whatever else is in that level. The game map will then need to trigger a message to the resource manager: I'm entering level one, or whatever, and the resource manager immediately loads the necessary package. Done correctly, this allow the game to guarantee that any resource it may need will be available exactly when required without taking up unnecessary memory at other times. (Assets are often duplicated between packages, and there's usually some concept of a "common" package containing all the player character data as well).

Quote:

Using what criteria does the responsible subsystem decide what/when to unload/reload resources?

See above - a combination of offline pre-parsing and triggers from the level itself. The details vary from game to game, of course.

You will often find streaming games have areas separated by long narrow corridors (Halo noticeably did this) during which not very much appears to happen, but behind the scenes the resource manager is going into overdrive. Still a better experience for the player than a loading screen.

An alternative is something like how God of War did it - the designers were told that any two consecutive levels can add up to a certain total, but no more (may have been 16MB, I'm not sure). So level A could be 12MB but then level B could only be 4MB... level C might only be 8MB, and that would mean level D could be 8MB as well. At some point on level A, it would load level B into memory; by the time you reach level B, it can unload level A and load C in instead; and so on and so forth. So to the player it appears as one homogenous map, when it really is anything but.

Quote:

Any guides on how to implement such a system?

IMO it is absolutely critical to get the offline parsing working automatically. You don't want to have to spend hours, days or weeks trying to work out what is in each area of the game and package assets up manually... it'd drive you insane and you'd get it wrong. This implies that your offline asset management system must be able to accurately determine what other assets each asset is dependent on, including sounds and textures and models and animations and scripts and everything.

In our recent next-gen console game that was a persistent sandbox title, we had a resource manager that loaded packages very much as described above. Assets could be happily duplicated between packages, because disk space was far more readily available to us than system memory; each game entity would have a list of packages that it needed. When the game reported a game entity was going to be required, the resource manager got busy loading the right packages, and a second or so later the game entity could be added to the game. There will always be some degree of latency.

(The package builder itself knew that if for example one entity need assets ABCD, and another needed CDEF, the optimal layout would be three packages, AB, CD, and EF. However we had another layer on top of that, with packages containing sub-packages, so you'd have a super-package containing AB+CD, and another containing CD+EF, and if only the first entity was needed you'd load the whole super-package AB+CD, and if the second was then needed you'd load only EF from the second super-package, and if the first left the game you'd unload AB. This is simpler than I've probably just made it sound; the whole point was to reduce the number of files, because loading a single 100MB file is much, much, much faster than loading 1000 100KB files).

The total size of our assets was approx. 800MB, but the packages took up about 5GB due to duplication. Interestingly, our asset manager was only about 95% accurate in terms of determining dependencies to build the packages, so we also put all the assets on the disc in individual files. As a last resort, if a suitable package could not be found, we'd load the individual file. I'd have been happier if we hadn't needed to do that but it worked.

Small games wouldn't necessarily need any of this, and could use single files, load on demand, and use simple reference counting to unload. A large virtual environment like you are talking about (Oblivion, etc) would not be able to work like that, especially on console, due to the overhead of opening individual files and reading them from the DVD.

Hope this helps.

Thank you for taking the time and sharing the info. I do appreciate it.

So the general idea is to break the world into non overlapping sections, each of which contains resources that would add up to a total of an approximately equal size and group each section's resources into packages that would be loaded upon request. All these section related stuff (their size, their boundaries, their location and a list of packages belonging to each one of them) is then stored in that mission's map I assume. Correct?

So now the player/camera is moving around and when it crosses a section boundary some message is triggered and the scene manager asks the resource manager to load all the assets of those package(s) that belong to that section. The resource manager looks up in its cache and returns corresponding resources or loads them from the disk on a cache miss. The scene manager then put those entities in whatever tree-like structure it's managing. We can now either unload the previous section or keep as many sections in memory as possible.

But I've got two more question before those puzzle pieces fall in the right place.

1) How is such a message triggered? Are we keeping a list of sections (containing their spatial properties) in memory and check for a cross situation each frame? Where do we check for such a condition?

2) Well, the nature of such a system necessitates a lot of loading and unloading of resources (even in a reference counted system, unloaded resources may need to be loaded again in a following section) and as you also pointed out loading and unloading a lot of resources on and on can cause memory fragmentations. Any ideas on how to avoid that? Memory pools?

Thanks again.

Quote:

Original post by Ashkan 1) How is such a message triggered? Are we keeping a list of sections (containing their spatial properties) in memory and check for a cross situation each frame? Where do we check for such a condition?

There are several ways to tackle this. One way is to use the sections similar to how it's done in Half Life 2 - you simply have entry- and exit points to each sections and once an exit point (or rather an exit area) is entered, a load request is triggered. Another idea that is better suited for large outdoor environments or fast-pasted action like car races, is to use the visibility system. Once a new sector is about to be visible (can be predicted from camera direction and movement vector), a data-preloading request can be triggerd.

The best suitable mechanism will, however, depends on the game's specific requirements.

Quote:

2) Well, the nature of such a system necessitates a lot of loading and unloading of resources (even in a reference counted system, unloaded resources may need to be loaded again in a following section) and as you also pointed out loading and unloading a lot of resources on and on can cause memory fragmentations. Any ideas on how to avoid that? Memory pools?

Memory fragmentation is unlikely to occur if you allocate sufficiently large chunks of data. Memory fragmentation is only an issue if you do a lot of allocations/de-allocation of small chunks of memory. Game assets, however, aren't that small. Also make use of the available OS features. If your resources are placed in a package file and access that file using memory mapping, you will not only save disk space but also a great deal of OS resources (e.g. file handles) and time (I/O will impact your performance much more than memoy allocations if you plan to reload resources a lot).

Just my two cents,
Pat

[edit]
Whether or not memory fragmentation poses a problem should be determined via profiling after the system itself is up-and-running. I do not want to say it isn't, but *guessing* what might impact performance without even having a working prototype is probably a waste of time - IMHO, that is (though backed up by very recent real-world experience [smile]).
[/edit]

Yeah, you've pretty much got it. I daresay there are other methods, but this is the one I am familiar with.

Quote:

How is such a message triggered? Are we keeping a list of sections (containing their spatial properties) in memory and check for a cross situation each frame? Where do we check for such a condition?

This is obviously likely to be game specific. In the particular case you are talking about, a large virtual environment along the lines of Oblivion or perhaps a FPS, there are likely to be a large number of triggers placed on the level for general gameplay reasons (eg. "make monsters appear") anyway, and one of these trigger types is a "moving to new area trigger", placed some distance back from the new area itself (for obvious reasons - though in the worst case, you might still have to insert a load screen if you reach the new area before the load has finished). The triggers might take the form of planes placed on the level by the designers, or a list of regions defined by polygons, etc.

This is exactly analogous to the techniques used for old 2D tile-based games, wherein only the tiles for the area the player is currently in, plus the eight surrounding areas, are stored in memory at once. As the player crosses the boundary between areas, the areas are swapped in and out. The only difference is, rather than being areas of a set size, there's likely to be some flexibility; and as a result, placeable triggers are required rather than simply checking the position. Though I guess it could still be done that way, depending on the game layout? For an Oblivion-like game, a simple quadtree would probably do it - just work out which square you're in (based on XZ position) each frame, and make sure the assets needed for that square and any neighbours are available. This would be pretty cheap I think.

The obvious extension would be to make splitting the map up into regions, and placing the load triggers, automated at build time. I suspect this would be non-trivial.

Quote:

Well, the nature of such a system necessitates a lot of loading and unloading of resources (even in a reference counted system, unloaded resources may need to be loaded again in a following section) and as you also pointed out loading and unloading a lot of resources on and on can cause memory fragmentations. Any ideas on how to avoid that? Memory pools?

The usual techniques I guess, of which pools are indeed one. I tend to find that having fewer, larger packages also makes this easier than having many smaller resources (another win for big packages), because unloading always leaves a big space, rather than possibly only freeing a couple of KB you might not be able to do anything with!

If you think of the God of War example I gave earlier, as soon as you free level A you know you have 12MB spare to put level C into (B being in memory already), and as soon as level B is freed, that's another big chunk of memory spare for level D. Presumably the GoW team would have used a memory manager that placed alternate levels at the top and bottom of the memory space, always therefore leaving the maximum available for the next level.

Something more free-roaming like you are thinking about would obviously be rather more complex and the memory manager might need to be tuned quite carefully.

Also, as I mentioned before, you might decide that rather than game resources storing pointers to memory, they might store handles into the resource manager, and the resource manager is the only thing that stores direct pointers. This would have a couple of advantages - first, if an asset was in memory, and another package was loaded that contained a duplicate of that memory, you could free one of the duplicates, even though game entities still need to use it. This wouldn't help with fragmentation much, though the total memory usage could be reduced; but the second advantage is that you could reallocate the asset in memory, which could certainly help with fragmentation. I have to admit I haven't tried this but it should be possible. I guess the disadvantage would be that there's a greater distance between the game entities and the assets they need to use; also, presumably at some point the entities would need a direct pointer, even if only within one function; they just wouldn't be allowed to store pointers, storing handles instead.

Quote:

Once a new sector is about to be visible (can be predicted from camera direction and movement vector), a data-preloading request can be triggerd.

Good idea. I like that one.

Quote:

Another idea that is better suited for large outdoor environments or fast-pasted action like car races, is to use the visibility system. Once a new sector is about to be visible (can be predicted from camera direction and movement vector), a data-preloading request can be triggerd.

I like this one. But the problem is that in large environments where the player can wander about anywhere, it's hard to predict the direction that he will be moving to next. We might load some section into memory just to know he has turned back the last minute. We then have to unload that section and load a new one. One solution to this IMHO is to keep more (than one) loaded sections in memory very much like the 2D tile based approach SunTzu pointed out, where we always keep the player in center of some (8 for example) loaded sections.

Quote:

For an Oblivion-like game, a simple quadtree would probably do it - just work out which square you're in (based on XZ position) each frame, and make sure the assets needed for that square and any neighbours are available. This would be pretty cheap I think.

Another nice idea.

Quote:

Also, as I mentioned before, you might decide that rather than game resources storing pointers to memory, they might store handles into the resource manager, and the resource manager is the only thing that stores direct pointers.

Using handles would give us the freedom to unload resource or move them around in memory and the resource manager would keep track of everything for us, but having to do the translation between handles and pointers every time a resource is needed may cause performance problems. Using a hash table helps. I saw another alternative to this in Game Programming Gems IV : "The Beauty of Weak References and NULL Objects" which replaces that search with a second pointer dereferencing. But then, following pointers often is not cache friendly ... just in case someone finds that interesting.

Quote:

Also make use of the available OS features. If your resources are placed in a package file and access that file using memory mapping, you will not only save disk space but also a great deal of OS resources (e.g. file handles) and time (I/O will impact your performance much more than memoy allocations if you plan to reload resources a lot).

I have to keep that in mind :)

Thank you all for sharing your knowledge.