🔒 Locked

File handling

Started by CTar Feb 13, 2007 at 2:50 PM 11 replies 2.7k views

Original Post

Feb 13, 2007 at 2:50 PM

I'm wondering how files are handled in complex systems, like games with big detailed worlds. How does the system determine what to load and when to load it? For instance let's say there is a 60% probability we will need world1.pack in 25 seconds and the loading time is estimated as 15 seconds. At the same time we have a 50% probability that world2.pack will be needed in 20 seconds and the load time is estimated to 15 seconds. How does the system figure out what to do? It doesn't have time to load both, so if it tries we will definitely get a loading screen, but it will be done very quickly. If we choose to start loading one and then load the other then there is a certain probability that we will need a loading screen, but there is also a chance that we won't need a loading screen. Of course it could get much more complex. For example we may have known that world1.pack and world2.pack will never have to be loaded at the same time and that the user doesn't care as much for a little delay when reaching world2.pack because they will need to read some text there anyway. I know there are no definite answers to what to do based on certain constraints (at least not if we want a decent algorithmic complexity), but I'd guess that there are some algorithms which will provide a decent order of loading which based on the information it gets is able to almost optimally load the resources. When resources finally are in memory they will take up more space and at some point we will need to free them to make room for new resources or other applications, but how is this determined? Least recently used or least frequently used may be decent strategies in the general case, but in a game we are likely to know much more. We may have loaded both ogre.m3d and goblin.m3d 5 minutes ago and have used both the same amount of times, however we have reached the next level and we have a 10% higher probability of meeting ogre.m3d than goblin.m3d, in addition we are now near the ogre encampment and therefore this increases the probability of finding ogres by another 10%. Standard Least Recently Used and Least Frequently Used doesn't take things such as this into consideration, but have any games experimented with alternative caching algorithms depending on stuff like this? Also instead of freeing goblin.m3d or ogre.m3d we may choose to compress them instead so that they only take half the memory, but will be much faster to load then through file IO. No cache algorithms I'm familiar with takes grey areas like these into account, it's either stay in memory or get freed. So, I guess what I want to know is what approaches have been tried by others and if they proved to be useful. It doesn't necessarily have to be limited to games, I just thought I would use games as an example since we are on a game development site.

Kitt3n

Feb 13, 2007 at 3:05 PM

We are using a sector-based system. We can predetermine which objects (trees, rocks, houses, even spawned enemies) are going to be in the sector and so we
can load them way in advance. However it's always a matter of balancing memory
against speed.
Also, I wouldn't try to make the determination to smart, but rather use a
KISS-approach (Keep It Simple, Stupid) :)

In our system we are streaming _loads_ of data, and harddisks are pretty
fast in reading big files.

If you are talking about streaming from cdrom then you have to watch your
seek-times, and maybe even store "memory-dumps" (in Unreal they call it
pre-baked) to quickly load whatever is needed.

Also you could let your artists work with model-sets (ie a desert set, and
make sure they don't mix it with other sets) - so it's easier to predict
what is needed.

Oh, and yes - compressing them really is a smart idea - it'll be quicker :)

CTar

Feb 13, 2007 at 3:17 PM

Thanks for the response.

Quote:
Original post by Kitt3n
Also, I wouldn't try to make the determination to smart, but rather use a
KISS-approach (Keep It Simple, Stupid) :)

Why is this? I can think of a couple of issues, but I'm not sure whether any of them really are problems in practice
1) It will be too time-consuming to implement
2) The determination itself will be too slow
3) Unexpected behavior which is hard for a human to follow
4) Flexibility, a complex approach may not easily be ported to another project and even changes in the current project could require changes in the determination algorithm.

Quote:
In our system we are streaming _loads_ of data, and harddisks are pretty
fast in reading big files.

So you haven't really found hard disk IO to be a bottleneck when working with sufficiently large files? How large are the files you work with?

Quote:
If you are talking about streaming from cdrom then you have to watch your
seek-times, and maybe even store "memory-dumps" (in Unreal they call it
pre-baked) to quickly load whatever is needed.

I expect everything to be run directly from the hard disk, but good to know in case I'll ever need to stream from CDs.

Quote:
Also you could let your artists work with model-sets (ie a desert set, and
make sure they don't mix it with other sets) - so it's easier to predict
what is needed.

I'm not sure I get what you say here. Do you mean like packing all resources which are found in a certain area into one big file to avoid searching for many small files?

Sirisian

Feb 13, 2007 at 6:21 PM

Yeah, for continuous worlds I use a sector based system too. Break the world into manageable parts that can be easily streamed in. My way is probably different than most as I use a prototype object method (made it up, but I bet a lot of people use the same method). My game loads in the map file which tells which game objects to load. If they ID isn't in the prototype object list it loads the object found in my objects folder where each object is a .bin. Then it precedes to load the object and define the prototype object which is cloned to make the game objects. If say the textures are not found it goes into the textures folder and loads them. Then if the boundaries ect aren't loaded it does those too. The resources all have timers so if they are not used for more than 20 seconds they are wiped from memory. This is clean and works very well.

However, I have yet to get into advanced ways of file handling. I use a binary class I made to keep the file size very low so it all works out.

Kitt3n

Feb 14, 2007 at 3:11 AM

>Why is this? I can think of a couple of issues, but I'm not sure whether
>any of them really are problems in practice
Because I expect the performance gain to be minimal (compared to the
time invested, especially if you need to make special-cases for various
things)... Of course you can prove me wrong :)

>So you haven't really found hard disk IO to be a bottleneck when working
>with sufficiently large files? How large are the files you work with?
For our extreme-high-detail textures we are using tga's, some in 4096x4096
and in that case we do get into streaming problems (we might try to load
40+MB in a few secs and there is some stutter).

However if you work with reasonable size textures/models, I don't think
you'll get into problems. For high detail we compressed almost all ultra-hq
textures into dds and in this case we don't have any problems with streaming.

Again, it's a matter of balancing the amount of data, the size of the
cache and the time you have before you need it. If our pre-streaming
failed we do a forced load, and in this case there are noticable hickups.
As a reference, our model & texture archive is around 6.5 gb -
or 2.5gb in dds-only (without the tga's).

>I'm not sure I get what you say here. Do you mean like packing all resources >which are found in a certain area into one big file to avoid searching for
If the level-designers don't put too many different graphics into a
sector/level (from all kind of different item-sets), but re-use stuff
better (ie stick to the orc-items and not put other things for other
regions there) there just will be less assets to load.

Kitt3n

Feb 14, 2007 at 12:13 PM

>may choose to compress them instead so that they only take half the memory,
>but will be much faster to load then through file IO.
As a sidenote - In "sacred" we were compressing the textures with zip (into
a custom archive format) and it proved a _really_ big improvement speedwise.
Loading a big file from hdd vs loading a small file and decompressing was in
favor for the compression approach (same for networking, with zip on
a low compressionlevel, you can get a factor 5-10 for network packets -
the difference with higher compression levels is negligable ie much slower
and only a little bit smaller packets ).

As another sidenote, we had 2d graphics for things like trees, so at the bottom
of the trunk there would be lots of "alpha/nothing". We cut entire trees into
eg 32x32 minitextures, checked which chunks were empty (and didn't store these)
and then stored the offsets of these minitextures (which even more drastically
reduced size :)

CTar

Feb 14, 2007 at 12:36 PM

Once again, thanks a lot. It's really great and interesting to hear this stuff from someone who has some real experience. However I would be amazed to find out that you never had problems with too big sectors or unused loaded resources. When some sector was loaded would you just keep everything in memory until nothing in the sector was used anymore or did you use some technique to free some of the resources? If you did free some resources, and if you're allowed to talk about it, what kind of algorithms did you use?

Also nice idea with mini-textures, I wouldn't have thought that it actually reduced overhead. I would have expected the performance hit of assembling these mini-textures into one texture again to clearly outweigh the gained IO performance, but it's a clever idea if it works.

KulSeran

Feb 14, 2007 at 1:31 PM

CTar, a lot of sectoring systems work on the premis that the user can only see what is within one
sector width from the camera. So, you load 9 total sectors, and every time a player moves sufficiently
far into a new sector, you drop the 3/5 far sectors and load 3/5 near ones. Assuming you are on
some sort of outdoor area, where a grid approach works.

If you are working with a portal system, you can walk the portals and find adjacent rooms, thus performing
a similar load/unload process of keeping all rooms 1 portal from a visible portal in memory and
unload anything once it gets 2 portals away from a visible room.

CTar

Feb 14, 2007 at 3:06 PM

Quote:
Original post by KulSeran
CTar, a lot of sectoring systems work on the premis that the user can only see what is within one
sector width from the camera. So, you load 9 total sectors, and every time a player moves sufficiently
far into a new sector, you drop the 3/5 far sectors and load 3/5 near ones. Assuming you are on
some sort of outdoor area, where a grid approach works.

It just seems like a waste of resources to keep so much useless stuff in memory. If we have square sectors with a length l on a grid and the nearest 9 sectors are loaded. Assuming we can only see l forward and we make a 360 degree turn then we have still seen less than 35% of the sectors. We have probably seen more than 35% of the content in the sectors, but there is really a large amount of memory wasted by having the rest of the content loaded. That is why I looked for ways to choose what to keep and what not to keep. The only thing I have heard so far is that people use sector-based resource loading systems (kitt3n did describe some techniques to supplement it though). Judging from the responses it almost seems like a silver bullet, but we all know that there aren't such things as silver bullets.

Kitt3n

Feb 14, 2007 at 4:29 PM

>I would have expected the performance hit of assembling these mini-textures into
Actually after getting the minitextures, you can put them back into
a big texture and store only uv's as a preprocess step (it's already
a long time ago so I forgot to mention that little detail hehe - and
it really looks funny to see your original object 'messed up' like
that *grin*

You can improve a bit on the 3x3 sector idea. When moving over a sector
border, you need to load n new sectors. When loading a sector, register
all objects in the entered sectors - ie register them in a manager as
having a certain 'type' and a position (but do not load the bulk-resource
data at this stage).

Now combine this with an 'streaming-range' and an 'activation' range.
Streaming is a circle larger than your max view-radius - with this
you can check if an object came into range and start streaming it
(models, animations, textures, ...) in a seperate thread.

Now when your object enters the activation range, you actually load
the bulk data - either the streaming was in time and it's already loaded,
or it wasn't and you either force-load, or alternatively mark the object
for a 'retry' (this can cause 'popping', obviously).
If you tune your ranges & timing correctly, you can minimize the number
of times your streaming was not in time.

If the object leaves the streaming range, throw away all the bulk data
again (but keep the object with type/pos registered in the manager).
Alternatively if you have a manager with a certain budget, just let it
keep the data, as long as you didn't go over the budget this manager
is allowed to manager (in that case, first throw away resources with
0 refcount, and then start doing lru).

This way you have a sector-based approach, but you don't load the bulk data
of all the objects on far-away edges which you are unlikely to need.

Concerning objects behind you - keep it simple again, make sure they
are loaded. In our case the camera can be turned 360 - and it's _really_
annoying to get framerate hickups just because you do a few full 360deg
turns.

You can even improve on the 3x3 sector-loading system by having
different criteria for when to load & when to unload sectors
(eg to avoid the case of running circles exactly on the intersection
point of 4 sectors - and triggering lots of load/unloads).

CTar

Feb 14, 2007 at 4:53 PM

Quote:
Original post by Kitt3n
Now combine this with an 'streaming-range' and an 'activation' range.
Streaming is a circle larger than your max view-radius - with this
you can check if an object came into range and start streaming it
(models, animations, textures, ...) in a seperate thread.

Now when your object enters the activation range, you actually load
the bulk data - either the streaming was in time and it's already loaded,
or it wasn't and you either force-load, or alternatively mark the object
for a 'retry' (this can cause 'popping', obviously).
If you tune your ranges & timing correctly, you can minimize the number
of times your streaming was not in time.

If the object leaves the streaming range, throw away all the bulk data
again (but keep the object with type/pos registered in the manager).
Alternatively if you have a manager with a certain budget, just let it
keep the data, as long as you didn't go over the budget this manager
is allowed to manager (in that case, first throw away resources with
0 refcount, and then start doing lru).

This way you have a sector-based approach, but you don't load the bulk data
of all the objects on far-away edges which you are unlikely to need.

That is actually a very good idea, thanks.

jbadams

Feb 14, 2007 at 10:43 PM

Quote:
Original post by CTar
It just seems like a waste of resources to keep so much useless stuff in memory.

...and what else are you going to be using that memory for whilst the game is being played? If the answer is nothing then there's no reason to keep it free.

You of course don't want to go over the top (i.e. loading the whole world into memory rather than the 9 sectors (or some equivalent) approach), but unless you're running in Windowed mode and expecting players to be doing other things (and if you're running in Windowed mode you should make at least some allowance for this) while running your game there's no reason you shouldn't use as much available memory as you need -- all that RAM your players have invested in is being wasted if you're avoiding using it without a good reason.

CTar

Feb 15, 2007 at 9:47 AM

Quote:
Original post by Kazgoroth
Quote:
Original post by CTar
It just seems like a waste of resources to keep so much useless stuff in memory.
...and what else are you going to be using that memory for whilst the game is being played? If the answer is nothing then there's no reason to keep it free.

Well file loading isn't the only thing which needs memory, by keeping the memory consumption down in this system we can allow for more memory consumption in other systems which may result in a better end-result or better performance. I think the important thing is trying to keep the peak memory consumption down. There is, of course, no reason to have 2 KB memory consumption 80% of the time if we use 8GB memory the last 20% of the time.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.