Packed files

Well, the easiest way is just to include the directory in the file header for a given file. For all the files in the sound directory, their file name stored in the package would read "sounds/????????.???"

Now, if you want to get tricky, you use recursive package files. As in, the sounds directory is really a package stored in the master package file. This speeds stuff up tremendously when accessing files, period.

My package format supports/uses both.

- Splat

Neat idea, Splat. So you header conatins an entry, say, sounds that actually points to a nested header with info about all of the sounds info?

I never thought of that. hmmm.... Gonna give it a try. Thanks.

By the way, anybody here used zLib? Anygood? I'm not sure if I should use it, as it seems to be a little overkill.

Yeah, pretty much. The "sounds" entry could actually be extracted from the main package and would yield a completely valid archive on its own. Quite cool. The packing utility took a little tinkering before this feature worked as intended, but its cool now. The packer is almost ALWAYS more complex than the depacker (in-game).

- Splat

[This message has been edited by Splat (edited November 17, 1999).]

I havn't had problems makeing packers,my problem is reading the data back in then making good use of it

I agree with Zenroth. I've found it relatively easy to pack the data. It's extracting it properly that seems to be more complicated.

Are you using recursive packages? With me, the loading process is VERY straightforward. First, the directory of the main package is loaded, which contains file headers containing the filename, size, offset into the current package, and compression format.

Then, for each subpackage, the process is repeated, creating a tree of file headers.

Whenever my game needs a file, it passes the complete filename it wants, ie "sounds/death.wav". First, it checks to see if a file header with that exact name is present. If not, it "peels" off the directory and looks for a file header of a subpackage with that name. Recurse the same function into the subpackage. Repeat until file is found or conclusively not there.

Once the file is found, we have (automagically using the loader function) an absolute offset from the beginning of the WHOLE package. Its just a matter of reading it in.

The packer, however, is a real recursive nightmare. Reason being, each file header is NOT the same size. The file name has a variable size and is stored by writing a byte for how long it is then the actual contents. So to pack the archive, you need to reserve the file header space at the beginning, then compress the files, then come back and record the file header's with offset, size, and compression type.

Its much more important to have a simple loader than a simple packer. You have unlimited time to pack the data, but a game user does NOT want to wait a minute while your game loads its data. Redesigning a file format for easy loading is critical.

- Splat

[This message has been edited by Splat (edited November 17, 1999).]

That is very interesting. Thanks for the clarification. The only question I have left is, why do you use a variable size header, especially if you're concerned about speed? Wouldn't it be faster to use a static size?

Well, here's the thing: I am concerned about speed, but only on the loading end. While it takes a little longer to pack, because you have to traverse the file list twice in effect. (note: I may be fixing this by placing the file header at the end of a package rather than the beginning: this also speeds up additions and removals to the package without a need to completely repack it)

However, reading a byte then a variable number has neglible effect on performance, and that is all the loader does. The trick is, loading of the package directory is done at registration (in my game engine you register a package file to get an integer package id). So that is also just a one shot thing. Plus, it keeps support for long filenames while not having to reserve 255 bytes per entry. You got several thousand files, that 255 bytes per will start to add up and make the file header so very large.

I guess in the end its just a personal choice. 400k here or there isn't that big a deal.

- Splat

Ok. SO one last question.

How do you deal with compression? DO you compress the packed file, or each file before packing it? I see advantages both ways:

one big archive - Less load time, decompress at startup. After that, just extract data. More memory is required to hold the decompressed file.

bunch of archives - less memory. Only unpacked as they're needed. Slow because they're only unpacked as they're needed.

So, what do you think? Compress before or after packing, or just skip packing altogether. Although, if I'm gonna skip it, I might as well just compress it before and after startup.

Did any of you guys consider not re-inventing the wheel and use ZIP? Just wondering if there's something important I missed

/Niels

If you want compression, you should definately compress each file on an individual basis, or else you'll have to de-compress the whole archive just to get at one de-compressed file, and that would suck wouldn't it?

Splat, I'm not familiar with the details of your system, but I would say that fixed size file headers (or sub-archive headers, whichever you prefer) would definitely NOT add significant size to the file on your hard drive. For reference, I use a fixed 128 byte header for each directory/file, so if I had 2000 files in the archive, i would need 256K for the headers. Ok, this might seem like a lot, but how big would 2000 files be? Easily half a gigabyte, so 256000 / 500000000 = 0.000512 = .0512% or one twentieth of a percent of the total archive size. Don't worry about fixed size headers...

Also, you usually don't need to modify an archive at run-time (during a game) so I wouldn't bother using an on-the-fly system where you tack on file/headers at the end of the archive for "easy modifiability". To me it isn't worth it. I put all directory/file info at the start of the archive, and the file data after. Works better for me.

Out of curiosity, what kind of speed advantage can be gotten by using a packed file format? Currently I am storing several thousand files as bitmaps each in their own separate file. Totals up to about 150 megs. What kind of speed increases can one get by using a packed file system over just relying on the user's filesystem?

Gees, a lot of messages!

Zer: first off, that 150 will drop a lot because of the excess size due to cluster size. A hard drive is formatted with a cluster size that defines the increment for files. So if you are using a 2k cluster size, a file will be rounded up to a 2K multiple. Packing reduces that a lot.

Niels: Reasons for writing my own:
1) Learning, as was said above.
2) Portability, althought I do concede that libraries like zLib are much more portable than mine and support compression.
3) Test bed for my own compression algorythms, which may or may not prove quite efficient.
4) Perfectionism - I like it my way! My package format is simple, straight-forward, small, fast, and expandable. I'm not really tied into ANYTHING with this.

foofightr: Your files are much bigger than mine. And you have less. I like the idea of variable size names. I have support for unlimited size names, unlike your *hard coded* (hate those words) 128 character limit. I'm trying to cut what I see as the fat on typical package formats. Also, if you really want to get technical, variable sized strings are faster because a) you load less data from disk, b) you already know the length, which saves you time from searching for the first null char.

As for the moving the file headers to the end, no, it's not for on-the-fly updating in game. Its for dynamically recompressing a 150MB directory when only 200k changed, including added files. The file header at the end means you can move the file header down and add the new files, and add their entries to the file header, and recursively update the affected offsets and sizes in the rest of the file. Saves a tremendous amount of time.

And lastly, it's my game (er, game engine) damnit! hehe

_kill_: http://www.wotsit.org has lots of stuff on ALL file formats, including most common compression types.

*gasps for breath*

- Splat

[This message has been edited by Splat (edited November 18, 1999).]

Oh shoot, forgot Stark:

You definately compress individual files in the package for several reasons. Mainly speed, memory restrictions, and the fact that different types of data are more easily compressed by different types of compression. Libraries like zLib even split files into "blocks" that can each be compressed differently.

- Splat

the datafile format that i use happens to use zlib for compression, and i must say it's very easy. i compress the files individually, and stick them into the datafile. zlib address: http://www.cdrom.com/pub/infozip/zlib. the zlib tool article is also very useful: http://www.dogma.net/markn/articles/zlibtool/zlibtool.htm

------------------
http://members.xoom.com/mutex0

Splat, I'm looking at it from a practical standpoint. I believe my approach is more practical, it suits my needs just fine, end of story.

I agree to disagree.

Put a fork in it, this thread is done!

- Splat

na-na-na-na-na, I'll get my foot in here too

And YES, learning qualify as a good reason! I Use a C++/Java combination, and since the client is downloaded, I have to worry about code size - ZIP compression is already there I.e. smaller code, faster download !!

/Niels