🔒 Locked

Archive Format

Started by Strayfire Jul 11, 2004 at 2:07 PM 6 replies 1.5k views

Original Post

Jul 11, 2004 at 2:07 PM

A while ago I started working on a way to archive multiple files for my future games. I thought up a few different formats and created console programs to test them. The problem is that both have their pros and cons and I would like to know which format you guys think is more efficient before I making a more complex Win32 app and API for my games. First Setup ----------- [main header] [file header 1] [file/name data 1] ... [file header n] [file/name data n] The main header includes an id, the number of files, etc. Then each file has a main header with file and name size, and some other information. Second Setup ------------ [main header] [file header 1] ... [file header n] [file/name data 1] ... [file/name data n] In this version all of the headers are at the beginning of the file, and all of the data comes after that. --------- The first one is easier when it comes to adding files because it can simply add the next one right after the last file, unlike the second setup which requires all of the data to shift down when a new header is added before the file data. The second one is faster when it comes to reading files though because you can simply calculate [main header size + file header size * file number] to get the file offset then quickly move to that location in the file. The first version on the other hand requires you to read a file, jump ahead, read a file, jump ahead, etc. Which one should I stick with?

Kibble

Jul 11, 2004 at 2:18 PM

Neither. Put the file table at the end, store the offset to it in the main header. Do it something like this to save it:

 - seek to the size of the header (or just write it if you are feeling lazy) - loop through the files to write   1. Store offset of this file by using ftell(), ostream::tellp(), etc   2. Write the file - remember this position, it is the file table offset - loop through the files to write   1. Write out the offset, name, and size of each file, the offset was worked out 'automatically' by writing it - seek back to 0 - write header, with the file table offset you learned after writing the files

to load it

 - read header - seek to file table offset - read in file table which has file name, offset, and size and store this information

then when you want a file, seek to its offset and read a chunk its size and load/deserialize it.

when you add a file, you just seek to the file table offset, write the file, and rewrite the file table (all the other file information should still be good from loading it)

deleting on the other hand will almost always require a full rewrite, but you could just leave the data without the file table entry until the deleted unused space reaches a certain threshold to prevent tons of rewriting the file.

Hope this helps, I struggled with which of these 3 strategies to use and settled on this one as it seemed to have more pros, and no cons the others didn't have.

BradDaBug

241

BradDaBug

Jul 11, 2004 at 2:21 PM

When it comes to doing file IO, you're gonna want this thing to be able to find and read inside there as FAST as possible. Do NOT do the first method.

As for the second method, what happens if files are different sizes? Store the filenames and offsets in a big list at the beginning of the archive.

Edit: Oh, okay. Never mind about the second one. Out of the two I'd pick that one.

Strayfire

Author

Strayfire

Jul 11, 2004 at 2:23 PM

Thanks, that does sound a lot more efficient.

[added]

Well, that's what the second method was doing. Because the headers are always the same size the offset to the *header* could easily be calculated. With that, the offset could be read to find the start of the name and file data.

Clueless

Jul 11, 2004 at 2:53 PM

Damn. Kibble's method should work? Should it work with ofstreams as well?

When I try to update something inside a file it always clears the file as if I opened it with trunc.
I tried it with ate and binary/out.

I can only append stuff safely.

But it should be possible to change a fixed size header somehow?
Guess I'll look into that again ...

Kibble

Jul 11, 2004 at 3:21 PM

I used the windows file I/O so I can't help you there, it is more flexible with its creation and whether it overwrites or not. Now that I think about it, it might not be possible to keep the file contents and still open it for writing with std::(o)fstream. The windows file I/O is nice because you can specify TRUNCATE_EXISTING or not, and you can open it without trashing the file if you don't specify that flag. It is also faster in my experience.

The other option I forgot is to just have a fixed size file table. You would have a limit to the number of files you could save in an archive, but it could still be relatively large and the archive would only be a few kb empty.

edit: The advantage to a fixed size table being you can put it at the beginning of the file. I would still go with the table at the end, but its up to you.

Strayfire

Author

Strayfire

Jul 11, 2004 at 3:45 PM

ofstream truncates by default. If you want to open a file for updating you should use

fstream file(filename,ios_base::in | ios_base::out | ios_base::binary);

DJSnow

Jul 12, 2004 at 4:31 AM

hi.

we here are doing the following:
we have a chunk based fileformat, where each chunktype is described by an identifier; we have tools to convert and store arbitrary other files in one of our chunkfiles; so we are ending up with one big file, which contains all the media for a distinct scene, model, ect.; we can even store texts and other information types in the file.
the loader itself is fully adaptive: chunks, which are unknown by the current loader version are skipped; so it's assured that you can read at least all for-you-known information out of the files.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.