🔒 Locked

Thoughts on File System

Started by Bagpuss Apr 9, 2002 at 6:47 AM 7 replies 300+ views

Original Post

Apr 9, 2002 at 6:47 AM

I have completed a resource file system for my game, and am nearly done with my editor too. But one thing has struck me about the manipulation of the objects in my file. The problem is, the file could contain a large number of bitmaps / text etc. If one of these is edited, and it''s size changes then I Save it to the end of the file and adjust a file_item record to record it''s new position. At some point I will operate a compress routine which tidies up the file, and reorganises it so that all bitmaps for a screen are together for example. This is all done on the live file, and so if something happens eg a power outage the file may be corrupted. So, when I am running a Tidy_File routine is it better to make a copy of the file in a tmp directory, and only when the work is finished delete the temp (and if so, can someone give me some pointers as to the best way to achieve this), or to create a new file (eg Data1, copy and update from the old file into the new file, close the new filw, delete the old file and rename the new file ? My game also has a variety of data stores including vectors of data, text, bitmaps etc. Using Streams I can output all my data from my structures into the file, but is there a more elegant way than having a function to parse each of my structure types in turn into the file, and hence I also need one to re-load them. I have a load of functions that take the appropriate type and dump the data to file, but is there some commands / operations that mean I need a single function that takes a pointer to an object and irrespective of it''s data sends or loads it from disk ? TIA, BP

S1CA

Apr 9, 2002 at 7:19 AM

1. I''d go for the temporary file solution. The tmpfile() function will create a temporary file for you. For cases where the size of all sub-file hasn''t changed but you need to add a new file, an interesting idea is to store the file list at the *end* of your resource file, that way you don''t have to move data around the file to make room for a new ''catalogue'' entry.

2. If this is under Windows, there are a couple of things you can do to simplify loading of structures from file:

a) Use a "memory mapped file" (look in MSDN for details) for the resource file, these allow you map any of the sub-files into the virtual memory address range so that you can access it a plain memory - no streams, no loading etc... all the loading work is done by VMM in exactly the same way as it does with the page file. This is what we did in our engine, it works a treat - a file access is like:

  
GameFile file;
// map a view of into the .pac file into memory

file.Open( "blahfolder", "blahfile.tga" );

// a seek is simply a pointer manipulation internally

file.Seek( sizeof(TGAFooter), SEEK_END );

// get the current pointer

TGAFooter* footer = (TGAFooter *)file.GetPtr();

// use memory

if ( strncmp(footer.szSignature, "TRUEVISION-XFILE.\0",18))
...

file.Close();

If the whole file was a fixed length structure, there''d simply be an open, a GetPtr and a close. Internally there is no file manipulation or memory management required - simply a single MapViewOfFile() and a pointer addition.

b) Also handy in some situations in combination with the above, is the MSVC specific __based keyword (once again, take a look in MSDN for detailed docs).

--
Simon O''Connor
Creative Asylum Ltd
www.creative-asylum.com

Dean Harding

Apr 9, 2002 at 7:28 AM

I''m not sure there''s any value in keeping your files in the resource files during development. To me, the best method is to support both resource files and a regular directory structure. That way, during development, your normal tools (photoshop, etc) can be used to save and load, and there''s no need to re-add them into the resource files. Once you''re ready to distribute, just put them all in then.

As for the second problem, one way to do this is you have each object which can be saved/loaded to a file implement a Serializable class, then the class just implements the ToFile() and FromFile() methods itself.

codeka.com - Just click it.

S1CA

Apr 9, 2002 at 8:34 AM

Yup, during development I''d agree - use separate files.

However, expandable files are useful if your final release game is user-modifiable.

--
Simon O''Connor
Creative Asylum Ltd
www.creative-asylum.com

Bagpuss

Apr 9, 2002 at 9:08 AM

Thanks For your replies, I will look into it. I accept that Keeping lots of Files and a directory structure is easy, but I wanted to make the file system thing just to see how it worked as much as anything else !

Any more comments out there ?

JTAnderson

Apr 9, 2002 at 10:30 AM

I would also have the ability to store a file in multiple locations within the large resource file. Thus, if a file is accessed alot (and you can not keep it in memory) the seek times are lowered. This is for a CD scheme (ie Console development).

-JT

S1CA

Apr 9, 2002 at 12:04 PM

If we''re going for a "it''d be nice if we had" list, I''ll extend on JTAnderson''s with...

files location sorted by access pattern. Almost all games load certain things in a fixed order - it makes sense to put those things on the physical media in the exact same order to reduce seeks and maximise use of hardware and OS caching.
For assets which get used in multiple levels, either finding an average "best location" between two levels or storing a file multiple times (as JTAnderson suggests) is desirable.

--
Simon O''Connor
Creative Asylum Ltd
www.creative-asylum.com

Bagpuss

Apr 10, 2002 at 3:41 AM

Ok, Thanks, it look slike this could be becoming a bigger project than my noddy little tile engine

At the moment though, I still have a series of methods, all called WriteData() that each write a specific data structure to the file.

i.e. I have a WriteData That Outputs a Vector, One that Outputs a List, one that outputs a String, etc...

As my game may have bitmaps, vectors and structures in it, is there a unified way (other than the overloaded method I am using) of writing any data into the file (using streams if possible at the moment as it was trying to learn those that made me write this code)

i.e. my WriteData(* PC)

Has to pass through my PC object and output all the variables as binary data as some are char, some are int etc.

If I add to this object then I need to update my WriteData function. It owuld be really handy if I could just do WriteData(* anyobject) and it would dump all the contents of the object out to the file but I suspect that this feature doesn''t exist.

Thanks for your comments and feedback so far, I have found it useful.

cearny

Apr 10, 2002 at 8:39 AM

Take a look here:

http://ogre.sourceforge.net/docs/class_ogre__archiveex.html

May not be exactly what you asked for, but it''s a nice way to (no, not show-off

) separate the interface from the implementation. Basically, we create ArchiveEx pointers that point to implementation classes that can handle anything from real filesystems to .tar.zg archives.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.