🔒 Locked

Serializing a struct

Started by mr_mo Oct 17, 2008 at 2:49 PM 3 replies 800+ views

Original Post

Oct 17, 2008 at 2:49 PM

Hello all, I'm attempting to serialize the following:


typedef struct 
{
	const char* name;
	const char* layer1[12];
	const char* layer2[12];
	const char* layer3[12];
	const char* layer4[12];
	bool passability[12*800][800];
}DataMap;


typedef struct 
{
	DataMap map;
}MapManager;

And my code for serializing this is:


		 // Save
private: System::Void SaveMap(MapManager* map)
		 {
			 ofstream ofs(StringToChar(CharToString(map->map.name) + ".map"), ios::binary);
			 ofs.write((char *)&map->map, sizeof(map->map));
		 }
		 // Load
private: System::Void LoadMap(System::String^ fname, MapManager* map)
		 {
			 ifstream ifs(StringToChar(fname), ios::binary);
			 ifs.read((char *)&map->map, sizeof(map->map));
		 }

(This is in .Net but I need to serialize in Cpp cause the game is in non-managed Cpp. You can ignore the StringToChar and CharToString.) However, I recently learned that I can't serialize pointers which puts me in a bind because I don't know any other way to go about this. What I'm trying to do is make a simple map editor that saves map data and loads it. I also want to be able to load this data from the game I'm programming. So I decided to use a simple struct the hold the data. note: I don't necessarily need to use "const char*" if there is a way to convert an object to "const char*" but like i said, I don't know any other way to go about this. The reason I'm using "const char*" is because the engine i'm using only accepts "const char*" for loading textures and so on. What should I do to save/load my data? Is there a workaround I can do? I don't necessarily want to know a way to serialize pointers themselves because it might become troublesome in the future, so a workaround would be better. Such as getting the data that the pointer is pointing to a temporary variable and saving that variable instead and loading it in reverse. (If you need more info than just ask :))

frob

Oct 17, 2008 at 3:17 PM

You are correct that you cannot serialize pointers. You must serialize the data that is being pointed to. If it contains pointers, you must process the data within those as well.

Two common approaches are:

Serialize the strings by writing the null-terminated strings. Deserialization reads data until it hits a null, sticks it into a buffer, then points you to the new buffer.

Or you can write the data length followed by the data. Deserialization reads the length, creates a buffer of the appropriate length, and loads the data directly into the buffer.

The second approach is more general purpose because you don't require special symbols in the data. You can store any arbitrary data array this way.

Zahlman

Oct 17, 2008 at 3:39 PM

Quote:
Original post by mr_mo
(This is in .Net but I need to serialize in Cpp cause the game is in non-managed Cpp. You can ignore the StringToChar and CharToString.)

1) '.Net' is properly capitalize '.NET', and is not itself a language. You seem to be thinking of managed C++. 'Cpp' also isn't a proper name for the language you're thinking of; the file name extension '.cpp' is sometimes used because some file systems don't like the character '+' in file names. But in case you were wondering, '+' works just fine in GDNet posts - just not in the preview.

2) I don't understand why you're giving a managed C++ code snippet when your game is in non-managed C++ and you want to serialize in non-managed C++.

Quote:
However, I recently learned that I can't serialize pointers which puts me in a bind because I don't know any other way to go about this.

Serialization.

Quote:
note: I don't necessarily need to use "const char*" if there is a way to convert an object to "const char*" but like i said, I don't know any other way to go about this. The reason I'm using "const char*" is because the engine i'm using only accepts "const char*" for loading textures and so on.

With std::string, you just call the .c_str() member function.

However, you need to understand that you can't "just serialize" objects, either, unless they're POD. See the link.

I'm wondering why you have that constant '12' lying around all over. It's like you have a struct of arrays instead of an array of structs (generally considered the better-organized layout).

Also, noone writes 'typedef struct' in C++ unless they need to interoperate with C code, or unless they learned it from someone incredibly behind the times.

Also, data members being const is a bit suspicious in general.

You should end up with something like:

struct DataMap {	std::string name;	struct State { // just a wild guess at what the 12-fold-ness is about		std::string layer[4]; // why not an array here instead of separate 'layer1', 'layer2' etc.???		// There are standard library classes that you might want to		// use here so that the bits will pack together in memory,		// instead of requiring at least one byte for each bool		bool passability[800][800];	} states[12];};ostream& serialize(ostream& to, const DataMap::State& what) {	for (int i = 0; i < 4; ++i) {		to.write(what.layer.size());		to << what.layer;	}	// You might want to consider doing something smarter here...	to.write(reinterpret_cast<char*>(&(what.passability)), sizeof(what.passability));	return to;}ostream& serialize(ostream& to, const DataMap& what) {	to.write(what.name.size());	to << what.name;	for (int i = 0; i < 12; ++i) {		serialize(to, what.states);	}	return to;}

frob

Oct 17, 2008 at 3:56 PM

I might as well write up a super quick example.

Well written formats can be loaded directly into memory as a large block, and operate as offsets within the file.

I'm sure this has a few bugs because it is just ad hoc without going through a compiler or anything, but it should get the points across:

Some structures and stuff for headers:

/* This header is the first item in your file.   Make certain that your data is written in memory format,   so be careful of big endian or little endian changes. */struct MapHeader { int nodeOffset;    /* Offset in the file for where nodes start */ int tileOffset;    /* Offset in the file for where tiles start */ ...;               /* Assorted other offsets */ char padding[...]; /* Proper padding makes sure the following data is properly aligned for your system. */ char data[0];      /* This blindly points to everything else.  Above padding should make sure it lies on byte boundary. */};/* Make sure the size hasn't changed. */STATIC_ASSERT( sizeof(MapHeader) == 32 );/* This is the thing that starts at TileOffset. *//* It has general information about all the tiles */struct TileHeader {  int numTiles;  ...;  char data[0];};STATIC_ASSERT( sizeof(TileHeader ... /* This is the array of tiles which has been serialized. *//* It contains several different classes of tiles in the array. */struct TileData {  int length;             /* Total length of tile data plus details */  TileTypeEnum tiletype;  /* Indicates the type of tile data */  ...  char data[0];           /* Tile-specific structure starts here */};STATIC_ASSERT( sizeof(...   /* These are individual tile-specific data structures */struct BackgroundTile {  int x;  int y;  int depth;  ...};STATIC_ASSERT( sizeof(... struct AnimatedTile { ...struct ActivatedTile { ...

And some source files to actually use those:

/* Load your big file into memory */MapHeader* map = LoadMyFileIntoMemory();... /* Null check, assertions, and similar stuff *//* Figure out the base positions of the various things within your file */mapNodesBase = (char*)map + map.nodeOffset;TileHeader* tileBase = (char*)map + map.tileOffset;...LoadTiles( tileBase );.../* Function to actually load this from file memory directly into the game */LoadTiles( TileBase* base ) {  assert( base != null );    TileData* currentTile = base->data;  for( int tile = 0; tile < base->numTiles; tile++ ) {    assert( currentTile != null );    /* Load the individual tile data */    switch( currentTile->tileType) {      case eBackgroundTile:          CreateBGTile( (BackgroundTile*)currentTile->data );        break;      case eAnimatedTile:        CreateAnimatedTile( (AnimatedTile*)currentTile->data );        break;      case eActivatedTile:        CreateActivatedTile( (ActivatedTile)currentTile->data );        break;      ...    };    /* Advance to the next tile */    currentTile = (char*)currentTile + currentTile->length;  }  }

mr_mo

Oct 17, 2008 at 5:46 PM

Thanks for the fast replies!
@frob:
What I understand is, you divided the specific data(MapHeader, TileHeader, TileData, BackgroundTile) and than used the previous data to read the next data. So it went like:

MapHeader > TileHeader > TileData > BackgroundTile|OtherTile

So you showed me the second approach you mentioned.

Quote:
Original post by Zahlman
1) '.Net' is properly capitalize '.NET', and is not itself a language. You seem to be thinking of managed C++. 'Cpp' also isn't a proper name for the language you're thinking of; the file name extension '.cpp' is sometimes used because some file systems don't like the character '+' in file names. But in case you were wondering, '+' works just fine in GDNet posts - just not in the preview.

Sorry, about that and I know .NET is not a language, but when .NET is mentioned, it generally means managed code(probably a bad habit though) and on top of that, my code is all in C++ so I didn't really thought about writing "This is managed C++". And yes, you are right, I fell victim to the preview.

Quote:

2) I don't understand why you're giving a managed C++ code snippet when your game is in non-managed C++ and you want to serialize in non-managed C++.

That is because I didn't attempt to serialize and deserialize in the game yet, only the map editor, which I'm coding in .NET for speed. I want to finish it quickly and move back to working on the game itself. So you see, the problem is not with the game but the serialization method.

Quote:

Quote:
However, I recently learned that I can't serialize pointers which puts me in a bind because I don't know any other way to go about this.

Serialization.

Quote:
note: I don't necessarily need to use "const char*" if there is a way to convert an object to "const char*" but like i said, I don't know any other way to go about this. The reason I'm using "const char*" is because the engine i'm using only accepts "const char*" for loading textures and so on.

With std::string, you just call the .c_str() member function.

However, you need to understand that you can't "just serialize" objects, either, unless they're POD. See the link.

I'm wondering why you have that constant '12' lying around all over. It's like you have a struct of arrays instead of an array of structs (generally considered the better-organized layout).

Ok, I'll learn what the difference is exactly. The way I learned to make arrays is:

int array[size];

So I did the same thing with const char because this project was the first time i actually used const char. I normally use string really. The way I understood chars before this project was that they hold bytes like so 'a','b','c', instead of like strings "abc" but then when i started this project, I saw that const char* acted similar to a string in respect of "abc".

(Coming back after reading seeing the example code you posted, I can see what you mean by the structure and you are totally right.)

Quote:
Also, noone writes 'typedef struct' in C++ unless they need to interoperate with C code, or unless they learned it from someone incredibly behind the times.

Also, data members being const is a bit suspicious in general.

Haha, you got me on that one. When I initially started to learn C/C++ I had to communicate between the two in order to use a certain engine, so its a habit from then.

That's what I thought about const as well. It totally confused me (see the above paragraph for why).

Quote:

You should end up with something like:

*** Source Snippet Removed ***

The 12 fold is.. well I should have explained what I'm doing more in-depth. I'm making a side-scrolling game but not tilebased. All the art will be hand-drawn. So in order to build the maps, I just want to cut the map by 800*800 pieces and just load them in order without really "placing" them. I want to allow 12 max pieces of these 800*800. All those variables will contain is just the name of the files(the 800*800 piece) like so:

layer[0][0] = "bgmap1-0.png"
layer[0][1] = "bgmap1-1.png"
layer[X][N] = "Xmap1-N.png"

Note: I'm using 800*800 to test, its not final. I'm probably gonna shrink the size. The main purpose of this editor is for passability edit, nothing more yet. 800*12 was because i was testing size, performance and all. Since the game is not tile based, I want to calculate passability pixel-by-pixel and not tile by tile, hence the 800*12, 12 pieces of 800width files.

So I think your guess is wrong because I might be wrong but isn't having a layer for EVERY(12) piece less efficient then having 4 layers that contain 12 pieces?

State[0].Layer[0] = "bgmap1-0.png"
State[1].Layer[0] = "bgmap1-1.png"

but I can do this instead i guess:

#define MAPSIZE 12struct DataMap {	std::string name;	struct Layer {		std::string pieces[12];	} layers[4];	// There are standard library classes that you might want to	// use here so that the bits will pack together in memory,	// instead of requiring at least one byte for each bool	bool passability[MAPSIZE*800][600];};

I only need one passability array because of the static nature of the maps. Also, I'm interested in the library classes you are talking of, can you please name a few or point me to the right direction?

So after testing, I've got 3 errors(but 2 different).
1) error C2440: 'reinterpret_cast' : cannot convert from 'const bool (*)[9600][600]' to 'char *'
EDIT: fixed by : to.write((char*)&map.passability, sizeof(map.passability));
2) error C2660: 'std::basic_ostream<_Elem,_Traits>::write' : function does not take 1 arguments

Here is the new code:

#define MAPSIZE 12#include <string>struct DataMap {	std::string name;	struct Layer {		std::string pieces[12];	} layers[4];	// There are standard library classes that you might want to	// use here so that the bits will pack together in memory,	// instead of requiring at least one byte for each bool	bool passability[MAPSIZE*800][600];};typedef struct {	DataMap Map;}MapManager;

		 // Saveprivate: System::Void SaveMap(MapManager* map)		 {			 ofstream ofs(StringToChar(std2gc(map->Map.name + ".map")), ios::binary);			 serialize(ofs,map->Map);		 }		 ostream& serialize(ostream& to, const DataMap::Layer& layers) {			 for (int i = 0; i < 12; ++i) {				 to.write(layers.pieces.size());				 to << layers.pieces;			 }			 return to;		 }		 ostream& serialize(ostream& to, const DataMap& map) {			 to.write(map.name.size());			 to << map.name;			 for (int i = 0; i < 4; ++i) {				 serialize(to, map.layers);			 }			 to.write(reinterpret_cast<char*>(&(map.passability)), sizeof(map.passability));			 return to;		 }

Also, how would I go about reading it again? I would think doing the reverse but the fact that you also wrote the size kinda puzzles me because the size of the variables are static so I didn't think you would need to write the size as well.

I don't have time right now but I'll play around with this later.

[Edited by - mr_mo on October 17, 2008 8:46:03 PM]

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.