non-XML based mapping formats ...

Why not just create your own binary format storing the data you need?

That was kinda what I was getting at, but was just wondering if anyone knew of a format already that I could kinda read through and get some ideas from.

The structure

I'd divide the file into several parts, or chunks:

• the header
  
• the tilemap data
  
• the game object data
  

The header would contain vital information such as map width, height and number of layers, as well as additional data such as version number, map ID, referenced tileset, etc. This block will likely have the same size for every map file, depending on how you handle the tileset (filenames can vary in length, so in that case, storing it's length and then reading that amount of bytes should do, it just means that not every header has the same size).

The tilemap data is the most bulky part, so if you want to keep files small, this is the place to optimize things. Rather than storing tile elements with their position and properties, it's often better to just store all tiles in one big row, all after each other. You can derive their posision on the grid based on their position in this row if you know the map dimensions and number of layers - which are stored in the header. Rather than listing properties such as movement direction, player starting point, and so on, we'll just list the texture ID's here. If you're using less than 256 textures, you can get away with a single byte per tile.
The size of this block depends on the size of the map, but it can easily be calculated based on the header data.
For collision purposes, you can bind collision data to tile images, or you can use an additional layer and use it's content for collision checks rather than to display it.

The game object data is where I would store all game-specific elements. After all, only few tiles will have a specific movement direction, and there are only few spawnplaces in a map, compared to the number of tiles, so storing this data for every tile will bloat it, while most of the tiles will only contain a lot of additional 0's.
It probably makes sense to use xml (or a similar, easily parsable structure) for this part, simply because some game objects may have a lot of parameters (object type, name, position, sound file, volume, etc.) while others may be relatively simple (object type, position). This means that you'll be dealing with several bits of data of variable size. Since this part usually isn't as big as the tilemap data, using xml here isn't such a big deal.


An example

A small example file could look like the following (numbers represent bytes):

2 2 1 12 41 8 tileset2 3 2 4 5

The first two bytes represent the width and height of the map, the next byte the number of layers. 12 is the map ID, 41 the version number, 8 the length of "tileset2". After that comes the tile data: 3, 2, 4, 5. These numbers refer to the tile images in tileset2, so the tile at (0, 0) uses tile number 3, the tile at (1, 0) uses tile 2, and so on.
Then comes the game object data: a player start, a levelchange, pretty straightforward. The music object also has a name, which can be used by other game objects to toggle it: the next object, a trigger, will turn it on when the player steps on the trigger.

Of course, this game data would need to be parsed and you'll need to write a system that actually creates these objects with their functionalities. This is just a way to store them in a file. The advantage here is that you can add new game object types and properties, without changing the file format.


I hope that gives you some inspiration and guidelines. :)

so, putting some kind of table of contents in the header saying where certain sections of data appear is the idea. thats good. yeah i was gonna pad every single tile with all the collision info, etc. and you're right that would bloat the file greatly.
i've playing around with that file carver tool, and since it allows you to edit binary with relative ease (once you specify its structure), i thought i'd use something more obfuscated besides xml.

It would probably be much more efficient to store the map as it is in some sort of encrypted zip format. You'll get smaller resource files and only have to make small changes to the loading code itself.

Edit: By efficient, I meant in terms of time and work required to implement.

I recommend setting up an object-based text-parse system. In other words, a goblin has its own text-parsed file, an automatic door has one, and a wall has one. You could even allow the object parsed files to accept parameters. For example, the wall text file could look at a special variable "color", and paint itself different depending on the value of that variable.

Then you can use a simple binary map format that just stores a list of objects. Each node in the list needs a position (or transform), a text filename of an object, and values for any parser parameters. Then, in your map editor, you just select a filename, insert it onto a position, then move on.

Here's an example of how the map file could be stored:

06,03,"TallWall.xml";
02,08,"TallWall.xml";
14,55,"AutoDoor.xml";
30,40,"Goblin.xml", GREEN;
31,40,"Goblin.xml", BLACK;
etc

That's it. Just a list. Regardless of how many crazy changes you make to your game objects, your map file format will never need updated. You can completely change "Goblin.xml", and it will change the goblins in every map you've made so far.

From what I've seen of XML, it may be too slow to do this reasonably. The language also seems extremely long winded. But there are other text based formats. Some scripting languages would work just as well as a parser language.

What I'm doing for storing data is simply serializing my objects. I have the advantage of working in Lua, where such things are relatively straightforward. So I have a map-creation program which I use to make the map, and then when I save it, it simply examines the table (object) the map is stored in and serializes it. Then all I need to do to retrieve the map is to load the file it's stored in as a Lua object.

I wouldn't recommend binary, personally, because binary is a pain to work with with respects to reading, writing, and working across platforms (you run into endian problems; this is less of an issue now that Macs are on Intel, but it's not gone entirely). I strongly suggest making a human-readable format to start out, and then once you have something working well enough to release, you can add some obfuscation so it's not trivial to read. Remember, it's always easy to make something harder to read, but it's not always easy to make it easier to read. Start out with something easy while you do development, so you retain your sanity. Only make it hard to read later.