Binary / compresed XML (tinyXML)

there are free zip libraries so you can zip/unzip the xml at run time. I think there is a tutorial on this site about such a zip library.
Ofcourse the stupid user can figure out its a zip file, open it with winzip and edit the xml...

www.icculus.org and there search for physfs

its a virtual file system that can be plugged into your application easily and it supports reading of several compressed fileformats like zip ...

The 0ad project uses XML to store most text data. Loading plain XML is slow (especially when it drags in Xerces, ugh), so we decided to go with a binary variant.
At time of evaluation, there was no existing library for this; we implemented our own version that runs to about 1000 LOC. This could be made available, but some project-specific code needs to be stripped out first.

Some details on how it works: code uses only the XMB interface to read in data. Both the XML and XMB are on disk; at runtime, XMB is created anew if the XML has changed (the aim here is the opposite of yours - encouraging moddability).
Currently - due to use of UTF16 - XMB files can sometimes be larger than XML, but this would be reversed if using UTF8, to which we will switch at some point.
Main advantage of XMB is reduction in load time due to less string parsing.

I wasn't so much thinking of a different file-format, just having the data stored in a file on which you'd need fopen(path,"rb") - I wonder if I could do a very simple tweak to TinyXML to support this - all the C library functions work on both text/binary files I think?

I think you misunderstand the point of text vs. binary files. They are the exact same thing, with the only difference that "binary" files do not attempt to do any CR/LF <-> '\n' newline translation when reading/writing.
As to the problem of XML files being user-legible or slow to load, fopen(fn, "r") vs fopen(fn, "rb") makes no difference.

If you're worried about users modifying the files, then why don't you just use some basic form of encryption? Just scramble the data a little.

If the data in the file CAN be modified by users without breaking your program or, stop trying to control the world and trust them. Their desires are in fact the reasons the game exists in the first place. You should never take customization away from the user for no valid reason.

If you are worried about them making changes which don't work well together simply do 2 things - 1 change the file extension of the data files, so only users who actually KNOW they are taking a risk will be doing so (anyone who digs into unknown / undocumented files and makes changes realizes they might screw something up). Or, simply provide 2 copies of the file, one labeled something like "original" or "default" that only gets used if the first copy doesn't exist. This way users can experiment and then back out if they mess it all up.

If there are in fact things in the file you don't want them to change for fairness reasons, then you really need to encrypt the file or at least add a checksum type item to it.

Quote:

Original post by Jan Wassenberg I think you misunderstand the point of text vs. binary files. They are the exact same thing, with the only difference that "binary" files do not attempt to do any CR/LF <-> '\n' newline translation when reading/writing. As to the problem of XML files being user-legible or slow to load, fopen(fn, "r") vs fopen(fn, "rb") makes no difference.

You try writing 1000000 characters to a file in text mode and to binary mode. The binary file will be a lot smaller.

Quote:

Original post by d000hg Quote: Original post by Jan Wassenberg I think you misunderstand the point of text vs. binary files. They are the exact same thing, with the only difference that "binary" files do not attempt to do any CR/LF <-> '\n' newline translation when reading/writing. As to the problem of XML files being user-legible or slow to load, fopen(fn, "r") vs fopen(fn, "rb") makes no difference. You try writing 1000000 characters to a file in text mode and to binary mode. The binary file will be a lot smaller.

Heh. Uh, okay.

[bsunshin@Monster ~]$ cat > filemaker.c#include <stdio.h>int main(){    FILE* a = fopen("text.fil", "w");    FILE* b = fopen("binary.fil", "wb");    int i;    for(i=0; i<1000000; i++)    {        fprintf(a, "Z");        fprintf(b, "Z");    }}[bsunshin@Monster ~]$ make filemakercc     filemaker.c   -o filemaker[bsunshin@Monster ~]$ ./filemaker[bsunshin@Monster ~]$ ls -l text.fil binary.fil -rw-r--r--  1 bsunshin grads 1000000 Jun  4 13:16 binary.fil-rw-r--r--  1 bsunshin grads 1000000 Jun  4 13:16 text.fil[bsunshin@Monster ~]$ 

Convinced yet, or do you want to try it yourself?

Um, now I'm confused. Text must be more bloated:

say you want to write the number 1 billion to a file. In a text file that's 1000000000 - 10 bytes. But in binary you can just write the 4 bytes making up an integer.

What have I got wrong?

Quote:

Original post by d000hg Um, now I'm confused. Text must be more bloated: say you want to write the number 1 billion to a file. In a text file that's 1000000000 - 10 bytes. But in binary you can just write the 4 bytes making up an integer. What have I got wrong?

What you have wrong is that you a) don't want a "different format" but at the same time seem to expect that b) reading it as binary file will magically let you know if the next byte is a char or the next four byte are supposed to be number. If you still just fprintf your numeric data it will still be stored as text and that means it's buying you nothing.

You would have to write your file with fwrite and reading that back in without any kind of extra information is pretty impossible, unless your format is completely strict and rigid (which xml definitely isn't, you can't know how tags are arranged and what kind of data is stored where).

If you want a binary version of XML, it MUST be different, as the file has to store type information, use an id instead of full tag names and possibly a table of id->string unless you expect programmers to know all the ids by number.

You could argue, that XML with really short names and small numbers isn't much larger than a binary version with all the extra info. And the only way to have tinyXML work with that is if you write your own loader that turns it back into real xml, at which point using tinyXML will most likely be pointless. So you can either look for a different library using a "different format" that is essentially binary XML (like XDS from GPG4 seems to be) or stick with "bloated and easy to change".

Hint: look at the 3ds format and the chunk-approach it uses, that should give you enough ideas how to roll your own solution.

Quote:

Original post by d000hg Um, now I'm confused. Text must be more bloated: say you want to write the number 1 billion to a file. In a text file that's 1000000000 - 10 bytes. But in binary you can just write the 4 bytes making up an integer. What have I got wrong?

What you have got wrong is how you think one "writes binary". Check it out:

int i=839734955;fwrite(&i, sizeof(int), 1, myfile); // writes the 4 bytes making up the number 839734955fprintf(myfile, "%d", i); // writes the 9 bytes making up the ASCII representation of the number 839734955

Now, notice in there that I didn't include the fopen() call. That's because it's not how you choose which of these things you're going to do. However, doing the second one if you hadn't opened the file in binary mode would be a bad idea. Why? Because the machine representation of 839734955 includes one byte with the value 0x0D. If the file was opened in text mode, depending on the operating system, the C runtime might transparently stick a 0x0A byte immediately after it. This would result in shifting all the other bytes in the file down one, causing, well, nothing good. So you see, although fopen(..., "wb") is ESSENTIAL for properly writing data in a non-textual form, it is not SUFFICIENT.

How about checking out CDATA sections? Maybe putting your binary information into a CDATA block... In the past I have used zlib to compress information, then throw it into a CDATA block so I can read out the zip file later. Basically, embed your binary inside of your xml..

Just a suggestion

-JT

It's been a while since I looked such an ignoramus. Thanks for clearing up my misconceptions.