XML Data Compression and Readability

Umm, why the fuck would you store your geometry in XML? XML is not meant for that.
Make your own binary format.

quote:

---

Original post by Raduprv
Umm, why the fuck would you store your geometry in XML? XML is not meant for that.
Make your own binary format.

---

...
XML is metadata
data about data
my geometry is data
and my xml is data about geometry
substitute geometry for data and you have xml is data about data
this is not at all coherent, so don''t worry, its not you
in short
I CAN DO WHAT I WANT WITH MY FILE FORMATS AND YES IT IS MEANT TO BE USED THIS WAY!

I am doing it this way because I can easily create specific data files for effects, animations, special textures and other stuff as such.


As a note, maybe i could compile my vertex data, RLE it, and insert it into the userdata of a element. But that would fuck everything up so probably no

Well, have fun with loading 50 models with 2K polys each.
I had an ascii file format too, at the very beginning of my engine. I gave up using ascii data after being bored waiting for my maps to load.

quote:

---

Original post by Raduprv
Well, have fun with loading 50 models with 2K polys each.
I had an ascii file format too, at the very beginning of my engine. I gave up using ascii data after being bored waiting for my maps to load.

---

I think you are missing what my topic is about





COMPRESSION
err...
yeah

Also, how fast is a decompression algo in a vertex shader? could i stream in compressed vertex data and decompress it??

Well, they will compress pretty well, if that''s what really matters for you. I estimate a 90% compression.

quote:

---

Original post by Grizwald
Is it worth it?

---

No. I can see good use of XML for various geometry, but not your. I''m assuming you''re not very familiar with XML or you haven’t worked with it _enough_ so I''m not going to flame you for using a metadata language (and single tags, for that matter) to describe geometry.

Using XML does not make you look cool and new-age.

quote:

---

Original post by Raduprv
Well, they will compress pretty well, if that''s what really matters for you. I estimate a 90% compression.

---

I''m just trying to piece together a kick-ass data file system. This allows me to sort through ideas and create data files easily. I could maybe use this as an intermediate format, and compile the data and place it in the elements....but that messes up my well-formed xml

File systems are binary for a good reason. Needing to parse XML data is relatively slow, if done all the time.

quote:

---

Original post by aftermath

quote:

---

Original post by Grizwald
Is it worth it?

---

No. I can see good use of XML for various geometry, but not your. I''m assuming you''re not very familiar with XML or you haven’t worked with it _enough_ so I''m not going to flame you for using a metadata language (and single tags, for that matter) to describe geometry.

Using XML does not make you look cool and new-age .

---

I am using attributes because they are easier to parse

This is all in its infancy, i just wanted to get up a sub-system of datafiles. I am using XML because I like it. It makes perfect sense for describing data for me.

topic aside, I agree with raduprv. While XML can be used that way, I would consider the XML bloat. It takes a decent amount of extra space and processing to load XML. I really hate waiting for load times. If you have a lot of geometry, the XML will be a downside to your engine. Take the game soul reaver. Big world for the playstation, the only load time is a short one in the beginning. The rest is loaded seamlessly. That would probably not be possible with XML. XML is great for many things, but I am not sure sure it is great for geometry. Plus, is no point in making geometry human readable.

quote:

---

Original post by Witchcraven
topic aside, I agree with raduprv. While XML can be used that way, I would consider the XML bloat. It takes a decent amount of extra space and processing to load XML. I really hate waiting for load times. If you have a lot of geometry, the XML will be a downside to your engine. Take the game soul reaver. Big world for the playstation, the only load time is a short one in the beginning. The rest is loaded seamlessly. That would probably not be possible with XML. XML is great for many things, but I am not sure sure it is great for geometry. Plus, is no point in making geometry human readable.

---

The file i gave at the top is just a demo. Vertex data would most likely be REALLY compressed and fed to a shader. I wanted to use XML so I can have a universal data format. As you can see, i already have type attributes. I have an XML loader that returns a corresponding object based on the files type. The given example returns a "geometry chunk"

Once i get my engine caching vertex data properly, I don''t think loading time will be much of a problem. Just load one hugeass vertex buffer, and do with it as I please.

I''m considering a global vertex data file, with each model having nothing but indices (or a few more Vertex buffers: dynamic, static, whatever). That is why I want XML. It is already there, I already have parsers for it. It does what i need to, so i don''t need to reinvent the wheel. So what if my vertex buffer is a kilobyte larger than needed. My executable will be 40K smaller because i don''t individual loader classes.

Edit: nevermind. The above post clarified it.

[edited by - aftermath on June 6, 2004 12:40:58 AM]

The only sensible reason for using XML (for this kind of thing) is for prototyping. As you say "That is why I want XML. It is already there, I already have parsers for it. It does what i need to, so i don''t need to reinvent the wheel."

The only thing is - it''s slow as heck compared to loading in a binary file.

I''m using XML right now for my geometry data, but there''s no way that''s going to make it into the final product.

If you want to compress XML, put it in a zip file. Use ZLib to get it out and read it.

Using XML to store what would normally be binary data isn''t the best way to use XML. Adding binary metadata to XML is just asking for trouble.

XML is good for geometry while it''s still being altered by the tool chain, because it''s flexible, and can be fixed by hand in the rare cases where that''s necessary.
It''s terrible for storing final data to be loaded by the game though, due to the high parsing overhead.

So... uh... yeah, what everyone else said.

As for compression, RLE won''t get you much, but other compression methods (not just the complex ones) should give a pretty good compression ratio. RLE won''t be much use because you don''t have long sections of repeated characters (mainly the indents if you''re using spaces, and repeated 0''s in your vertex data, I''d have thought).
What you want is a compression scheme that references strings that have appeared before; I would suggest you just use zlib rather than writing your own routines, but it''s your decision.

John B

I like XML. I like XML a lot. XML is a great way to store data in a human editable format.

However, for data that doesn''t need to be human editable, binary data is the way to go.

Make a little utility to compile your XML data into a binary format, and will also convert back to XML.

On a conceptual level, XML data isn''t all that different from, say, the format of a WAV or AIFF file. It''s just represented differently.

I don''t get it. Either you don''t care about performance and file size and you want a human readable file format and then there is no need for compression, or you DO care about performance and file sizes and then using XML the way you are is idiotic.

Simple Geometry Compression

If that is what you''re after. There are more complex ways to do geometry compression, just search in citeseer, but I''d guess it''s overkill in your case considering the current way you store geometry.

Does anyone have any actual statistics on how much slower XML is to parse compared to a binary format? Do any of you know for a fact that the binary equivalent of the XML file will be slower or are you just guessing? If the times are insignificant, I would much rather have a human readable version of my data files, along with other advantages such as easy transformations and querying.

As for the compression, just use an implemented library that does the compression for you. They''ll give you much better compression than RLE.

Well, I used to store our models as .asc files (.asc is a 3D max format). It''s faster than XML, because you don''t have to look for tags and tags endings, etc.
And let''s just say that it took like 5 seconds to load a big tree (1K triangles).
The binary equivalent is loaded in a few MS.