Compression question

Quote:

Original post by WebsiteWill More questions... Hopefully search will be fixed soon as I am sure this stuff is already out there. Anyway, here goes. What kinds of compression/compression software are being used in games? I'm assuming that the pros likely apply a crc to their data and then compress (or vice versa) then expand the data as necessary at startup or even at runtime, depending. But what are they using to do this? In-house software, 3rd-party like PKZip or WinZip? Something other (is there anything other)? I promise I'll stop asking questions when the search is running again :) That, and I've had extra time this weekend to work on hobbies so more questions came up that normal. Don't hate me :) Thanks, Webby Edit** Placing your username as the subject line will not likely generate many helpful replies ;/

First, please don't feel like you have stop asking questions or that people will hate you because you don't know the answer to something. Alot of the traffic on these boards are question, with out them these boards would pretty dead. The majority of people on these board are very helpful.

As for your question, It depends on the developer but they probably aren't using external 3rd party software like WinZip to handle all their games compression needs. Likely they're using a compression library like Zlib[http://www.gzip.org/zlib], Bzip[http://sources.redhat.com/bzip2/], or a proprietary in house solution.

I hope this helps.

Thanks prh99. I wasn't actually wooried that people would get annoyed by me asking questions, it's just that a few of the questions I am sure are already answered here, but it's near impossible to find them right now :)

Thanks for the answer. I checked out PKWare last night and don't really want to pay $1000 + for a developers license, not to mention redistribution costs if I ever need to...

Webby

There aren't that many novel compression algorithms out there; I don't think anyone would care if you used zlib in your game. Actually I think there is an article somewhere on gamedev about making a virtual filesystem out of a zip file.

I would point that the license for zlib is in the zlib.h header file. According the license (copied below)you can use zlib in your games. Since bzip is written by the same guy, I assumming that the license is similar.

Copyright (C) 1995-2003 Jean-loup Gailly and Mark Adler  This software is provided 'as-is', without any express or implied  warranty.  In no event will the authors be held liable for any damages  arising from the use of this software.  Permission is granted to anyone to use this software for any purpose,  including commercial applications, and to alter it and redistribute it  freely, subject to the following restrictions:  1. The origin of this software must not be misrepresented; you must not     claim that you wrote the original software. If you use this software     in a product, an acknowledgment in the product documentation would be     appreciated but is not required.  2. Altered source versions must be plainly marked as such, and must not be     misrepresented as being the original software.  3. This notice may not be removed or altered from any source distribution.

Cool.

A guy at work here also gave me this to look over.
http://www.icsharpcode.net/OpenSource/SharpZipLib/Default.aspx

I'l evaluate them all tonight and make a choice.

Thanks guys,
Webby

It ofcourse depends on the data. From a console programming point of view:

- If you want packed files (archives), then you're already able to start with the other ppl's answers (i.e.: use Zlib or Blib or some other nice library :)
- Some consoles have decompression routines in ROM. Some developers stick to them, some don't. (i.e.: GBA has Huffman, LZW, RLE and delta (difference) decompressors in system ROM.)
- If you really need high decompression speed (or a high compression rate for a well-known set of data), it is often best to write a custom compressor/decompressor.
- For textures DXT compression is used when the hardware can decompress it for you.
- MPEG2 is used a lot for video and audio on Dreamcast.
- ADPCM is used a lot for soundeffects in general (when PCM is not the way to go).
- MPEG4, ATRAC3(plus) and ADPCM is currently the choice for PSP a/v compression.

Is there a way to deserialize from a compressed file without first having to uncomress the file?

What I am thinking is if I store my terrain data in a zipped file. When the terrain engine needs it I will have to open and uncompress the zipped file (into a new file), then I can deserialize from that new file. However, at this point, the uncompressed data would be on the users machine in a readable (or at least uncompressed) format. Is there a way I can get around this?

Thanks,
Webby

Decompress directly into memory? (of course, that's not foolproof, what with VM and all...)

Most compressed data formats are fairly serial; i.e. you can extract them a small piece at a time, and it's not too hard to figure out how far you need to read ahead in order to have enough compressed data for a real data element. However, it gets pretty messy if you try to do something like that manually. You should see if whatever API you're using supports the operation natively (i.e. decompress from 'stream' to memory).

For game-textures vector quantization is pretty good,
although it doesn't suit to low-frequency content very well.

You can achieve bitrates well below 1 BPP with outstanding quality (relative to data size). It also can be decompressed on the fly, which makes it a good choice
when you deal with images in software (software rendering, decompressing from files, etc.).

However, there is a drawback, compression is tedious, that is to get good results you have to perform very long iteration-cycles during compression.
This is why I use VQ as the last step in content production, when artwork is finished I compress it in a over-night-batch-job.

For a regular texture I use a codebook at most 1024 entries long, vectors in 12 dimensions ( 4 Texel à 3 scalars -> 12 scalars ).
For most textures containing only few areas of low frequency (terrain textures, etc.), I can shrink the image to <0.3BPP without sacrificing quality to much,
at least a regular player won't notice the difference.

I read that VQ is also used to compress sound, but in much higher dimensions.

Hope that helps a bit


On the left side the original image, on the right the compressed one, on the bottom are a few close-up comparisons:

Thanks Martin

I'm not certain yet but I don't think I will be needing to umcompress textures (or any data) on the fly. I plan to have specific points set to loading so all new terrain and terrain elements and textures can be loaded into memory. Otherwise, it's just a matter of keeping the character data in memory at all times.

Now, if if my resource sets for models becomes excessively large then I won't want to stop just to load in the characters T-Shirt when he changes armor :) so I will then implement some sort of streaming functionality.

Heck, I might go ahead and do it anyway just so it's in from the beginning instead of as an afterthought.

Thanks,
Webby