Small Text Compression Algorithm

Why not use some sort of huffman compression? That works particularly well for text, and you can choose to use a different tree thingy (I forget the word) for each buffer of text, or just use the same on for all text.

Also, what's wrong with TEA?
EDIT: Oh, it's for encryption >_<

If you want real encryption, you need to use a real encryption system, and you need to implement the protocols well and manage your keys correctly. Designing your own, or using something not peer reviewed and well researched is just asking for trouble, because SOMEONE will break it if they work at it. Whether you use DES, Twofish, AES (Rijndael), or RSA depends on your needs (symmetric vs. public key), but there's code available online, and discussions of proper implementation in the books "Practical Cryptography" (an easy read) and "Applied Cryptography" (a much less easy read).

If all you want to do is obfuscate some text, huffman compress it and XOR encrypt it. The unskilled and unmotivated will be confused, but the die-hard hackers will figure it out in short time and release a tool to undo your work for everyone.

Just remember to compress before encrypting.

Thanks for the copy-pasted "hackers are under your bed" rant StaticEngine, but I'm a grown-up programmer now...

Quote:

Original post by StaticEngine If you want real encryption, you need to use a real encryption system, and you need to implement the protocols well and manage your keys correctly. Designing your own, or using something not peer reviewed and well researched is just asking for trouble, because SOMEONE will break it if they work at it. Whether you use DES, Twofish, AES (Rijndael), or RSA depends on your needs (symmetric vs. public key), but there's code available online, and discussions of proper implementation in the books "Practical Cryptography" (an easy read) and "Applied Cryptography" (a much less easy read). If all you want to do is obfuscate some text, huffman compress it and XOR encrypt it. The unskilled and unmotivated will be confused, but the die-hard hackers will figure it out in short time and release a tool to undo your work for everyone. Just remember to compress before encrypting.

He's asking for compression, you're saying random crap about encryption.

Anyways, to the OP:
If you're only going to use the lower 128 chars in the ascii table, you can do a funky trick, remove the most significant bit of each byte (since it's not used for you) then stick all the 7-bit characters together (so 8 bytes become packed in 7 bytes) and then use zlib on the new stream. This way you get the advantage of zlib plus an extra guaranteed 12.5% compression for removing 1/8 of the data.

What sort of text are you compressing? Is it single lines of text, or large chunks of text?

The most common method I know of for compressing small sequences of text (such as chat text in games or IM programs that are from 0 to say 512 bytes - too small for regular algorithms because the header negates savings) is to use huffman compression OR some form of LZ* where the dictionaries/trees/statistics/etc are generated a single time from some representative text (such as a huge english document) and thereafter remain unchanged. Basically, you 'hard code' the statistics and presume all messages will use simmilar statistics. Of course, this means that in practice some messages grow larger, but typical text is reduced in size a small amount

Here's a simple idea which would compress most text documents by about 5 times and is not too mathematically complicated. But it would probably only be very efficient if you have hundreds of documents.

First get hold of a dictionary in txt form, then number each word in the dictionary 1...30000. Then translate all your documents replacing each word with a short integer (2-bytes). Any word that is not in the dictionary say "squirgle" replace with (0,8,S,Q,U,I,R,G,L,E). You will also need a few codes for full-stop, comma, numbers, etc. Now provided all your documents contain mostly words from the dictionary and that you have considerably more words in your documents than in the dictionary you will have good compression!!!

To be honest I don't know how good this would be. I just made it up just now.

Quote:

Original post by Pouya He's asking for compression, you're saying random crap about encryption.

I misread the TEA line to mean that he was looking for encryption as well as compression.