package encryption

I don't really know how fast it is, but maybe Blowfish would work:

http://www.schneier.com/blowfish.html

Also, XTEA seems a good choice. I don't know how fast or good it is, but the implementation is very simple.
You also can have a look at block ciphers on Wikipedia, maybe something will look appealing to you.

How big are the packets? If they can be any integral number of bytes, then XTEA or Blowfish will require padding up to the two-byte boundary. This will hardly destroy your data throughput, but it may require some implementation changes. You can avoid this by using Blowfish-32 or similar.

Can you also tell us what the packets contain? If you're sending people's souls over the internet, then you may want to look into something wiuth a bit of muscle such as AES or DES3. However, if you're encrypting simply to prevent hacking of an MMO, then security is barely even a concern: pick an algorithm that runs quickly, such as a weakened version of TEA (with fewer rounds than the recommended 64).

Admiral

Be careful that security protocols are often more important than encryption algorithms - it's easy to use e.g. 256-bit AES in a completely insecure way. See WEP for an example where professionals (but not security professionals) got the protocol horribly wrong.

Assuming you care about doing it properly, you should work out the requirements of the protocol (variable-sized messages? packet loss and reordering? peer-to-peer / server-client? authenticity to avoid spoofing? secret just from observers on the network, or from other participants in the game, or from players on their own computers? etc), and then find someone who knows a bit about security to design a protocol for those requirements (or to find an existing one, but I guess that's unlikely if you have quite specific requirements), and then preferably find some other people to try to find holes in it, and then implement it, and then hope nobody else finds any holes without telling you.

Quote:

If you're sending people's souls over the internet, then you may want to look into something wiuth a bit of muscle such as AES or DES3.

You might want an HMAC too - there's nothing worse than getting a corrupted soul because the system designers remembered to ensure secrecy but forgot about integrity ;-)

We are developing a MMO and the packets are random size. Maybe we should seperate sensitive information (like account number and password) from other packets.
I still have a doubt: is it possible that some people can scan the RAM and get to know our protocol?

Quote:

Original post by Keisni We are developing a MMO and the packets are random size.

Is it running over TCP? If it's running over UDP instead, is there some reliable stream layer built on top of that?

It's much easier when you do have a reliable stream - I believe you can just exchange secret keys at the beginning of the session (e.g. with Diffie-Hellman) and then use a stream cipher (I guess RC4 or XTEA or whatever) for the rest of the session, plus some integrity check. (But never ever reuse one key for multiple sessions.)

Quote:

I still have a doubt: is it possible that some people can scan the RAM and get to know our protocol?

You should assume that a user can see and control everything that is going on in their own computer. You should also assume that any attacker has all the source code to your program and knows precisely what algorithms and protocols you're using - the security should come from using secret keys (which the user still has access to, but nobody else does).

take a look at crypto++ library, i believe it concentrates most of the algorithms in use today.

Like others have said, the algorithm you choose is only a small portion of being secure. Protocol is another huge one, and properly using the selected algorithm is one of the most often done wrong. If you use a block cipher (and you might as well), then make sure you utilize the proper block cipher mode of operation. Look at the example of ECB and notice how easily patterns emerge when you incorrectly use encryption.