Encryption on the network

one way you could encrypt is a hashfunction that distributes the information to 128 bit the weakness is if someone backengineers a client program he can easily get the hash function and thus could decode your information

I have heard about software that rewrites its program code at runtime but i guess this would be to difficult to do infact i have no clue how this could work on windows or linux systems

Have a look at IPSec, particularily the ESP Protocol, which is responsible for encrypting packet data. You need a public/private key pair and a shared secret.

Here's more detail: IPSec Howto

You can probaly find something useful in the OpenSWAN Source Code as well.

I use OpenSSL for all my encryption needs (not just SSL). At least one major MMO out there uses http://www.libtomcrypt.org/ (not naming names...).

This depends on the degree of encryption you want. For example, if you just want to stop script kiddies from logging the text floating back and forth, it'd be a lot more productive to use zlib to compress the streams. Of course, the decompression part is publically known, so its easily cracked and not apropriate for things like passwords. Then you'll want to investigate into SHA2 hashing, which is, as far as I know, irreversible. Still not secure because then spoofers can just log the hashes and send those. Then along comes stuff like Blowfish two-way encryption, which is actually beyond the scope of my knowledge, but I'm pretty sure you start getting into insecure channels and certificates and the like.

If the information if vitally secure, and I mean like credit care number and expiration date secure, then you'll probably save yourself a lot of time and legal grief by acquiring the services of a middleman. Doesn't Paypal do secure credit payments?

Define streams. For simple communication between client and server applications, OpenSSL is your best bet. To encrypt the entirety of the communications, such as to protect a wireless link, the IPSec suggestion and other tunnelling/VPN protocols are your best bet.

[edit: though as basiror alludes to, if the suspected malicious entity has root/admin access to either side of the communication, it's rather trivial to read the data after the app decrypts the messages]