RSA Engine

I am using C# with the http://www.bouncycastle.org/ RSA Engine. I am trying to use a publicKey provided to me, to decrypt. Let me explain, because what I said so far is impossible. Currently, I have a client that shoots over an RSA. I am trying to mimic this (IE Create the client). I currently have working the RSA from the Client comes over, and I decrypt it.

        public static AuthPair DecryptRSAAuthPair(byte[] data)
        {
            if (data.Length != 176)
            {
                throw new CryptographicException("Invalid RSA Auth Pair");
            }
            byte[] encrypted = new byte[128];
            Buffer.BlockCopy(data, 1, encrypted, 0, encrypted.Length);

            byte[] decrypted = RSAEngine.DecryptValue(encrypted);

         }

Now. The packet being sent is something as such :

Writer.WriteInt32(sessionId);
            Writer.WriteInt32(0x0000c621); // Protocol Revision

            Writer.WriteByteArray(publicKey);

            Writer.WriteInt32(0x29DD954E);
            Writer.WriteInt32(0x77C39CFC);
            Writer.WriteInt32(unchecked((int)0x97ADB620));
            Writer.WriteInt32(0x07BDE0F7);

            Writer.WriteByteArray(blowfishKey);
            Writer.WriteByte(0x00);

So In my new client, Let me read the data :

reader.ReadInt32();
                    byte[] rsaKey = reader.ReadBytes(128); //TODO RSA key.  Probably have to descramble this, what a PITA
                    reader.ReadInt32();
                    reader.ReadInt32();
                    reader.ReadInt32();
                    reader.ReadInt32();
                    Encryption.SetRSA(rsaKey);
                    proxy.RandomBlowfishKey = reader.ReadBytes(16);

I get the RSA being used. And attempt to encrypt my own RSA back for MY server.

byte[] data = new byte[37];

            //Why? Always 36, unknown
            data[0] = 36;

            byte[] userNameBytes = Encoding.ASCII.GetBytes(userName);
            byte[] passWordBytes = Encoding.ASCII.GetBytes(passWord);

            Buffer.BlockCopy(userNameBytes, 0, data, 3, userNameBytes.Length);
            Buffer.BlockCopy(passWordBytes, 0, data, 17, passWordBytes.Length);

            byte[] encrypted = RSAEngine.EncryptValue(data);

But when I try to decrypt this data I just encrypted, I don't get my starting data array of 37 length, but an array of 128 and its garbarge when attempting to decode it. I know this is a lot in a small post, and probably unclear. Hopefully someone has worked a lot with RSA to be able to answer this one. I believe i have to use

public static void SetRSA(byte[] scrambledKey)
        {
            RSAParameters p = RSAEngine.ExportParameters(false);
            p.Modulus = scrambledKey;
            RSAEngine.ImportParameters(p);
        }

Somehow or another.