🔒 Locked

Transfer Data in Java

Started by Sephirox Mar 19, 2007 at 7:03 PM 3 replies 1.2k views

Original Post

Mar 19, 2007 at 7:03 PM

Hello, I wish to pose a question to the vast body of knowledge which is GameDev:) For a project in 3rd Comp. Eng., I have to make a P2P file transfer program. I am specifically working on the File Overlay using Kademlia topology. Anyways, the project has to be in Java. I ran into trouble when the .write() function of an OutputStream class does not return an integer (containing the number of bytes written). Here's the story: everything was working, except when transferring non ASCII files (ie. a bitmap or jpeg etc). So, the logical thing to do is move from sending data over a PrintStream to sending it over the OutputStream. So, everything is good while programming, except that .write() returning void looks very fishy based on my socket programming experience in C/C++... anyways, finish the code and test it out. Well, it works some of the time:) Ok, so debugging it reveals that some threads finish the read as expected, while others will read a few bytes and then will wait forever for the rest (example: thread1 could get 1024 out of 1024 bytes as expected. but thread2 could get only 6 out of 1024). Now, .read() actually returns an integer, so I can make sure I read the right amount, but unfortunately I can't see a way to make sure I send the entire array of bytes I give to the .write() function. So, can anyone help me out here? This IS the reason it's not working right (I'll look pretty dumb if it's not:) )? I've checked out some stuff online - SocketChannel has .write which returns integer, but I don't think thats where I should go with this... PS: I would be using .write(byte[], int off, int len) if it would return int - since it doesn't its no different than .write(byte[]) or .write(byte[], int len). GRRRRRR, I wish this were in C++ :) (not at home in Java yet, so I'm biased towards C++, lol) Thanks in advance

Wyrframe

327

Wyrframe

Mar 19, 2007 at 11:55 PM

After you write(), often you want to flush() to gaurantee the data actually gets sent. That's my first guess.

Antheus

216

Antheus

Mar 20, 2007 at 6:17 AM

Streams work exactly like std::cout. No return values there either.

While sockets do return a certain value, this doesn't mean the data got sent. They merely report they forwarded an ammount of data to underlying buffers.

So returning something on write is redundant, once you write it, it's written and you're either done with it, or you'll get an exception indicating failure in the network stack, identically to what socket error would be, just different mechanism.

If you're using TCP/IP, then keep in mind that you will not always receive the data in same chunks as you sent them. Sending x bytes will not result in x bytes, it may get fragmented, or several blocks will be received in one call.

If you're doing one read for each write, then you'll miss on a lot of data.

For sending large ammounts of data it always makes sense to include the description of the data (length, or data type). The socket reader then first reads this description, then incrementally fills the buffer with incoming data.

If you want to know how much client actually received, you need to write your own ack mechanism, indicating which data was received (even with TCP/IP). That is the only way to know that client did indeed process the data sent - TCP/IP only guarantees delivery, and closes connection if that fails, not knowing which data wasn't received.

Sephirox

Author

Sephirox

Mar 20, 2007 at 11:42 AM

Wyrframe - Yes, I had the flush in there originally, but it still had the problem. I just added it back in, same problem. Thanks for the suggestion.

Antheus - it's not one read for each write - here's the exact code i'm using:

Server code:

DataOutputStream binout = new DataOutputStream(socket.getOutputStream());

//binaryData is a byte[] with the data I want to send
binout.write(binaryData, 0, binaryData.length);

binout.flush();

Client Code:

DataInputStream binin = new DataInputStream(socket.getInputStream());

//numToRead is known prior to this
while(numRead < numToRead)
{
n = binin.read(tempData, numRead, numToRead - numRead);
if(n == -1)
break;

numRead += n;
}

Ok, so, on the read side, I can make sure that I read exactly 'numToRead' bytes. Now, with C sockets (or WinSock), you do the same thing with write:

while(numWritten < numToWrite)
{
//neglecting error returns and coding from memory here
numWritten += send(sock, buf + numWritten, numToWrite - numWritten);
}

basically, the first binin.read() could read say, 17 bytes. if numToRead == 5120 for example, then it will loop and try to read the remaining bytes, starting from offset 17 in the tempData array. What I'm getting is the second .read() will block until it receives the rest, but the rest never comes.

I'm just not sure what Java is or isn't doing for me in the call to .write(). Do I need the looping around the .write() as in C or no? If no, then is my loop around .read() redundant??
and (if no need for looping around .write()), then why does all data get sent sometimes but not always?

For now, I've set the socket timeout to 1 sec, so if the first read cannot get all the bytes, then the second will timeout and an exception thrown (the piece of the file will be retried again sometime later).

Antheus

216

Antheus

Mar 20, 2007 at 4:35 PM

Ok, I messed around with this a bit.

Client

int n = 200000;byte[] values = new byte[n];socket = new Socket(InetAddress.getLocalHost(), 999);socket.setReceiveBufferSize(3 * n);            int nRead = 0;int offset = 0;while (nRead < n & offset > -1) {   offset = socket.getInputStream().read(values, nRead, (n - nRead));   nRead += offset;   System.out.println(socket.getLocalPort() + " received " + offset + " bytes.");}

Server

int n = 200000;byte[] values = new byte[n];for (int i = 0; i < n; i++) values = (byte)(i & 0xff);try {  socket.setSendBufferSize(3 * n);  socket.getOutputStream().write(values);  System.out.println("Server client @" + socket.getRemoteSocketAddress().toString());} catch (Exception e) {...}

One thing to look at is the socket receive and send buffer size. Although the largest block that socket can read at one time seems to be 65k, setting the buffer above that seems to make things run smoother.

Client checks for -1 (EOF signal) that effectively means the server closed connection.

I ran this test with 50 threads constantly requesting data and didn't notice any glitches whatsoever or any unusual exceptions.

Another thing is that if you already have byte[] array, just send it to output stream, or read it from input stream directly. You don't need readers or writers for that. I don't really remember right now how various readers transform the data, may have something to do with that.

Java 1.5, JDK runtime.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.