Skip to main content
gamedev.net
🔒 Locked

[java] NIO server programming

Started by SteveTaylor 4 replies 2.5k views
Original Post
SteveTaylor
SteveTaylor
After receiving some good advice on these forums, I've been playing with asynchronous socket I/O using java.nio. The problem I have with the examples available all over the place is that they deal properly with input but not output. For instance, Sun's own examples use non-blocking input but use blocking output (using a PrintStream object). That's poor form from Sun. Other examples I've seen use busy loops for the output. That's just wrong. So I've been thinking of how to handle input. When a connection is accepted and its channel is registered with the selector, at first it would make sense to register it as interested in reading and writing. Question #1: If the connection is ready to send, will SelectionKey.isWritable() return true on every select? This seems to be a waste of time if there's nothing to send. Question #2: In a typical game server, would it be practical to keep calling interestOps(int ops) to register interest in writing only when there's something to write, or would that crap out select loop due to the possibility of interestOps blocking? Question #3: Is there a nice (and proven) way of toggling interest in writing that avoids blocking? I was thinking of using a ConcurrentLinkedQueue in its own thread to process interestOps operations. Would this work? Thanks in advance for any advice.
SteveTaylor
SteveTaylor
Here I go, replying to myself again...

I just had another idea that is really simple (obvious) and possibly better.

Use a ConcurrentLinkedQueue to queue write operations, processing them in a thread. The SocketChannel.write(ByteBuffer src) operation doesn't block unless another thread is trying to write to the same channel. I won't need to worry about that because it doesn't make sense to write two or more messages to the same channel concurrently, especially because there is no guarantee that the entire message is sent at once.

It makes me wonder why anyone would write a program that registers a SocketChannel with OP_WRITE as an interest operation.
SteveTaylor
SteveTaylor
And here's the code. Please feel free to criticise it.

[source="java"]package game.server;import java.io.*;import java.nio.*;import java.nio.channels.*;import java.util.*;import java.util.concurrent.*;/* * This class implements a multiplexed output system for multiple * socket channels. It takes requests to write ByteBuffers into * the given SocketChannels. It processes these requests in its own * thread. This is implemented internally using a ConcurrentLinkedQueue. */public class MultiplexedSocketWriter{	private static MultiplexedSocketWriter instance = null;	private SocketChannelOutputBufferPool bufferPool = SocketChannelOutputBufferPool.getInstance();	private ConcurrentLinkedQueue<SocketChannelOutputBuffer> outputQueue = new ConcurrentLinkedQueue<SocketChannelOutputBuffer>();	private boolean terminateThread = false;		/*	 * Start sending the message queue.	 */	private MultiplexedSocketWriter()	{				new Thread(new Runnable()		{			public void run()			{				while (!terminateThread)				{					while (!outputQueue.isEmpty())					{						Iterator<SocketChannelOutputBuffer> iter = outputQueue.iterator();						while (iter.hasNext())						{							SocketChannelOutputBuffer buffer = iter.next();							buffer.send();							if (buffer.sent())							{								bufferPool.release(buffer);								iter.remove();							}						}					}					// No messages to send, so wait until notified.					try					{						synchronized (MultiplexedSocketWriter.this)						{							wait();						}					}					catch (InterruptedException exc)					{					}				}			}		}).start();	}		/*	 * Get the only instance of MultiplexedSocketWriter	 */	public static MultiplexedSocketWriter getInstance()	{		if (instance == null)		{			instance = new MultiplexedSocketWriter();		}		return instance;	}		/*	 * Terminate this MultiplexedSocketWriter. It will never be restartable.	 * Once calling this method, no more messages will be able to be sent	 * using this writer for the remainder of program execution. This method	 * should be called if and only if the program must exit. Calling this	 * method will cause the internal thread to terminate once all pending	 * messages have been sent.	 */	public void terminate()	{		terminateThread = true;	}		/*	 * Send the ByteBuffer via the SocketChannel. This actually places	 * the ByteBuffer in a queue to be sent as soon as possible.	 */	public void send(SocketChannel channel, ByteBuffer buffer)	{		outputQueue.add(bufferPool.acquire(channel, buffer));		// Notify the queue consumer to wake up in case it was waiting.		synchronized (this)		{			notify();		}	}			/*	 * Mutable wrapper for a SocketChannel and a ByteBuffer to send	 * through it. Also contains a send method to signal the SocketChannel	 * to send the ByteBuffer.	 */	private static class SocketChannelOutputBuffer	{		private SocketChannel channel;		private ByteBuffer buffer;			/*		 * This constuctor does nothing in particular. It is only		 * of interest to the SocketChannelOutputBufferPool class.		 */		public SocketChannelOutputBuffer()		{			channel = null;			buffer = null;		}				/*		 * Set the SocketChannel and ByteBuffer to use.		 */		public SocketChannelOutputBuffer set(SocketChannel channel, ByteBuffer buffer)		{			this.channel = channel;			this.buffer = buffer;			buffer.flip();			return this;		}				/*		 * Send the ByteBuffer through the SocketChannel. This may or		 * may not send the entire ByteBuffer. Subsequent calls to this		 * method will attempt to send any remaining bytes.		 */		public void send()		{			try			{				channel.write(buffer);				if (!buffer.hasRemaining())				{					buffer.clear();				}			}			catch (IOException exc)			{				buffer.clear();		// This causes the next call to sent() to return true.				try				{					channel.close();	// Close this channel.				}				catch (IOException ignore)				{				}			}		}				/*		 * Returns the sending status of the ByteBuffer (true is send completed).		 */		public boolean sent()		{			return !buffer.hasRemaining();		}	}			/*	 * Pool of SocketChannelOutputBuffers. This saves newing an object each time	 * a message is to be sent. Let's hope LinkedList uses node pooling, otherwise	 * this is a waste of code!	 */	private static class SocketChannelOutputBufferPool	{		private static SocketChannelOutputBufferPool instance = null;		private static final int initialSize = 100;		private List<SocketChannelOutputBuffer> pool = new LinkedList<SocketChannelOutputBuffer>();				private SocketChannelOutputBufferPool()		{			allocate();		}				/*		 * Get the only allowable instance of this pool.		 */		public static SocketChannelOutputBufferPool getInstance()		{			if (instance == null)			{				instance = new SocketChannelOutputBufferPool();			}			return instance;		}				/*		 * Acquire a SocketChannelOutputBuffer from this pool		 * and set it to the given parameters.		 */				public SocketChannelOutputBuffer acquire(SocketChannel channel, ByteBuffer buffer)		{			if (pool.isEmpty())			{				allocate();			}			return pool.remove(0).set(channel, buffer);		}				/*		 * Release the SocketChannelOutputBuffer back into this pool.		 */		public void release(SocketChannelOutputBuffer buffer)		{			pool.add(buffer);		}				/*		 * Allocate some SocketChannelOutputBuffers in this pool.		 */		private void allocate()		{			for (int i = 0; i < initialSize; ++i)			{				pool.add(new SocketChannelOutputBuffer());			}		}	}}
Aldacron
Aldacron
Quote:
Original post by SteveTaylorIt makes me wonder why anyone would write a program that registers a SocketChannel with OP_WRITE as an interest operation.


Exactly. In practice you don't need to register for OP_WRITE at all. Select readable sockets, but write to them anytime.

Quote:
And here's the code. Please feel free to criticise it.


I just glanced through, but one thing that really stands out is your use of the ConcurrentLinkedQueue. I suggest you replace it with a LinkedBlockingQueue. You've implemented the same functionality yourself, anyway. In your output thread, call the queue's take() method. If the queue is empty, the method will block. It will return only when another thread adds an object to the queue. You can even set a capacity on the queue, if you need to, and any method that adds objects to it will block until space comes available. Before the BlockingQueue interface was added to the core API, such functionality was often implemented manually (like you have done). No need to do it anymore.

[Edited by - Aldacron on January 26, 2006 8:12:23 AM]
SteveTaylor
SteveTaylor
Thanks for the info. That's a good idea. I was using the ConcurrentLinkedQueue so i wouldn't have to worry about synchronizing it. Lazy me. Since my code isn't working properly anyway, now's a good time to try the LinkedBlockingQueue. I'll post the update when it's done.

Cheers.
- Steve
SteveTaylor
SteveTaylor
I've learnt that blocking queues are actually a bad thing to use in this case. Why? Consider the thread that's pulling items off the queue and sending them to their channels. Sometimes a message is too big for a channel to process all at once, so it must be placed back onto the queue. But what happens if, between popping off the queue and placing it back on, the other thread places a new message on the queue for that same channel? The new message will be inserted in the middle of the old message. So we must make the process of popping off the queue, partially writing, and placing the rest back on the queue atomic. So we must use synchronized. The problem is that when BlockingQueue.take() is called inside a synchronized block, there's quite a big potential for deadlocking. It happened to my code every time.

I was thinking of all sorts of work arounds, such as using a hashtable to prevent duplicate elements. In the end, I decided to use a LinkedList as a queue and make the abovementioned process atomic using a synchronized block. I deliberately chose LinkedList because it doesn't perform any blocking whatsoever, so no risk of deadlocks. And I'm using wait() and notify() again. It all works quite well now.

I've been to java.nio hell and back.

EDIT: May as well post code.

MultiplexedSocketWriter.java
import java.io.*;import java.nio.channels.*;import java.util.*;/* * This class implements a multiplexed output system for multiple * socket channels. It takes requests to write ByteBuffers into * the given SocketChannels. It processes these requests in its own * thread. This is implemented internally using a ConcurrentLinkedQueue. */public class MultiplexedSocketWriter{	private static MultiplexedSocketWriter instance = null;	private LinkedList<OutputWrapper> outputQueue = new LinkedList<OutputWrapper>();	private boolean terminateThread = false;	private Runnable thread = new OutputThread();		/*	 * Start sending the message queue.	 */	private MultiplexedSocketWriter()	{		new Thread(thread).start();	}		private class OutputThread implements Runnable	{		public void run()		{			while (!terminateThread)			{				// Don't allow the output queue to be modified just in case the output				// cannot be sent entirely. For any given channel, we don't want new				// output to be sent before old output.				synchronized (OutputThread.this)				{					while (outputQueue.isEmpty())					{						try						{							System.out.println("Waiting...");							wait();						}						catch (InterruptedException exc)						{						}					}					System.out.println("Got notified.");					// Take an output message from the queue					OutputWrapper output = outputQueue.removeFirst();					// Send the buffer.					try					{						output.buffer.get(output.channel);					}					catch (IOException exc)					{						// Not sure what to do yet. Perhaps just ignore silently.						// This could work because this indicates the channel is						// now closed, which can be handled in the selector loop.						System.out.println("Encountered IOException while draining output queue.");					}					// If the buffer has not been completely sent then put it back into the queue.					if (!output.buffer.isEmpty())					{						outputQueue.addLast(output);					}				}			}		}	}		/*	 * Get the only instance of MultiplexedSocketWriter	 */	public static MultiplexedSocketWriter getInstance()	{		if (instance == null)		{			instance = new MultiplexedSocketWriter();		}		return instance;	}		/*	 * Terminate this MultiplexedSocketWriter. It will never be restartable.	 * Once calling this method, no more messages will be able to be sent	 * using this writer for the remainder of program execution. This method	 * should be called if and only if the program must exit. Calling this	 * method will cause the internal thread to terminate once all pending	 * messages have been sent.	 */	public void terminate()	{		terminateThread = true;	}		/*	 * Send the SocketChannelOutputBuffer via the SocketChannel. This actually places	 * the SocketChannelOutputBuffer in a queue to be sent as soon as possible.	 * If the buffer is already in the queue, this method returns true (success).	 * If the queue is full, the buffer is not added to the queue and this method	 * returns false.	 */	public void send(SocketChannel channel, SocketChannelOutputBuffer buffer)	{		synchronized (thread)		{			System.out.println("Sending...");			outputQueue.addLast(new OutputWrapper(channel, buffer));			thread.notify();		}	}		private static class OutputWrapper	{		public SocketChannel channel;		public SocketChannelOutputBuffer buffer;				public OutputWrapper(SocketChannel channel, SocketChannelOutputBuffer buffer)		{			this.channel = channel;			this.buffer = buffer;		}	}}


SocketChannelInputBuffer.java
import java.io.*;import java.nio.channels.*;public interface SocketChannelInputBuffer{	public int put(SocketChannel channel) throws IOException;	public SocketChannelInputBuffer put(byte b);	public SocketChannelInputBuffer put(byte[] src);	public SocketChannelBuffer put(SocketChannelBuffer src);	public boolean isEmpty();}

SocketChannelOutputBuffer.java
import java.io.*;import java.nio.channels.*;public interface SocketChannelOutputBuffer{	public int get(SocketChannel channel) throws IOException;	public byte get();	public SocketChannelOutputBuffer get(byte[] dst);	public boolean isEmpty();}

SocketChannelBuffer.java
import java.nio.*;import java.nio.channels.*;import java.io.*;public class SocketChannelBuffer implements SocketChannelInputBuffer, SocketChannelOutputBuffer{	private static final int STATE_READ = 1;	private static final int STATE_WRITE = 2;	private ByteBuffer internalBuffer;	private int size;	private int byteCount;	private int state;		public SocketChannelBuffer(int size, boolean direct)	{		this.size = size;		internalBuffer = direct ? ByteBuffer.allocate(size) : ByteBuffer.allocateDirect(size);		internalBuffer.clear();		byteCount = 0;		state = STATE_READ;	}		public synchronized int put(SocketChannel channel) throws IOException	{		if (byteCount == size)		{			return 0;		}		setRead();		int bytesRead = channel.read(internalBuffer);		// We don't want to update byteCount if bytesRead is -1.		// May as well also catch btyesRead == 0 for efficiency.		if (bytesRead > 0)		{			byteCount += bytesRead;		}		return bytesRead;	}		public synchronized int get(SocketChannel channel) throws IOException	{		if (byteCount == 0)		{			return 0;		}		setWrite();		int bytesWritten = channel.write(internalBuffer);		byteCount -= bytesWritten;		return bytesWritten;	}		public synchronized SocketChannelBuffer put(byte b)	{		if (byteCount == size)		{			throw new BufferOverflowException();		}		setRead();		internalBuffer.put(b);		++byteCount;		return this;	}		public synchronized byte get()	{		if (byteCount == 0)		{			throw new BufferUnderflowException();		}		setWrite();		--byteCount;		return internalBuffer.get();	}		public synchronized SocketChannelBuffer put(byte[] src)	{		if (byteCount + src.length > size)		{			throw new BufferOverflowException();		}		setRead();		internalBuffer.put(src);		byteCount += src.length;		return this;	}		public synchronized SocketChannelBuffer get(byte[] dst)	{		if (byteCount < dst.length)		{			throw new BufferUnderflowException();		}		setWrite();		internalBuffer.get(dst);		byteCount -= dst.length;		return this;	}		public synchronized SocketChannelBuffer put(SocketChannelBuffer src)	{		while (!src.isEmpty() && byteCount < size)		{			this.put(src.get());		}		return this;	}		public synchronized boolean isEmpty()	{		return byteCount == 0;	}		private void setRead()	{		switch (state)		{		case STATE_READ:			return;		case STATE_WRITE:			if (byteCount == 0)			{				internalBuffer.clear();			}			else // Still some bytes remaining.			{				System.out.println("Compacting.");				internalBuffer.compact();			}			state = STATE_READ;		}	}		private void setWrite()	{		switch (state)		{		case STATE_READ:			internalBuffer.flip();			state = STATE_WRITE;			return;		case STATE_WRITE:			return;					}	}}

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.