Skip to main content
GameDev.net gamedev.net
🔒 Locked

Blocking

Started by MaximusCranius Apr 4, 2005 at 8:30 PM 6 replies 1.8k views
Original Post
MaximusCranius
MaximusCranius
Hi, so basically I'm working on a wrapper for a client/server setup. As of right now, the server is setup using select(), and I'm trying to figure out how to deal with blocking on the client side. In a game, I was planning on polling for new data at the beginning of every x frames. The problem is, with blocking sockets, I don't want it to hold. My question is, in games are blocking sockets used in combination with threading, or are non blocking sockets used? Any guidance would be appreciated.
John Schultz
John Schultz
Quote:
Original post by MaximusCranius
Hi, so basically I'm working on a wrapper for a client/server setup. As of right now, the server is setup using select(), and I'm trying to figure out how to deal with blocking on the client side. In a game, I was planning on polling for new data at the beginning of every x frames. The problem is, with blocking sockets, I don't want it to hold.

My question is, in games are blocking sockets used in combination with threading, or are non blocking sockets used?

Any guidance would be appreciated.


Threading is not necessary. Use blocking sockets on both the client and server and use select() to check for available data to read. By polling with select(), you won't have to set the sockets to non-blocking (I recall some issues on Win32 when turning off blocking).

Example echo client-server with the client using select().
hplus0603
hplus0603
Using blocking sockets are usually not great, because you don't know how much data to ask for. With UDP, you can ask for a single datagram when select() returns ready, but with TCP, you only know there's a single byte available.

Typically, games use non-blocking sockets and select() in conjunction. There are no WinSock problems with non-blocking sockets, although there are some MFC socket class problems -- you don't want to be using MFC sockets in general.

If you put each blocking socket in its own thread, and use some hand-off to push data into the world update loop, then that will work, but will have scalability problems once you get into "lots" of players, where "lots" varies wildly depending on your implementation and OS.
enum Bool { True, False, FileNotFound };
John Schultz
John Schultz
Quote:
Original post by hplus0603
Using blocking sockets are usually not great, because you don't know how much data to ask for. With UDP, you can ask for a single datagram when select() returns ready, but with TCP, you only know there's a single byte available.

Typically, games use non-blocking sockets and select() in conjunction. There are no WinSock problems with non-blocking sockets, although there are some MFC socket class problems -- you don't want to be using MFC sockets in general.

If you put each blocking socket in its own thread, and use some hand-off to push data into the world update loop, then that will work, but will have scalability problems once you get into "lots" of players, where "lots" varies wildly depending on your implementation and OS.


WRT Win32 (Winsock) and blocking: in my tests I found that it was only possible to change the state of blocking once, after which the socket did not behave properly. While this may not be a problem if just changing the state once (the stack may even be fixed now), I chose to avoid a potentially unstable (internal TCP/IP stack) code path by using select() exclusively. Also, reading a non-blocking socket when no data is available returns an error (-1, internally EWOULDBLOCK). This complicates error checking code (WSAGetLastError() should be called to see if a real error has occurred). It's cleaner and more elegant to use select().

You can use blocking sockets with TCP as well (and polling), without issue:

My HTTP request code:
  bool httpRequest(char * hostName,char * request,char * response,int maxResponseLength,int timeOut) {    if (!s.createTCP()) return false;    if (!s.setAddressAndConnect(hostName,80)) return false;    if (!s.send(request,strlen(request)))     return false;    int bytesRead=0;    if (!s.read(response,maxResponseLength,bytesRead,timeOut)) return false;    if (!cleanupResponse(response,bytesRead)) return false;    s.close();    return true;  } // httpRequest


The read() function using select() and a blocking TCP socket:
  bool read(void * buff,int buffLength,int & bytesRead,int timeOut=0) { // For TCP reads    if (type != ST_TCP) return false;    char * data = (char *)buff;    while (1) {      bool error=false;      bool bytesReady = isReadable(error,timeOut);      if ((!timeOut || bytesReady) && !error) {        bytesRead = ::recv(s,data,buffLength,0);        if (bytesRead == SOCKET_ERROR) {          return false;        } // if        buffLength -= bytesRead;        if (buffLength == 0 || bytesRead == 0) {          bytesRead = data - (char *)buff;          return true;        } // if        if (buffLength < 0) return false;        data += bytesRead;      } else {        return false;      } // if    } // while  } // read


The isReadable() function (isWritable() is similar):
  // Also used for TCP listen()/accept(), if isReadable() after listen(), accept() won't block.  bool isReadable(bool & error,int timeOut=0) { // milliseconds    fd_set socketReadSet;    FD_ZERO(&socketReadSet);    FD_SET(s,&socketReadSet);    timeval tv;    if (timeOut) {      tv.tv_sec  = timeOut / 1000;      tv.tv_usec = (timeOut % 1000) * 1000;    } else {      tv.tv_sec  = 0;      tv.tv_usec = 0;    } // if    if (::select(s+1,&socketReadSet,0,0,&tv) == SOCKET_ERROR) {      error = true;      return false;    } // if    error = false;    return FD_ISSET(s,&socketReadSet) != 0;  } // isReadable  bool isWriteable(bool & error,int timeOut=0) { // milliseconds    fd_set socketWriteSet;    FD_ZERO(&socketWriteSet);    FD_SET(s,&socketWriteSet);    timeval tv;    if (timeOut) {      tv.tv_sec  = timeOut / 1000;      tv.tv_usec = (timeOut % 1000) * 1000;    } else {      tv.tv_sec  = 0;      tv.tv_usec = 0;    } // if    if (::select(s+1,0,&socketWriteSet,0,&tv) == SOCKET_ERROR) {      error = true;      return false;    } // if    error = false;    return FD_ISSET(s,&socketWriteSet) != 0;  } // isWriteable
hplus0603
hplus0603
Quote:

        bytesRead = ::recv(s,data,buffLength,0);



This line will block until you have read "buffLength" amount of data, or the connection on the other end closes, unless your socket is set to be non-blocking. A malicious user could connect to your server with telnet and send a single character, then nothing else, which would make you wait forever. Repeat N times in parallel, and your server will run out of sockets or threads, if you're threaded, or just lock out your server entirely, if you're single-threaded. Try it yourself!

There is an ioctl() you can call to get the amount of data ready to be read; you can use that if you want to ensure you won't block in reading. However, it's easier to just set the socket as non-blocking, and it works fine (I see no need to toggle the socket back to non-blocking).

One thing I've noticed in a lot of code is calling WSAStartup() with an old WinSock API version. There were certainly bugs and flakiness in older versions of WinSock, that the API will attempt to reproduce when you ask for an older version. I recommend requesting version 2.2, which will work fine on any modern version of Windows, or any non-modern version that has a modern Internet Explorer installed. I don't know if this caused the issue you saw, but it might be worth looking into.

Last, calling select() for a single socket is a waste of CPU resources. Just make the socket non-blocking, or use the ioctl() to read the amount of data waiting (depending on your platform). select() is intended to be used where you want to service more than one socket at a time, to let you pick the sockets that need attention (i e, have data waiting).
enum Bool { True, False, FileNotFound };
John Schultz
John Schultz
Quote:
Original post by hplus0603
Quote:

        bytesRead = ::recv(s,data,buffLength,0);



This line will block until you have read "buffLength" amount of data, or the connection on the other end closes, unless your socket is set to be non-blocking. A malicious user could connect to your server with telnet and send a single character, then nothing else, which would make you wait forever. Repeat N times in parallel, and your server will run out of sockets or threads, if you're threaded, or just lock out your server entirely, if you're single-threaded. Try it yourself!

There is an ioctl() you can call to get the amount of data ready to be read; you can use that if you want to ensure you won't block in reading. However, it's easier to just set the socket as non-blocking, and it works fine (I see no need to toggle the socket back to non-blocking).

One thing I've noticed in a lot of code is calling WSAStartup() with an old WinSock API version. There were certainly bugs and flakiness in older versions of WinSock, that the API will attempt to reproduce when you ask for an older version. I recommend requesting version 2.2, which will work fine on any modern version of Windows, or any non-modern version that has a modern Internet Explorer installed. I don't know if this caused the issue you saw, but it might be worth looking into.

Last, calling select() for a single socket is a waste of CPU resources. Just make the socket non-blocking, or use the ioctl() to read the amount of data waiting (depending on your platform). select() is intended to be used where you want to service more than one socket at a time, to let you pick the sockets that need attention (i e, have data waiting).


From MSDN Winsock Docs:
Quote:
For connection-oriented sockets (type SOCK_STREAM for example), calling recv will return as much data as is currently available—up to the size of the buffer specified. If the socket has been configured for in-line reception of OOB data (socket option SO_OOBINLINE) and OOB data is yet unread, only OOB data will be returned. The application can use the ioctlsocket or WSAIoctl SIOCATMARK command to determine whether any more OOB data remains to be read.


When select() indicates data available for a TCP socket, recv() reads all the data that's available on the socket, up to the buffer length. If less data is available than the buffer size, the amount currently available is returned (without blocking), even if only one byte was present.

Are you saying that WS2 recv() is broken for TCP? I have not yet see this behavior (to be fair, I currently use TCP only to perform HTTP/GET. All game code uses a specialized high-performance reliable UDP system). The MSG_PEEK flag could also be used in the case of a TCP/IP stack bug (provided that MSG_PEEK works correctly!).

read() (on *nix), on the other hand, should block in this case.

[Note that the example code on the above referenced MSDN link has a bug. The return value from recv() should be checked against SOCKET_ERROR (not WSAECONNRESET), and if true, WSAGetLastError() should be called to determine the actual error].

WRT WSAStartup() I recommend including winsock2.h and using:
  WSADATA wsaData;  ::memset(&wsaData,0,sizeof(wsaData));  int res = ::WSAStartup(WINSOCK_VERSION,&wsaData);  // WINSOCK_VERSION is currently 2.2.


Another issue I had with non-blocking sockets and TCP was a proper (error checked) shutdown sequence:

  bool shutdown(void) { // Call only on TCP sockets before close. Socket should be set to blocking before this call.    if (type != ST_TCP) return false;    int res = ::shutdown(s,SD_SEND);    if (res == SOCKET_ERROR) return false;    bool error;    while (isReadable(error)) {      int buff,bytesRead;      if (receive(&buff,sizeof(buff),bytesRead)) {        if (bytesRead == 0) {          return true; // Successful shutdown.        } // if      } else {        return getLastError() == WSAECONNRESET; // Also OK to get a Reset.      }// if    } // while    return true;  } // shutdown  bool receive(void * buff,int maxLength,int & bytesRead) { // For UDP & TCP reads    bytesRead = ::recv(s,(char *)buff,maxLength,0);    if (bytesRead == SOCKET_ERROR) {      bytesRead = 0;      return false;    } // if    return true;  } // receive  int getLastError(void) {    return ::WSAGetLastError();  } // getLastError


What is the proper (error checked) method to shutdown() a non-blocking TCP socket (instead of just closing the socket)? (I probably derived my shutdown code from Steven's Unix Network Programming book).

WRT using select() compared to non-blocking and generating an error: WSAGetLastError() must still be called for each "non-blocked" read attempt, thus requiring another library call (*nix reads the global variable errno, faster). If it were benchmarked that select() was (significantly) slower than WSAGetLastError(), for example on a large MMP server, that would make a good case to use the non-blocked I/O method. However, on Win32/Winsock2, I/O completion ports with overlapped I/O may be faster than either method. I would certainly rewrite my code to use the fastest method (provided it was stable on all platforms and the performance difference was significant).

I/O Completion Ports
Overlapped I/O
Quote:
The transports allow an application to invoke send and receive operations from within the context of the socket I/O completion routine and guarantee that, for a given socket, I/O completion routines will not be nested. This permits time-sensitive data transmissions to occur entirely within a preemptive context.


Have you used these Win32 features? Benchmarked any improvements? I could see how a saturated server, spinning on a single socket could be the fastest method (constantly reading data from the queue). However, behind-the-scenes buffer management could allow the Win32 IOCP/OIO methods to be faster (or *nix Async I/O for Posix (aio_*)). This is the kind of benchmark that could show if there is any difference between using select() and non-blocking I/O (with error checking).
hplus0603
hplus0603
Quote:
Are you saying that WS2 recv() is broken for TCP?


No, actually, I was thinking of the read() behavior. My only defense is that they both start with "r" :-)


Regarding an error-checked shutdown behavior, I'm not sure there's any guarantee that you can get that the connection is actually shut down in all cases. For example, think of a protocol translating NAT firewall, or the case where the other end FINs first and your FIN (or his ACK) is dropped.

But it doesn't matter, because the other end will attempt to re-ACK to get some more data, and your node (or the firewall) will just respond with a RST and the connection goes away anyway, or you'll time out after two maximum segment lifetimes (IIRC). In general, the kernel will handle shutting down the connection just fine for you by just closing the socket, or by ending the process (which implicitly closes the socket), so shutdown() is really only necessary if you want to explicitly signal that you've stopped sending data, but want to keep on receiving data.
enum Bool { True, False, FileNotFound };
John Schultz
John Schultz
WRT shutdown(), yeah, just closing the socket is probably fine in most game cases. Closing a socket is also a clean way to wake a thread (e.g. to exit) which is blocked on a socket operation (e.g. a threaded TCP design).

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.