🔒 Locked

Overlapped File IO Interfering with Winsock

Started by CadetUmfer Jul 26, 2011 at 4:13 PM 3 replies 2.7k views

Original Post

Jul 26, 2011 at 4:13 PM

I'm using enet for networking. It uses normal, non-blocking, non-overlapped Winsock 1.1 sockets. For some reason, my file reading code is interfering with it. Using a normal, synchronous fread, everything is fine. But when I enable my Overlapped IO file reading (which works fine when not using enet), I receive an access violation inside enet when it calls WSASendTo (this is the first socket operation performed after creating the socket/setting options).

All I'm doing in my file io code is calling ReadFileEx with the overlapped struct set, and no completion routine. I then poll all of the operations in progress every frame using HasOverlappedIoCompleted.

Does anyone have any idea why this might be happening? I need async file operations (I guess I could dump them on the thread pool but, eh) and I don't want to write my own socket library (at least not yet)

The only lead I have at this point is that I call sleep each frame in the main thread. If I change this to SleepEx (which should cause the thread to be woken up when an overlapped IO operation completes), I receive an access violation on the SleepEx call.

Washu

Jul 26, 2011 at 4:17 PM

Sounds to me like enet is using IOCP as well, since that's the general reason to use WSASendTo rather than just good old sendto. Since enet is calling, most likely, GetOverlappedResult or GetQueuedCompletitionStatus it will also be trapping your completion status for your overlapped calls.

Post some code, if you will, since they shouldn't be interfering with each other, since you should have a separate IOCP queue though. You did call CreateIoCompletionPort, right?

CadetUmfer

Jul 26, 2011 at 4:34 PM

That's what I was thinking, but looking through the code, it's definitely not. It's setting the socket to non-blocking (FIONBIO), but the send receive code doesn't use overlapped IO (the overlapped struct is set to NULL).

EDIT: I'm not calling CreateIoCompletionPort. I didn't think you needed to when using ReadFileEx. I'm just passing FILE_FLAG_OVERLAPPED to CreateFile, and passing the LPOVERLAPPED to ReadFileEx. Looking into this now..msdn says "After an instance of an open handle is associated with an I/O completion port, it cannot be used in the ReadFileEx or WriteFileEx function because these functions have their own asynchronous I/O mechanisms" which led me to believe I didn't need to call CreateIoCompletionPort.

EDIT2: Here is my file reading code:

typedef struct readop_f {
 void *user;
 HANDLE file;
 OVERLAPPED overlapped;
 byte *buffer;
 int size;
 readhandler_t callback;
 
 struct readop_f *prev, *next;
 } readop_t;
 
 unsigned sys_read(const char *name, readhandler_t callback, void *user) {
 HANDLE file;
 DWORD size;
 byte *b;
 readop_t *o;
 
 file = CreateFile(name, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_FLAG_SEQUENTIAL_SCAN | FILE_FLAG_OVERLAPPED, NULL);
 if (file == INVALID_HANDLE_VALUE) {
 LS_ERROR(va("CreateFile failed for %s", name));
 return 0;
 }
 
 size = GetFileSize(file, NULL);
 if (size == INVALID_FILE_SIZE) {
 LS_ERROR("GetFileSize failed");
 return 0;
 }
 
 b = (byte *)sys_malloc(size + sizeof(readop_t));
 o = (readop_t *)(b + size);
 o->user = user;
 o->file = file;
 o->buffer = b;
 o->size = size;
 o->callback = callback;
 o->overlapped.Internal = o->overlapped.InternalHigh = 0;
 o->overlapped.Pointer = NULL;
 o->overlapped.Offset = o->overlapped.OffsetHigh = 0;
 o->overlapped.hEvent = NULL;
 
 if (!ReadFileEx(o->file, o->buffer, o->size, &o->overlapped, NULL)) {
 LS_ERROR("ReadFileEx failed");
 sys_free(b);
 return 0;
 }
 
 DL_PREPEND(readops, o);
 
 return size;
 }
 
 void sys_tick(void) {
 readop_t *o, *next;
 
 for (o = readops; o; o = next) {
 next = o->next;
 if (HasOverlappedIoCompleted(&o->overlapped)) {
 DL_DELETE(readops, o);
 CloseHandle(o->file);
 o->callback(o->buffer, o->size, o->user);
 sys_free(o->buffer);
 }
 }
 }

Here's some relavent portiosn of enet:

ENetSocket
 enet_socket_create (ENetSocketType type)
 {
 return socket (PF_INET, type == ENET_SOCKET_TYPE_DATAGRAM ? SOCK_DGRAM : SOCK_STREAM, 0);
 }
 
 int
 enet_socket_send (ENetSocket socket,
 const ENetAddress * address,
 const ENetBuffer * buffers,
 size_t bufferCount)
 {
 struct sockaddr_in sin;
 DWORD sentLength;
 
 if (address != NULL)
 {
 memset (& sin, 0, sizeof (struct sockaddr_in));
 
 sin.sin_family = AF_INET;
 sin.sin_port = ENET_HOST_TO_NET_16 (address -> port);
 sin.sin_addr.s_addr = address -> host;
 }
 
 if (WSASendTo (socket, 
 (LPWSABUF) buffers,
 (DWORD) bufferCount,
 & sentLength,
 0,
 address != NULL ? (struct sockaddr *) & sin : 0,
 address != NULL ? sizeof (struct sockaddr_in) : 0,
 NULL,
 NULL) == SOCKET_ERROR)
 {
 if (WSAGetLastError () == WSAEWOULDBLOCK)
 return 0;
 
 return -1;
 }
 
 return (int) sentLength;
 }
 
 int
 enet_socket_receive (ENetSocket socket,
 ENetAddress * address,
 ENetBuffer * buffers,
 size_t bufferCount)
 {
 INT sinLength = sizeof (struct sockaddr_in);
 DWORD flags = 0,
 recvLength;
 struct sockaddr_in sin;
 
 if (WSARecvFrom (socket,
 (LPWSABUF) buffers,
 (DWORD) bufferCount,
 & recvLength,
 & flags,
 address != NULL ? (struct sockaddr *) & sin : NULL,
 address != NULL ? & sinLength : NULL,
 NULL,
 NULL) == SOCKET_ERROR)
 {
 switch (WSAGetLastError ())
 {
 case WSAEWOULDBLOCK:
 case WSAECONNRESET:
 return 0;
 }
 
 return -1;
 }
 
 if (flags & MSG_PARTIAL)
 return -1;
 
 if (address != NULL)
 {
 address -> host = (enet_uint32) sin.sin_addr.s_addr;
 address -> port = ENET_NET_TO_HOST_16 (sin.sin_port);
 }
 
 return (int) recvLength;
 }

adeyblue

Jul 26, 2011 at 10:11 PM

All I'm doing in my file io code is calling ReadFileEx with the overlapped struct set, and no completion routine.

That's the problem. Despite what the docs say, the completion routine isn't optional. If you set it to NULL, Windows will try and call it which will go bang. My comment about this has been at the bottom of the functions' MSDN page for about two years now. Guess they don't want to, or are unwilling to fix either the docs or the code.

Topic Locked