IOCP multi WSARecvs synchronization

Quote:

Original post by Escarab In order to unleash all the power of iocp, I should post more than one WSARecv on my iocp, so while one thread is processing a input, another one can gather more input from the same client, is that right.

No. You issue one request per socket. So despite having multiple threads, each individual connection behaves as single-threaded.

Quote:

If a player has a lot of bandwidth, he may be able to slow down the server by locking my workers with his inputs?

Of course. If recv handler performs calculatePiToMillionDigits command, and if enough clients request it, something will break.

Again, this is where the request at a time comes in.

You issue read request for a certain socket. When it's completed, you process it.
When multiple clients connect, you will have one request per socket. IOCP dispatcher will dequeue an event, get a free worker from thread pool, and execute it in that thread. But it can never have more than one thread processing the same socket. It's possible to do that obviously, but there's no real practical need for that.

In example as above, where one request takes very long, client's requests after that will simply not be processed.

Typically, the number of active workers will closely match the number of cores.

But that's not really networking problem, it's more how you deal with load distribution within your application.

IOCP solves the problem of "if I'm using TCP, how can I sumultaneously serve 1,000 sockets with some fairness and performance?"

If you're using UDP, you'll be using a single socket, and thus IOCP will be pretty useless for you. Instead, you'll typically have a single high priority thread that sits in blocking recvfrom(), and puts received datagrams into a queue, then you'll have one normal priority thread per core that block on this queue, waiting for work items. This is assuming that your world update can be threaded, and doesn't require excessive or global locking. If that's the case, you might as well just be single threaded.

I'm using TCP. The main question is: Posting more than one WSARecv in a same SOCKET is useful, is faster, is better, worth the synchronisation pain?

To answer your four questions:
No.
No.
No.
No.
Remember: The kernel already buffers TCP/IP data when it comes in. It has to do that, to ensure ordering. Your I/O completion proc will only receive the data once the kernel has received it and made sure it's in order.

Quote:

Took from here :Who Manages the Buffers? ... "Turning off receive buffering is not usually necessary, as long as the application takes care to always have a few overlapped WSARecvs outstanding on a connection. The availability of posted application buffers removes the need for AFD to buffer incoming data."

I've read something about that thing of no need to copying from the internal buffer to your buffer if you always keep at least one WSARecv waiting at any given time on the completion port.

Isn't this capability one of the most powerful things about iocp, avoiding buffers copy?

The point of the above is something else. If there is a pending receive, data is received directly into target's buffer. If there isn't one, it's cached internally.

The question about feasibility of this is - does the cost of one extra memcpy outweight the cost of manual thread-safe TCP stream reconstruction. I would speculate that even without contention, there aren't any gains. And still - this is exactly what the IOCP already does, so re-implementing such system one level above doesn't solve anything - memcpy will still be necessary to pass data to application.

When combined with the ability of scatter-gather operations (to always receive up to capacity of ring-buffer, wrapping as needed) and relatively fast processing in recv handler, a fair number of writes should be zero-copy already.

Main problem here is that TCP streams are sequential, and must be processed sequentially. There simply is no way of doing that concurrently (unless the we send single bytes in arbitrary order).

Also, as a matter of perspective - performing IOCP receive with a memcpy handler (when recv() completes, copy data into local buffer and discard it) puts 0% (task manager shows 0-2%) load on mid-range processor if 100Mb network bandwidth is capped. In other words - the cost of memcpy for TCP recv() on 100Mbit network is not measurable. This is in part related to memcpy being incredibly memory and cache friendly operation.