Async sockets or select() with threads?

If large numbers of connected clients is your goal, then the best solution by far is I/O Completion Ports. They can be a bit hard to get your head wrapped around, but once you get the basic framework up, they're actually very simple to code against. Certainly less code than having multiple threads listening with a select, and the state manegement is much simpler (in my opinion) than asynchronous sockets.

Hmm... I've played about with IO completion ports before (I wrote the previous version of my MUD using them, reasonable well), and there's something I don't like about them. Probably because I'm still trying to get to grips with them ;)
I seem to recall that async sockets could deal with up to 1000 connections, although that'd obviously be pushing things. Ideally I'd like to stay away from IOCP to be honest. I was thinking of 500 connections as an absolute maximum (after that I start refusing new connections with an error message). I'll be very surprised if the average load rises above 64 connections (1 thread).

However, I'd still like it to be as efficient as possible, but staying away from IOCPs.
If it matters (since I'll be running several threads), the server machine only has 1 CPU.

Some benchmarks of async sockets:
http://bulk.fefe.de/scalable-networking.pdf

Thanks, that's a very entertaining read :) Unfortunately, it only covers non-blocking IO with select(), not Async IO (The WSAAsync*() set of functions).

well, I'd probably say the management of the threads for the select solution is probably going to be more trouble than it's worth. For example, say you have 64 clients, then one more connects you start up another thread to handle it. Then one of the original 64 leaves. You've now got 64 connections again, but two threads. Do you shuffle things around so it's all on one thread again? If you have lots of joining/leaving there could be a lot of thrashing. You could do so that there's a bit of buffer space, and start a new thread when the "capacity" of your threads reach 80% or so, thus you can add/remove connections without creating lots of new threads. But like I said, it's probably more trouble than it's worth.

I/O completion ports work well because they minimize the number of context switches. And the OS is able to manage the number of threads in the pool.

Still, I think 500 connections will probably not overload a select solution too much. Plus, I think there may be a way to increase the 64 connections limit to something more reasonable (though non-portable, but if that doesn't matter...) I can't remember how exactly, but I think it was just a constant in a header somewhere..

Quote:

Original post by Evil Steve Thanks, that's a very entertaining read :) Unfortunately, it only covers non-blocking IO with select(), not Async IO (The WSAAsync*() set of functions).

Well, I'd probably stay away from WSAAsync*. They're mostly good for GUI apps that want to do socket I/O on the same thread as the UI. It's really only meant for one or two connections (i.e. a client app).

Quote:

Original post by Dean Harding Still, I think 500 connections will probably not overload a select solution too much. Plus, I think there may be a way to increase the 64 connections limit to something more reasonable (though non-portable, but if that doesn't matter...) I can't remember how exactly, but I think it was just a constant in a header somewhere..

Yeah, you can #define something before including the winsock header to increase the limit (up to a certain point). But that's also ugly ;)
If I was doing threads, I'd probably have some kind of loose thread pool, and try to keep as few threads as possible, but still keep the sockets balanced between the threads.
Hmm... You may be right about it not being worth it.

Quote:

Original post by Dean Harding Well, I'd probably stay away from WSAAsync*. They're mostly good for GUI apps that want to do socket I/O on the same thread as the UI. It's really only meant for one or two connections (i.e. a client app).

Really? I thought that WSAAsync was supposed to scale better than select()? Or it was better in some way. Perhaps I was thinking of something else.

I just remembered a chapter in Game Programming Gems 3 or 4 that had something about a very rough number of clients that each method could handle. I'll go and have a read at that and post what it says.

Thanks for all the replies,
Steve

Here's the paragraph (Game Programming Gems 4, Chapter 6.2):

Quote:

Clients: 500-1,000 Asynchronous and nonblocking I/O systems work well for 500-1,000 clients, but one-thread-per-client is unlikely to handle this many connections. Nor is it possible at this scale to have just one thread serving all clients, even with asynchronous I/O. Typically, servers for this model have 5-10 threads handling 100 clients each.

I think I might end up using multiple threads after all... :/

EDIT: Oh yeah, the #define is FD_SETSIZE, and it defaults to 64. I might bump that up to 128 if I go for the threaded approach I think. After all, if I do decide to go for linux (which I doubt), I'll have up to 1024 connections per thread (If I'm lucky).

Quote:

Nor is it possible at this scale to have just one thread serving all clients

I'd like to take exception to that particular statement. You can serve all the clients in a single thread just fine, given the proper software architecture.

Can I just get something clarified: Does async I/O mean using the WSAAsync* family of functions? And non-blocking I/O is polling with select()?

If that's correct, then is async I/O actually suited to handling up too 500-1000 clients, or is it geared more towards a few connections in a GUI app, as Dean said?

Non-blocking I/O means that you can recv() on a socket, and it'll immediately return "no data there" if there is no data. Using select() is optional with non-blocking I/O, although it's often a good idea.

Async I/O, on Windows, for sockets, means typically using the WSAAsync functions -- although any "overlapped" I/O operation would qualify as "async I/O".

I wouldn't worry TOO much about which method to use. Design a physical abstraction around network I/O, and implement whichever way feels most natural to you. If you have performanc eproblems later, you can go back and re-implent it in a better way without too much pain, assuming your abstraction actually serves its purpose.

Probably the one thread can't scale comment is referring to the fact that if your server only has one thread then it won't effectively utilize additional processors. The rule of thumb is one thread per cpu. How you get there can vary though. You could have 2-proc machine running two instances of a single-threaded server and get good scaling.

Quote:

if your server only has one thread then it won't effectively utilize additional processors

I would like to forward the argument that memory busses are more a limitation than CPU throughput. If you put two CPUs on a single memory bus, you'll just end up contending for memory, and may actually run slower than you would on a single CPU. If you are memory latency or throughput constrained, then multiple CPUs won't help you at all.

Ergo, it's often most cost-effective, AND highest performance, to design for single-CPU single-threaded nodes. If you want to scale, add more nodes.

Quote:

Original post by hplus0603 I wouldn't worry TOO much about which method to use. Design a physical abstraction around network I/O, and implement whichever way feels most natural to you. If you have performanc eproblems later, you can go back and re-implent it in a better way without too much pain, assuming your abstraction actually serves its purpose.

Yeah, that's true. I think I'll go for a multithreaded blocking socket method, and do a bit of profiling. I don't really expect the network IO to be a bottleneck until I have a lot of clients connecting anyway.

Quote:

Original post by Anon Mike Probably the one thread can't scale comment is referring to the fact that if your server only has one thread then it won't effectively utilize additional processors. The rule of thumb is one thread per cpu. How you get there can vary though. You could have 2-proc machine running two instances of a single-threaded server and get good scaling.

On a slightly-related note, the docs for IOCP (I can't be bothered to find the link just now) suggest creating 2 threads per CPU, I assume that's because the IO threads won't be doing anything for most of the time though, so the won't be much of a performace loss.

Thanks for the replies. I know that having a steady 500 connections really isn't going to happen, but even if I have 50 people on at once, I think that one thread per client may even then be a bit much.
Although, there's very little additional code and upkeep involved in extending that to x sockets per thread, so long as I don't get carried away (100+ sockets per thread would end up getting a bit silly).

About IOCP - I found it really weird to get to grips with, and I should probably go back and re-write my code, now I have a better understanding of how IOCPs work, since there's quite a few issues with my code. Perhaps I should have looked for a tutorial instead of reading the MSDN :P

Also, I really found that gem informative, thanks a lot for writing it :)

I advise just the opposite :). Thread-per-client is the worst way you can do things. Just wait until your server is acting weird and you have to wade through 500 threads to figure out which one is causing the problem. If you start a thought with "I'll just use a thread per-" and you end that thought with anything other than "cpu" then you've got a flawed design. Even if Linux manages the thread context switches better than NT then you've still got the fact that all those thread stacks are just chewing up memory that could be put to better use.

I find IOCP to be quite elegant. It neatly solves many of the scalability and thread-management problems for you. My understanding is that there is work to put something similiar to IOCP in the Linux kernel also but I don't really know.

My advise is to just stick with select for now. When you come to better grips with IOCP, and when scaling actually starts to be a real problem instead of a theoretical one, then switch over.

I've written a few muds for school projects and stuff and have used diffrent methods for network io. most muds that i have played rearly have 20 people logged on at once and most of the code bases that i've looked at stop accepting connections at 50. but most muds are written to run on a pentium 1 class processor and they still have latency with 40 players connected


the thing that is going to affect preformance the most is going to be how you design your mud to work with the connections and how you abstract your data whether it is io or player structures ect...

some of my test with my previous muds when profileing it at 30 fake connections only about 1% of my cycle was taken up by network stuff