C Socket question!

A thread per socket is probably best if you have 10 or less connections. if high performance is not an issue you can probably get away with a hundred or so connections this way.

However, if you want to do it the right way you need to use asynchronous sockets, IOCP and thread pools (assuming this is Windows). This will let you handle tens of thousands of connections efficiently.

I will say you need to be an experienced programmer with a good understanding of concurrent systems to implement the IOCP based server. If you’re just starting out a thread per socket is going to be much easier to deal with. If you have no experience with threads and scalability is not an issue then go with blocking sockets and use select.

Thank you! I will go with select :)

I am using Berkley sockets, under Linux! Will select be good with lots of connections?

If you are just moving data from one socket to another, than single threaded select/poll/epoll is your best bet. The kernel can move more data than your physical connection can handle, so no need to complicate things with threads.

If you are doing processing, as in clients make a request that needs to be serviced by the server, than it is a different ball game. For a low number of clients you can stay single threaded, but there is a threshold where you will hit 100% utilization and begin to see a slow down. At that point you need to start thinking about worker queues and their ilk.

Single thread per connection is easy, as you can sit and wait on a blocked send/recv, but will not scale very well. It gets worse if you need to communicate amongst connected clients as now you have synchronization issues.

epoll is my favorite api for many connection applications. poll is also very good. poll vs epoll may just come down to how your code is structured. If you have any kind of data attached to a socket, then epoll is nice as each event has a void * attached to it.

select is implemented as poll on many distributions. select has a fd limit of 2048. select is good for simple things, but will not scale well.

Don't forget that on linux you are limited by default to 1024 file descriptors, so you will need to change that if you need to accept more connections. The system hard limit will be much higher.

--Z

Quote:

Original post by brunooo Now I would like to know, what should I use? What I should use for few connections (< 10)? What I should use for a lot of connections (> 100)?

Just how many people do you expect to be playing the same game? Isn't 4 about as high as it gets in practice?

4 is just fine, I am just wondering about how mmo games works :)

I implemented it with select, and it was not difficult, if I use threads I would need to learn them first :p

So basically, on linux I cant have more than 1024 fds? But I can change that? The very limit of fds vary from pc to pc? How can I find the limit and change the '1024' limit? (Again, I wont do any mmo, I am just curious ^-^)

Thank you all for the answers!

To find the current hard limit on number of descriptors for the entire system, enter the following into your console:

cat /proc/sys/fs/file-max

This value can be changed.

To see how many fd's a single process can have, enter the following:

ulimit -n

The default is 1024. You can change this up to the hard limit by ulimit -n #. ( Note: Only root can change the soft limit ) You can also look at getrlimit and setrlimit. Specifically the RLIMIT_NOFILE param. You will need to be root to make this change. ( a good daemon would set it's limits, then drop root privileges for the remainder of it's execution life time ).

select() has a hard limit of 2048. poll()/epoll() do not.

--Z

Thank you! Whats the difference of select and pool(e)? Which should I use? But, if somehow I reach this limit I could separate on another set of fds?

The problem with select() is that you have to do a bunch of busy-work for file descriptors, even if they don't have any traffic. You have to put them into one big bitfield, then the kernel has to unpack that bit field, then again you have to test every bit in the bit field to see which file descriptors were actually ready.

With the various poll/queue based approaches, you instead tell the kernel once that you'd like to know about a certain file descriptor, and when/if something actually happens, the kernel lets you know. Very similar in idea to I/O completion ports, really.

Humm, I see.. I will change from select to epoll :p

Now changing a little the focus, I am trying to find a way to test my server (with cpptest), but its hard to test only a part. I need to test the server as a big block (I dont have much experience on testing). I was thinking in creating a client with other thread on server, and there I will do the tests, what do you think?

On many systems select() is implemented internally as poll anyways. epoll() is my favorite wait mechanism for high traffic daemons for 2 reasons: 1) You add the fd once, no building of lists each cycle, and 2) part of the structure used contains a void * param making it very quick to find the object ( class / struct / etc. ) that you need to operate on.

For server testing, I create a tool that will do a specific task related to what I am testing. Then I can run this tool with the server in gdb to test. I also run the daemon used by the other developers under gdb so when they cause it to crash, I have a backtrace.

--Z

What kind of tools do you use to test your server? Which ones? :o

Quote:

On many systems select() is implemented internally as poll anyways.

Which makes select() even less efficient, because it needs to build and tear down the list of file descriptors for each call to the kernel. For large numbers of file descriptors (dense select() bit arrays) this gets really expensive.

Quote:

testing

There are two layers of testing (at least): Unit testing, and functional testing.

With unit testing, you write code inside your .cpp files. That code runs when in test mode, and calls an individual function of an individual object instance, to verify that given known inputs, the function returns a known output. This is done one function at a time, with as much insulation as possible from the rest of the system. If the function called depends on a file system, a network, a timer etc, then those services are generally mocked (replaced with narrow shims that just return known dummy data).

With functional testing, you generally build some mode into your client such that it can be scripted from the command line, or load and run a script when invoked. That script performs user-level operations (connect to server at address X, send login Y/Z, issue request W, ...). It then verifies that the right thing happens. Generally, you want the ability for a server to run in a "sandbox" where it is self-contained, and you can set up a known good initial state. If you generally use MySQL for database operations, the sandbox might use a different database name, or may even use SQLite instead, for example. For persistent files, the sandbox may put them under /tmp/sandbox instead of whatever the persistent location usually is.

Both kinds of testing are important. Personally, I find it hard to break down certain systems all the way down to unit test level, and some of my unit tests will have a functional flavor -- like, to test the sockets implementation, I may create two sockets, listen on one, connect on the other, and send "hello, world!" and then verify that I can receive that. That exercises more than just a single function on the sockets class, but it can still be run at unit test time because it doesn't need any other information or support (outside of the underlying BSD sockets implementation, natch :-)

Similarly, at other places where your classes interact with the kernel, you may need to make "functional" style unit tests. For example, the function that tests your wrapper for epoll() will likely need to use multiple objects. A simple test for that might look something like:

create three sockets; one listening, two idle
run the epoll with a timeout
verify that you get nothing back
connect one of the idle sockets
run the epoll with a timeout, until timeout expires (fail) or you get "writable" from the connecting, and "readable" from the listening/accepting socket
create a fourth socket based on the "accepting" socket incoming connection
run the epoll with a timeout
verify that you do not get "readable" from any of the sockets
...

Thank you for reply!

So I guess I will need to do more functional test than unit tests. I am using CppTest, I guess it was made to do unit tests, but I will use it for both :)

Here is what I will try:

Create a socket in the server representing the client and do tests like you said, send "Bla" and see if you receive "Bla", all looking like unit tests :p

Is this a good idea?

Google talk on

">Clean Code Talks has an overview of designing code that lends itself well to testing (I'm not sure that is the right video, I only searched by title).

I personally do not fully agree with the topic of constructors being forbidden of doing anything at all (especially in C++ it prevents RAII), but emphasis seems to be on strict Dependency Injection frameworks which invoke indeterminate number of constructors which need to be as light-weight as possible.

All of the rest, especially de-coupling allocation from use is a sound advice, especially in C++.

Because the OS doesn't support dependency injection (without really tricky and expensive work), you will end up having to use functional testing for the OS interface code. There's really no good way around that.
Anything that sits on top of your OS abstraction, though, should be unit testable at the single-function level, if you design it with that as a goal.