Hi everyone,

I'm building my mmorpg from scratch, it's going pretty well I would say. ( I will surely open a development blog soon)

My network protocol is built on top of tcp, and a couple of days ago I implemented a way for me to spam other players in the world, that open their own connection to the server. (for now I'm running both client and server on localhost).

Everything is fine, (indeed I'm surprised to see 300 players easily handled in these conditions) except for a sublte bug that I'm having from yesterday.

My world is subdivided in islands, and when a player changes island, the server sends the player information about the new island, so that the client can open a new connection to the other server. (Yes I'm running 2 server on my pc)

When the numbers of players increase, sometimes the client don't receive that "critical" message, and so the connection to the other island never happen, even if the server has sended it. (or at least the call to send() returned the right number of bytes).

So my question is: could it be the case that even if the call to send() effectively returns the right number of bytes, in reality the clients dont receive them because their buffer is too full? that would be explained by the fact that all is running locally, so I guess both client and server are sharing the same tcp receiving buffers at their base.

Shound I implement an "acknowledge" for those critical packet? so that until the server don't receive the acknowledge it continues to send the critical packet again and again?

That would sound a bit odd, because that's the principle around tcp right? but I'm pretty sure the bug is not something in the logic of the game, is just that the server call to send return a "fake" value.

Maybe in real world scenario this will never happen?

Thank you all, Leonardo.

"No indication of failure to deliver is implicit in a send()."

Locally detected errors are indicated by a return value of -1.

that's on the man page, so I guess I was right... the client isnt really receiving the bytes.

BUT, at this point I'm asking, considering this fact, How can my server be sure that the packet has been sended? the only way is to wait for an acknowledge right? but now I'm implementing a reliable protocol by myself, which is exactly the reason I used tcp over udp...

How can my server be sure that the packet has been sended? the only way is to wait for an acknowledge right? but now I'm implementing a reliable protocol by myself, which is exactly the reason I used tcp over udp

If you need exactly the guarantees that TCP gives -- every byte is received, in the order it was enqueued -- then using TCP is great!
Note that TCP doesn't GUARANTEE that the data gets there -- it gets there, or you (eventually) get a timeout error. I mean, if the cable is cut or the computer turned off, no data will actually make it there.

Most game traffic can work better when the guarantees are different, though. For example, why re-transmit old object state, when you have newer object state available? And if each object update is un-ordered compared to each other, why delay the update for object B just because the update for object A was somehow lost?
It is in these cases that you CAN do better on UDP than TCP. But if that's not important to you, TCP can be just fine.

"could it be the case that even if the call to send() effectively returns the right number of bytes, in reality the clients dont receive them because their buffer is too full?" - Yes. Confirmation of sending is not proof of receipt. When it says "Locally detected errors are indicated by a return value of -1" that 'locally' doesn't mean "on my machine", it means "in the sending process".

"that would be explained by the fact that all is running locally, so I guess both client and server are sharing the same tcp receiving buffers at their base." - No, this isn't necessarily the reason, nor likely to be true anyway. One possibility is that you aren't reading the data on the client as quickly as you need to, in order to keep the buffer empty. But that would normally mean the connection gets dropped soon after.

"Should I implement an "acknowledge" for those critical packet?" - No, but you must be checking for errors everywhere in your networking code. It sounds more likely that there is some other problem that means this message is going missing. Does that client receive subsequent messages? Do subsequent messages to other clients work okay? Are you sure you're reading data quickly enough on the clients?

How can my server be sure that the packet has been sended? the only way is to wait for an acknowledge right? but now I'm implementing a reliable protocol by myself, which is exactly the reason I used tcp over udp

If you need exactly the guarantees that TCP gives -- every byte is received, in the order it was enqueued -- then using TCP is great!
Note that TCP doesn't GUARANTEE that the data gets there -- it gets there, or you (eventually) get a timeout error. I mean, if the cable is cut or the computer turned off, no data will actually make it there.

Most game traffic can work better when the guarantees are different, though. For example, why re-transmit old object state, when you have newer object state available? And if each object update is un-ordered compared to each other, why delay the update for object B just because the update for object A was somehow lost?
It is in these cases that you CAN do better on UDP than TCP. But if that's not important to you, TCP can be just fine.

Yeah, I totally agree with you on the old object-state thing.

Yesterday I implemented acks for the critical packets, and of course it worked.

So I guess that at this point tcp is basically useless to me, the ideal protocol for my game is clearly one that sits on top of udp for the client-server communication.

For my server-server communications, although, I think I will stick with tcp: I know that my servers will be always very near one to the another, so there wont be performance problems or overhead in the tcp ack mechanism. And of course there are more critical packets server-server than client-server, it would be hard to mantain all of them in memory.

It's safe to use udp for client-server communications while using tcp for server-server communications?

Moreover, IF I know that the host of the tcp communication are very near and will always have room to receive the packets, can I assume that if send return me the right number, the packet will be received by the destination?

clients dont receive them because their buffer is too full

That's unlikely. While it is true that clients will not receive data as long as the buffer is full (but then, just read more often!), as soon as the buffer has room again, clients will receive the data from resent datagrams. Unless... the server crashed in the mean time or someone pulled a cable, or several minutes passed with no sign of life from the client whatsoever.

non-blocking send

Here, there is indeed a possibilitiy that clients are not getting the message, that is if it exceeds the send buffer size. Normally this is harmless, the call to send will just block and will copy data in smaller chunks until none is left, then eventually return. But with non-blocking send, you get EWOULDBLOCK and that's it. If you don't check for send being successful, it just goes into oblivion and you never know. Be sure you do check the return code and errno.

the only way is to wait for an acknowledge

Yes, that is the only way of knowing the client has gotten the message, but the approach makes little sense. You already know for certain that unless catastrophic things happen (pull cable, crash, etc) the client will receive the message. You also know that data is delivered in-order, i.e. any message referring to what happens on the other island will be received after the "move to new island" message is received. Both things are guaranteed by TCP. No issue with consistency there.

Thus, resending does not make any sense. If the connection is broken, the resend will not make it through either, but if there is any way of getting the data to the client, it will be received. And, it will be received before anything you send after it. Your manual resend won't, it will be received after whatever you sent last (so it isn't terribly useful for maintaining consistency).

TCP can be tricky if one wants some particular things which are not directly supported by either the protocol or the API. You cannot tell how much the client received (not easily and portably, anyway), nor whether the connection is alive at all, other than by sending and getting a connection reset or destination unreachable error (even if you get no error, that still doesn't mean much since the condition may change the next instant), or by closing the socket and checking the return value. Neither one is particularly useful in practice(especially closing the socket is not if you're intending to continue sending!).

Luckily, almost every time when you think you need such a functionality, you really don't. Just send your stuff and rely that it will work. Your sends might not arrive with the lowest possible latency, but they will arrive, and in-order. Or, well, sometimes they won't, but then you will eventually get an error, and there's not much you could do to remedy the problem anyway. -- close socket, and wait for the client to connect again.

Or, as hplus0603 suggested, consider UDP instead. UDP sounds very scary because it is unreliable. But the truth is, it isn't unreliable at all -- the datagrams arrive just as reliably as any other packet, 99.9% of the time it just works. Only, UDP doesn't strictly guarantee an order, nor does it resend packets in those 0.01% of cases where they get dropped -- but in a realtime application like a game (or think telephony!) this is often not something you would want anyway (who cares about what was 5-10 seconds ago?). But if you do care because one datagram was particularly important, you can still just resend the datagram.

Good spot on the non-blocking send stuff; any failure to check every return value there could mean data is going missing.

"could it be the case that even if the call to send() effectively returns the right number of bytes, in reality the clients dont receive them because their buffer is too full?" - Yes. Confirmation of sending is not proof of receipt. When it says "Locally detected errors are indicated by a return value of -1" that 'locally' doesn't mean "on my machine", it means "in the sending process".

"that would be explained by the fact that all is running locally, so I guess both client and server are sharing the same tcp receiving buffers at their base." - No, this isn't necessarily the reason, nor likely to be true anyway. One possibility is that you aren't reading the data on the client as quickly as you need to, in order to keep the buffer empty. But that would normally mean the connection gets dropped soon after.

"Should I implement an "acknowledge" for those critical packet?" - No, but you must be checking for errors everywhere in your networking code. It sounds more likely that there is some other problem that means this message is going missing. Does that client receive subsequent messages? Do subsequent messages to other clients work okay? Are you sure you're reading data quickly enough on the clients?

Yeah, at one point probably the client is flooded with packet from the server (In some moments the client was receiving updates for maybe 100 players, and so the buffer gets filled out very quickly... in fact the bug never happens until I spawn a big number of players.

I'm not sure that I'm reading data quickly enough when there are 100 players in a small area (probably not), but how can I determine if it so? And if it is so, what can I do?

The simplest thing is to keep reading until you know the socket has no more data to give you. If you end up in an infinite read loop, you're not reading as fast as you're sending. Another common occurrence is that you end up spending something like 99% of your time reading and barely any time doing other processing, until eventually the connection gets dropped as data backs up.

clients dont receive them because their buffer is too full

That's unlikely. While it is true that clients will not receive data as long as the buffer is full (but then, just read more often!), as soon as the buffer has room again, clients will receive the data from resent datagrams. Unless... the server crashed in the mean time or someone pulled a cable, or several minutes passed with no sign of life from the client whatsoever.

non-blocking send

Here, there is indeed a possibilitiy that clients are not getting the message, that is if it exceeds the send buffer size. Normally this is harmless, the call to send will just block and will copy data in smaller chunks until none is left, then eventually return. But with non-blocking send, you get EWOULDBLOCK and that's it. If you don't check for send being successful, it just goes into oblivion and you never know. Be sure you do check the return code and errno.

the only way is to wait for an acknowledge

Yes, that is the only way of knowing the client has gotten the message, but the approach makes little sense. You already know for certain that unless catastrophic things happen (pull cable, crash, etc) the client will receive the message. You also know that data is delivered in-order, i.e. any message referring to what happens on the other island will be received after the "move to new island" message is received. Both things are guaranteed by TCP. No issue with consistency there.

Thus, resending does not make any sense. If the connection is broken, the resend will not make it through either, but if there is any way of getting the data to the client, it will be received. And, it will be received before anything you send after it. Your manual resend won't, it will be received after whatever you sent last (so it isn't terribly useful for maintaining consistency).

TCP can be tricky if one wants some particular things which are not directly supported by either the protocol or the API. You cannot tell how much the client received (not easily and portably, anyway), nor whether the connection is alive at all, other than by sending and getting a connection reset or destination unreachable error (even if you get no error, that still doesn't mean much since the condition may change the next instant), or by closing the socket and checking the return value. Neither one is particularly useful in practice(especially closing the socket is not if you're intending to continue sending!).

Luckily, almost every time when you think you need such a functionality, you really don't. Just send your stuff and rely that it will work. Your sends might not arrive with the lowest possible latency, but they will arrive, and in-order. Or, well, sometimes they won't, but then you will eventually get an error, and there's not much you could do to remedy the problem anyway. -- close socket, and wait for the client to connect again.

Or, as hplus0603 suggested, consider UDP instead. UDP sounds very scary because it is unreliable. But the truth is, it isn't unreliable at all -- the datagrams arrive just as reliably as any other packet, 99.9% of the time it just works. Only, UDP doesn't strictly guarantee an order, nor does it resend packets in those 0.01% of cases where they get dropped -- but in a realtime application like a game (or think telephony!) this is often not something you would want anyway (who cares about what was 5-10 seconds ago?). But if you do care because one datagram was particularly important, you can still just resend the datagram.

very useful reply, thank you.

I'm absolutely considering udp at this point, I'm just waiting for me to have anything more compelling to do on the game that to change a protocol that is working at the moment.

So, to say that one last time, IF send() returns me the right number (the total size of the packet), then the packet WILL arrive at destination (unless catastrophic events of course) and it will arrive before anything other that I send after this send, is that right?

If it's so, then it means that the client is closing the connection at some point, for a reason that I don't know at the moment... because the server of course keeps running and pushing packets all over the place.

Noooo, I know the reason... I spotted the bug in this exact moment...

After successfully sending the critical message, the server calls shutdown on the client socket, and then set a boolean that is "loggingOut"

Then, after having done all of the network stuff, the server checks if for every player the loggingOut variable it's true, and if it is so it calls closesocket.

Of course, when there is a snall number of players, the call to closesocket is NEVER done before the packet is actually delivered, because the receiving buffer isn't full.

But when the receiving buffer is full, even if the call to send succeds, the packet hasn't been delivered, but the variable loggingOut is setted to true anyway... and here we go closesocket is called.

What should I do? should I set a timer? wait for a special message?

Man, that was nasty.

But of course now I've implemented that useless acking thing, that will be exactly what I would do if I was using UDP... I will keep it off hand so that I have it ready when it comes the time to switch to udp.

what do you think about using udp for client-server and tcp for server-server?