Wierd lag / delay when sending multiple packages?!

If you're using TCP, then it's probably nagle kicking in to merge packets for better performance. You'll need to disable that, or switch to UDP.

Quote:

Original post by Evil Steve If you're using TCP, then it's probably nagle kicking in to merge packets for better performance. You'll need to disable that, or switch to UDP.

Yeah Im using TCP.
How do I disable that ?

EDIT:
I have TCP_NODELAY set already, I'm using it like:

Client: socket.setsockopt(IPPROTO_TCP, TCP_NODELAY, 1)

Server: server_socket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
And when a client connects to the server, I do:
new_socket.setsockopt(IPPROTO_TCP, TCP_NODELAY, 1)

Nagle tends to sit and wait for data for around 200 ms in most implementations I've seen, so that's probably not it. Please post the relevant source code (in particular, the client side message receiving/handling code).

Yep, post your code. Also, tell us whether you're sending data back regularly or not. (This can also add to latency if not, but again not to the order of 2 seconds.)

Ok first of all.
Yes, Im sending data between the client and server regulary.

And I've put togheter the most vital in my server and client code, and that's the network part, I removed the graphical and such because I have tried to comment that out to see if that was the problem, and the delay still occurs so..

Here they are:
server.py : http://pastebin.com/m794e634e
client.py : http://pastebin.com/m3797acf6
NetworkHandler.py : http://pastebin.com/m642a8aa7


I don't know if I left ouy too much, tell me if so.
And my NetworkHandler isn't the best one, I know, I should really implement a buffer, that might be the problem after all. But lets give it a try !

Thank you!

EDIT:
I forgot to say.
When there's only one client connected, the sending of packages seems to work just fine, at least I can't notice any delay.

[Edited by - pjuke on November 23, 2009 3:11:48 PM]

I can't see anything obviously wrong, apart from that you're not checking if the writes will block.

You should also decrease your time_accumulator by 1.0/packets_per_sec each time you do the sends instead of reducing it to zero, too. Otherwise you 'lose' the time remainder beyond the trigger threshold.

If nobody else chimes in with something I missed, I'd suggest adding better diagnostics. Write millisecond-precise timestamps to standard output immediately before and after every send, recv, and select call.

You can also use Wireshark to watch the network traffic and guess whether the delay is in pushing the data from app to operating system or the other way around at the receiving end - but I don't think you need to go that low-level for this code.

Quote:

Original post by Kylotan I can't see anything obviously wrong, apart from that you're not checking if the writes will block.

Here's a possibility. It's quite possible that the client socket isn't ready to send yet when you're trying to push the second message through. select() returns a set of sockets ready for writing as well. Buffer data on a per-client basis and send it when select() indicates that the corresponding socket is ready to write.

Quote:

Original post by Kylotan I can't see anything obviously wrong, apart from that you're not checking if the writes will block.

Ok can you please elaborate that?

Quote:

Original post by Windryder Here's a possibility. It's quite possible that the client socket isn't ready to send yet when you're trying to push the second message through. select() returns a set of sockets ready for writing as well. Buffer data on a per-client basis and send it when select() indicates that the corresponding socket is ready to write.

I'm not sure if I got you 100% there, do you recommend me to buffer the data a client wants to send to the server, and then when select() returns the socket in write, it will send it?

Because there's no problem sending data to the server from the clients, the server always gets them instantly.
But maybe I should do the same on the serverside; buffer the data and send it to the sockets that is returned in write from select() ?

Thank you for all your replies!!

The problem is with 90% probability one of the following:
- Nagle (TCP_NODELAY)
- not properly writing/decoding length+data for each message
- blocking on something else in one of the loops

The Forum FAQ has more details on the first two.

If you're saying the client and server are setting TCP_NODELAY, are those functions properly passing the address and length of the "one" parameter?

Writes blocking is probably your least possible cause, unless you're trying to write more data per second than the network between the two machines can sustain (thus flooding the bandwidth).

Quote:

Original post by pjuke Quote: Original post by Kylotan I can't see anything obviously wrong, apart from that you're not checking if the writes will block. Ok can you please elaborate that?

Check for write availability in select, the same way you do for reads.

Quote:

Quote: Original post by Windryder Here's a possibility. It's quite possible that the client socket isn't ready to send yet when you're trying to push the second message through. select() returns a set of sockets ready for writing as well. Buffer data on a per-client basis and send it when select() indicates that the corresponding socket is ready to write. I'm not sure if I got you 100% there, do you recommend me to buffer the data a client wants to send to the server, and then when select() returns the socket in write, it will send it?

That is the way I've always done it, yes.

(Fixed my broken quotes.)

[Edited by - Kylotan on November 25, 2009 7:00:45 AM]

Quote:

Original post by hplus0603 If you're saying the client and server are setting TCP_NODELAY, are those functions properly passing the address and length of the "one" parameter?

Seriously, I don't know. I looked around at the internet but didn't find any reference on TCP_DELAY on python sockets. I only found some example code somewhere and copy pasted it.

If you got the time, please check out the code snipped I provided before, because I might be setting it the wrong way.

Quote:

Original post by Kylotan Great answer

Ok thank you!
I'm re-writing it all at the moment, lets see what happens!

Quote:

Original post by pjuke Quote: Original post by hplus0603 If you're saying the client and server are setting TCP_NODELAY, are those functions properly passing the address and length of the "one" parameter? Seriously, I don't know. I looked around at the internet but didn't find any reference on TCP_DELAY on python sockets. [...]

Then call getsockopt and see what you get back? :)

Quote:

Check for write availability in select, the same way you do for reads.

AFAICR, WinSock will always return a socket as writable, even if the buffer is full, and then it will secretly block inside send() anyway. The only way around that is to use I/O completion ports with OVERLAPPED I/O.

The other question is: what would you do if it's not writable? You have data you need to send, right? The best thing you can do with clients that can't keep up is to detect the problem, and drop them, IMO. (With a proper log message so it's easy to track down why, for customer servive and analysis purposes)

Btw: Some time ago, I wrote a small Python "MMO" server (text based) using sockets. You might want to read it for reference. I haven't tested it very heavily, though.

Quote:

Original post by hplus0603 The other question is: what would you do if it's not writable? You have data you need to send, right? The best thing you can do with clients that can't keep up is to detect the problem, and drop them, IMO. (With a proper log message so it's easy to track down why, for customer servive and analysis purposes)

But you're not advocating doing away with an application-side send buffer entirely, are you? I can imagine a 'bursty' server occasionally generating far more data than can be sent immediately. (Which is probably still a problem, but not one worth disconnecting a client for.) I'd presume you'd set an upper limit on the send buffer and do the disconnection if they exceed that.

The kernel-side buffer is generally between 16 kB and 64 kB. If you have a "bursty" operation that generates more than 16 kB at a time, then you've got other problems :-)

However, no, I don't advocate getting rid of application-side outgoing buffers. In fact, I think "send" from within the application should just put the data in a buffer. Your general network pump (that selects and reads/writes) should then attempt to drain the outgoing buffer.

However, too much queuing is not good, because data will age as it's in the queue. Sometimes, queuing little workers who know how to generate data is better than queuing the data itself; when you actually need to form a packet, call each worker to generate data until you have enough to fit your currently available send window size.

Ok guys so I'm back!
I completely re-wrote my network-code.
But the problem still exists!!! (argh).

Let me tell you how it works:

Server
Each cycle in my loop, I call a function called tick() which selects on all sockets connected, to see if we got something from the client.
It uses a global incomming buffer (one buffer for all clients that is) to put the messages in.
Then a just do like this: for message in get_messages():.
get_messages() returns the buffer with messages. My message-handler is this loop.

Client
Same as the server.


For some reason, the packages are sort of delayed.
On each packet, I put a timestamp from the server's clock.
The time between packages are 50 milliseconds btw.

And the results are something like this:

Server Client
1.00 Got message at 1.00
1.05 Got message at 1.05
.... .... (skipping some time)
.... .... (skipping some time)
.... .... (skipping some time)
5.00 Got message at 2.05
5.05 Got message at 2.10
5.10 Got message at 2.15
.... ....
.... ....
12.00 Got message at 4.55
12.05 Got message at 4.6
12.10 Got message at 4.65


The client is receiveing "old" packages.
But my incomming buffer on the client is always 1 or 0, so it doesn't get stuck in there, it is on the sending side I believe, but I don't know...

And one more thing!
If I suddenly stop the sending from the server, the receiving will stop on the client as well. So there must be some kind of "buffer" between the client and server that secretly holds the packages and sends them when the buffer is big enough, or I don't know?


So therefore I'm begging you for help!
Thanks in advance!

EDIT
I found a solution of the problem, but I don't know if it's really it...
The thing is that I'm sending one position-packge per client, to each and every client. By other words, a client will be receiving a package containing its own position.
And for some reason, if I disable that, and just sending packages containing others positions, it works!

Why is this?
This can't be the problem really, I mean, why can't a client receive info about it self?


EDIT2
Forget tha last edit, lol.
I tried a few more clients and the problem was there again, so it is the same problem as before...

[Edited by - pjuke on December 14, 2009 1:41:15 PM]

Time to fire up Wireshark. Look at exactly when those messages hit the network on the sending side and arrive at the receiving side. That will help you narrow down where the problem is.

Your abbreviated results are not very helpful. Somewhere between 1.05 and 5.00 the times started drifting - where, and by how much? Did they slowly diverge? Or did there abruptly come a point where the messages took an extra 3 seconds?

And why is your client claiming to receive messages earlier than they were sent? That is not possible so you must be leaving out some information. Probably the best thing is to have the client log its current time as well as the timestamp in the message side-by-side so we can compare the two.

Also, tell us exactly what your testing setup is. One machine? Two machines? Internet? LAN? Is wireless involved?

And when do you put these timestamps in there? When you create the message? Or when it is actually sent? Make sure it's the latter.

(PS. Stop calling them 'packets' if you're using TCP.)

Quote:

Original post by Kylotan Time to fire up Wireshark. Look at exactly when those messages hit the network on the sending side and arrive at the receiving side. That will help you narrow down where the problem is. Your abbreviated results are not very helpful. Somewhere between 1.05 and 5.00 the times started drifting - where, and by how much? Did they slowly diverge? Or did there abruptly come a point where the messages took an extra 3 seconds? And why is your client claiming to receive messages earlier than they were sent? That is not possible so you must be leaving out some information. Probably the best thing is to have the client log its current time as well as the timestamp in the message side-by-side so we can compare the two. Also, tell us exactly what your testing setup is. One machine? Two machines? Internet? LAN? Is wireless involved? And when do you put these timestamps in there? When you create the message? Or when it is actually sent? Make sure it's the latter. (PS. Stop calling them 'packets' if you're using TCP.)

Ok!
Let's begin then. First of all, I'm not very familiar with Wireshark but one thing that I noticed is that my TCP stream are not shown. I'm sending data 20 times a second but nothing is showing up. Well there is something, but that is mostly HTTP requests when I visit a website or something. Nothing on my local PC.

At the moment, I'm running the server and client on the same machine. It's connected by cable, no wireless. I have also tried with my laptop and my PC, both with wireless and cable on the laptop. Same issue, and nothing in Wireshark.

I will give you a better analysis as soon as I get Wireshark to work.

And the timestamp is set almost excactly before the sending. You see, my message-handler works by using instances of a Message-class, where I have a variable that holds the time. I'm setting the timestamp exactly before serialization (using pickle) and sending it.



So my question for now, before I can give you some more detailed information, is how I get Wireshark to capture my messages?

Thank you for helping, I really appreciate it! :)

On Windows, Wireshark can often not capture traffic that's totally on the localhost. You should either run the server on a second machine, or set up port forwarding on your router and connect to the forwarded port on the router, rather than localhost.