I'm building the networking for an MMO engine and am working on tuning the TCP performance. In wireshark, I've been seeing slowdown scenarios like this:

T=0ms: Client sends 29 packets with len=500
(sends are accumulating)
T=54ms: Client receives acks for all 29
T=54ms+: Client starts sending again with len=1460

There's 2 things going on here:

1. It seems like the send buffer is getting full (I can send just about 16kb before it starts waiting).
2. Messages are being packed into max-MTU packets if there's multiple messages available (I static-compiled SDL_Net with TCP_NoDelay to make sure Nagle's was disabled.)

I don't care much about the latter since it isn't waiting, it would just be good to know what mechanism is doing it in case it needs to be tuned also.

For #1, it doesn't seem like SDL_Net gives me a way to adjust the buffer size. Am I missing something? If not, what are my best options for moving forward?

Things I'm considering:

1. I could fork SDL_Net, it seems simple enough that maintenance wouldn't be that bad. I'd be in the realm of writing platform-specific tuning code though, which I'd rather not do.
2. ENet seems interesting. I've been avoiding UDP to save on implementation complexity, since I likely won't need the performance, but this could be a way to go. (Edit: Also noticed that valve put out a lib https://github.com/ValveSoftware/GameNetworkingSockets)
3. I could roll my own reliable UDP (again, trying to avoid this).
4. Other libraries?

hplus0603 said:

Talking about “Reliable UDP” and “TCP" in the same post makes no sense, though. Either you need a bulk protocol with head-of-line blocking and in-order data delivery, or you don't. If you need it, use TCP.

Hah, seems obvious now, but I hadn't considered that. My app logic already works off the message's tick number, out-of-order delivery would only require adding a little logic to a couple spots. UDP it is, then. I still need reliability, so I'll have to decide on a library.

You can build reliability into UDP quite easily. A library can do it if you want, but the work isn't difficult if you decide to roll your own.

There are many variants on sliding window protocols and re-sending unacknowledged data. In their simplest form you keep track of old data for a few milliseconds. If they haven't replied that they received a block, you begin re-sending the data. When it is acknowledged it is discarded. Unlike TCP's requirements, this lets you choose to accept and process data as it arrives, or to wait around if you want to block until the sequence arrives in order.

frob said:

You can build reliability into UDP quite easily. A library can do it if you want, but the work isn't difficult if you decide to roll your own.

There are many variants on sliding window protocols and re-sending unacknowledged data. In their simplest form you keep track of old data for a few milliseconds. If they haven't replied that they received a block, you begin re-sending the data. When it is acknowledged it is discarded. Unlike TCP's requirements, this lets you choose to accept and process data as it arrives, or to wait around if you want to block until the sequence arrives in order.

Yeah, I'm almost to the point of just rolling my own. The options for libs are:

1. ENet - might be fine, but no longer supported
2. Raknet - also no longer supported
3. SLikeNet - Potentially good improvements over raknet, but I had to modify the CMake to get it to build, which isn't a good sign. Also the code quality of the improvements seems kinda shoddy. Also also, it's massive and I won't need most of it, which adds bloat to my project.
4. Yojimbo - Gaffer said it's only really supported up to 256 clients. Could work for more, but might need to optimize it myself.
5. GameNetworkingSockets - Code seems all over the place, API seems to still be in flux. Also potential performance issues with large connection counts.

Everything I choose to do myself adds time to my (already way too long-term) project, but I'd rather make my own minimal thing than take on other people's bloat and tech debt. Modifying Yojimbo might be a nice option, at the very least I'm gonna learn the codebase as inspiration for my own thing.

Maybe I'll just roll my own thin TCP wrapper as a halfway house to check performance (would be really fast to implement), then switch to reliable UDP if it isn't good enough.