Skip to main content
GameDev.net gamedev.net
🔒 Locked

Efficient.. "sendto"

Started by Ysaneya Apr 10, 2005 at 4:15 PM 31 replies 6.1k views
Original Post
Ysaneya
Ysaneya
Hey, I'm using datagrams on a small test program in order to maximize performance. I'm creating 2500 datagram sockets, each socket sending a packet of 50 bytes, in localhost, on a powerful machine (WinXP SP2, Athlon 64 3500+). Well, believe it or not, it takes 810 milliseconds to send 2500 packets (or 125 KB). The CPU usage rockets to 100%, from which 75% are used by the "SYSTEM" process, and 25% by the test.exe process. It doesn't matter if a server is listening to the destination address or not, or if the server is on the same machine or on a different one. I've made a dirty app demonstrating the "problem" (it's 100 lines, please have a look if you got 5 minutes):

#include <winsock2.h>
#include <windows.h>
#include <stdio.h>

#include "CTime.h"

class CClient
{
protected:
	SOCKET					m_socket;				// socket connection
	struct sockaddr_in		m_myAdr;				// client address
	struct sockaddr_in		m_adr;					// destination address

public:
	CClient(const char *hostName, const unsigned int port)
	{
		m_socket = INVALID_SOCKET;
		m_socket = socket(AF_INET, SOCK_DGRAM, 0);
		if (m_socket == INVALID_SOCKET)
		{
			throw "Can't create client socket";
		}

		m_myAdr.sin_family = AF_INET;
		m_myAdr.sin_port = 0;
		m_myAdr.sin_addr.s_addr = INADDR_ANY;
		if (bind(m_socket, (struct sockaddr *)&m_myAdr, sizeof(m_myAdr)) != 0)
		{
			throw "Can't bind client socket";
		}

		struct sockaddr_in destAdr;
		destAdr.sin_family = AF_INET;
		destAdr.sin_port = htons(port);
		destAdr.sin_addr.s_addr = inet_addr(hostName);
		if (destAdr.sin_addr.s_addr == INADDR_NONE)
		{
			throw "Can't find destination address";
		}
		m_adr = destAdr;

		/// flags the sockets as non-blocking
		u_long flag = 1;
		if (ioctlsocket(m_socket, FIONBIO, &flag) != 0)
		{
			throw "Can't ioctl socket";
		}
	}

	void process()
	{
		char buf[256];
		int bytes = sendto(m_socket, buf, 50, 0,
			(struct sockaddr *)&m_adr, sizeof(struct sockaddr_in));
	}
};


int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance,
	LPSTR lpCmdLine, int nCmdShow)
{
	/// init WSA
	WORD wVersionRequested;
	WSADATA wsaData;
	int err;
 	wVersionRequested = MAKEWORD( 2, 2 );
 	err = WSAStartup( wVersionRequested, &wsaData );

#define NBSOCKETS 2500
	CClient *clients[NBSOCKETS];
	for (unsigned int i = 0; i < NBSOCKETS; i++)
		clients = new CClient("127.0.0.1", 7000);

	while (1)
	{
		CTime ct;

		/// sending one packet per client:
		for (i = 0; i < NBSOCKETS; i++)
			clients->process();
		float e = ct.getElapsed();

		/// sleep a little (100 ms at worse)
		int sleep = 100 - (int)(e * 1000.0f);
		if (sleep < 10)
			sleep = 10;
		Sleep(sleep);

		char str[256];
		sprintf(str, "Elapsed: %.1f ms\n", e * 1000.0f);
		OutputDebugString(str);
	}

	return 0;
}
Why is it so slow ? I am quite amazed at these results. Y.
Ysaneya
Ysaneya
Why is it dumb ? 50 bytes is approximately the size of an "update" packet (ID, entity position/orientation, time, etc..). If you send 10 updates per second and wait until a buffer of 1 KB is full, you're only going to send the data every 2 seconds. Not good for any action game.

By the way, back on topic, the amount of bytes i send per packet is completely irrelevant in my problem. I still get 100% CPU usage if i send 5 bytes, or 1024 bytes (i just tested).

Y.
Ysaneya
Ysaneya
Also, it doesn't matter if the socket is set to blocking or non-blocking. If somebody could download my test application and let me know if the cpu usage also tops to 100%.. (warning: you'll have to kill it from the task manager). Thanks.
kinetik_mjg
kinetik_mjg
Out of curiosity, I ported your code to Linux and ran it. On my 3.0GHz P4 desktop box (hyperthreading enabled), CClientTest uses between 25% and 40% of one virtual CPU and of 652 sendto()s I'm seeing a min/avg/max 31.3/34.59/72.9ms per 2500 sockets.
John Schultz
John Schultz
Hi,

I tested your code: it appears you may have a problem with your specific machine (drivers, hardware, etc.). On my dev machine (ASUS P4C800-E 3.2, Intel CSA (not PCI, integrated gigabit LAN)) your test produced (debug build):

50 byte packets: ~27 ms
1400 byte packets: ~30 ms

John Schultz
John Schultz
I tested the code on an AMD 3500 machine (ABIT AV8) with a less powerful integrated GB LAN (VIA Velocity, on PCI bus), and it turned 380ms (not much difference between 50 or 1400 bytes). Giving it an IP address on the network (instead of localhost), it turned ~180ms.

Under the same conditions (IP address on LAN, not localhost), the ASUS P4C800-E with Intel CSA (again, not on the PCI bus, perhaps more efficient drivers as well), still turned around 27-30ms, with some higher spikes (180, etc.).

A 1.7 Centrino laptop (IBM T-42, Intel Pro1000 LAN chipset) turned 98ms (50 bytes), and 103ms (1400) bytes. IP address on network: 100ms (50 bytes).

It appears you'll want an Intel CSA Gigabit chipset (or similar, not on a PCI bus, or perhaps PCI Express x16, etc.) for a MMP game server.
Megahertz
Megahertz
ASUS P4C800-E here as well, not using onboard LAN (using a netgear FA310TX PCI network card)

Got an avg of 50ms on 50 byte packets.
Got an avg of 60ms on 1200 byte packets.


-=[ Megahertz ]=-
-=[Megahertz]=-
fenghus
fenghus
Just out of curiosity, why are you using 2500 sockets? Try sending 2500 packets using a single socket, maybe the performance will be better.
Ysaneya
Ysaneya
Thanks for all the reports. It would be nice if you could include these informations for comparisons:

- CPU type/speed, operating system,
- % of cpu time the application CClientTest.exe takes,
- % of cpu time the SYSTEM process takes,
- % of cpu time the IDLE process (remaining) takes (generally 0%)
- Average time in milliseconds it takes to send 2500 packets

I thought my home machine had a problem or had a specific config, but i tested many machines at work and it seems it's more the norm than the exception:

Machine1: Win2K SP4/P4 2.8 Ghz: APP:25%, SYSTEM:55%, IDLE:20%, Avg: 55 ms
Machine2: Win2K SP4/P4 1.3 Ghz: APP: 5%, SYSTEM:95%, IDLE: 0%, Avg: 1700 ms
Machine3: Win2K SP4/P4 1.3 Ghz: APP: 5%, SYSTEM:95%, IDLE: 0%, Avg: 1670 ms
Machine4: Win2K SP4/P4 2.0 Ghz: APP:47%, SYSTEM:49%, IDLE: 4%, Avg: 850 ms
Machine5: Win2K SP4/Dual AMD 2.0 Ghz: APP:45%, SYSTEM:20%, IDLE:35%, Avg: 500 ms
Machine6: WinXP SP2/Athlon 3500: APP:40%, SYSTEM:60%, IDLE: 0%, Avg: 850 ms

Y.
markr
markr
You really should just use one socket.

Creating so many sockets is not a good idea, it uses up a lot of resources needlessly.

2500 sounds like a lot, I guess it is a simulation of a large-scale server. Needless to say, it would still only need one socket.

What IP address are they being sent to? Is it on an ethernet LAN? Are there multiple IP addresses being sent to? If so, what is the ARP traffic like?

I think if it's going out of a physical ethernet interface, the network hardware will make a difference.

Also, did you check that all 2500 of them were actually received? The OS might choose to throw some of them away if it can't send them in a timely fashion.

Mark
Ysaneya
Ysaneya
As i said in the OP, it doesn't matter what IP address it is sent to. I tried on a LAN, in localhost, with or without a valid server listening. Nothing matters. The only thing affecting the performance seems to be the amount of connections.

Yes, it is to simulate a lot of connections to a server. How can i use only one socket ? If i create one socket, the server will detect only one client, and thus the test will not be a valid one anymore. I really want to test N incoming and different connections to the server. The sendto() call can take any destination address, but i can't modify the source address after the socket has been created, can i ?

Y.
Ysaneya
Ysaneya
I just tested one socket, sending 2500 packets, and it still makes little to no difference.

A server too, can be sending a lot of packets. If i can't send more than 3000 packets per second @ 100% CPU usage, how will my server scale with thousands of connections and other CPU work to do ?

Y.
markr
markr
Turn off your AV and software firewalls, and then try again.

Mark
Ysaneya
Ysaneya
They're off. That's the first thing i tried.
hplus0603
hplus0603
Ysaneva: it seems that you have something slowing you down in the send path. My guess is a poorly written network card driver, or just a network card that requires lots of CPU. I advise using VTune (for Intel platforms) or CodeAnalyst (for AMD platforms) to isolate where the time is going, and then looking for ways to improve that module. If the time is inside a driver, or inside a kernel function used by a driver (say, reading or writing hardware registers), then you need to get a network card of a different brand and chip set, and try that.
enum Bool { True, False, FileNotFound };
John Schultz
John Schultz
Network Device Comparison @ Tom's Hardware

As was reflected in your test, the Intel CSA design was the overall fastest network interface. A $50 Intel PCI NIC card was second fastest.

From the posts in this thread (on the same motherboard) the integrated Intel CSA was about twice as fast as an add-in card.

If using an Intel CSA network interface, and given that your server will send packets at around every 150ms (more or less, given average internet conditions), and that a 1400 byte packet takes about the same amount of time as a 50 byte packet (you can coalesce many smaller packets), 2500 players can be hosted without too much trouble. 150ms-30ms = 120ms left for other processing. If all 2500 players aren't likely to be in the same state space (not visible/collideable, etc.), the simulation/service can scale to many servers (with server hand-off as players changes zones, etc.).

Books that may be useful to you:
Massively Multiplayer Game Development.
Massively Multiplayer Game Development 2.
Ysaneya
Ysaneya
I can see it's hardware/drive related. But still, even with a poor network card, how can you explain the numbers i'm seeing? I tested a lot of different machines. Many posters in this thread reported similar problems. Is it normal that a network card can only process a few thousand packets per second at 100% of the CPU ? Even under the worst case, i would have expected any modern network card to process hundred of thousand of packets (assuming no bandwidth limitation, of course).

Y.
markr
markr
Call me stupid, but if you're sending packets to 127.0.0.1, no network hardware should actually be involved.

So any data you're getting is purely from the OS internal operation.

Mark
vNistelrooy
vNistelrooy
Pentium 4 530 LGA775 (overclocked to 3.7Ghz) with HT enabled
WinXP SP2
One of my virtual CPUs handles the SYSTEM proccess (100% on this CPU).
The other one handles your application (~14%).
Only the task manager takes a few %.
Everage time to send 50 byte packets: ~30ms
Everage time to send 5000 byte packets: ~25ms

Forgot: I have a PCIx Marvell onboard 1Gbit LAN with updated drivers (and not Windows generic drivers).

[Edited by - vNistelrooy on April 12, 2005 5:55:03 AM]
"C lets you shoot yourself in the foot rather easily. C++ allows you to reuse the bullet!"

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.