30k concurrent players on a (private) MMO server...is this possible ?

Hi all,

I know that private server is a touchy subject, but please bear with me.

Recently, a work colleague claimed that he has hosted a MMO server which can sustain 30000(!!) players, all are online at the same time.

Furthermore, all the players are in very small number of areas, doing PvP.

I am very perplexed by this claim, and personally feel that it is a very tall claim, but my other work colleagues said it's possible due to the game in question require very small bandwidth.

I would like to hear from you guys on what you think from MMORPG networking technical standpoint. What would be (very rough) estimate of bandwidth and also server hardware required to support such conditions. Is this possible at all ?

Further details :-

- He claimed this is way back in 2005. The server in question is single node powered by Pentium 4

- The MMO in question is Ragnarok Online, and the PvP event is War of Emperium

If you use dynamic allocation then you can keep adding players until you run out of memory.

The real constraint would be CPU and network of course.

EVE-online has had like 70,000 on a single realm but I do believe they have a cluster of physical machines that behave as one server.

It really depends on how its coded. If the amount of memory used by each player is very small and there isn't alot of CPU overhead then you can have tons of players.

Assuming the server has a 100mbps internet connection, divide that by 30000 and you're left with 416 bytes per second, per player.
It's probably possible to design an MMO architecture where each player uses less than half a KBps of bandwidth... I've never done it but it doesn't sound completely unbelievable.

As far as I know, commonly these games would use 1-3KBps, so 0.5KBps is quite low... but possible. Keep in mind that servers often run at extremely low frame-rates, such as 10fps, or lower!

It's definitely possible to design code to update 100k MMO-type characters on one CPU.

The hosting costs would of course be massive, unless it's hosted somewhere where 100mbps/gigabit access is common...
Also, the server machine would have to have lots of IP addresses assigned to it (or a large IPv6 subnet) else they'd run out of network ports!

Assuming the server has a 100mbps internet connection, divide that by 30000 and you're left with 416 bytes per second, per player.
It's probably possible to design an MMO architecture where each player uses less than half a KBps of bandwidth... I've never done it but it doesn't sound completely unbelievable.

It's definitely possible to design code to update 100k MMO-type characters on one CPU.

The hosting costs would of course be massive, unless it's hosted somewhere where 100mbps/gigabit access is common...
Also, the server machine would have to have lots of IP addresses assigned to it (or a large IPv6 subnet) else they'd run out of network ports!

Yeah, personally the server connection requirement is what caused me to hardly believe this claim. He justified that many players are online and in very small areas because of the weekly PvP siege event, but from my understanding that would even shoot up the bandwidth per player, considering RO PvP has lots of Area of Effects elements.

I read somewhere in this forum that MMORPG that time normally target 1kb/s per player on average, but in this situation it must be way higher than this right ?

I believe even if the server allow and can sustain 3000 (1/10 of his claim) players in one area, I believe the players internet connection might not.

Appreciate more input from you guys, hehe. I really want to believe him, but if this possible at all it must be a huge technical feat and I would be interested to know how ;)

Another thing I forgot to mention - I have a friend making an MMO that can handle over 500000 players off of a single server (Intel i7 CPU, 30mbps connection)... He does it by keeping most of the traffic peer-to-peer. Players discover who is nearby in the world, and then those players communicate directly without need for the server.

I believe even if the server allow and can sustain 3000 (1/10 of his claim) players in one area, I believe the players internet connection might not.

Let's say whenever you click, you send the server a structure describing that command - where in the world you clicked, what you clicked on, and what kind of click command you were giving (move/cast/shoot/etc):

struct PlayerInput
{
  int32 clickPosX, clickPosY;
  uint32 clickedObjectId;
  uint16 commandType;
}; size = 14

A high level player might have something like 300 clicks per minute, which is 5 clicks per second, which makes 70Bytes/s or 0.56kbps player upload.
On the other end, if there's 30000 clients, that adds up to 16.8mbps of download on the server side.

Lets say that whenever the server gets a movement command, it starts moving that player towards the click location. It then sends every player in that local area an update, saying where that character is moving to, and when they will arrive.
This doesn't have to be sent to everyone in the world, just those players who can actually see this character.

struct CharacterMovement
{
  int32 posX, posY;
  uint32 arrivalTime;
  uint32 playerId;
}; size = 16

Lets say that even though there's 30k characters online, you can only see 20 of them at once. If every player is sending 5 movement clicks per second (everyone is spamming), and everyone can see 20 characters, then the server has to send to each client:
20 visible characters * 16 bytes per update * 5 updates per second = 1600Bytes/s = 12.8kbps
However, the server has to send that to 30000 clients, so the server upload = 45.7MB/s = 384mbps

From there you can tweak things -- lets say that the server ignores click-spamming, so only sends at most 1 position update per character per second. In that case, suddenly the client-side download is 2.6kbps and the server-side upload is 76.8mbps.

You still need extra bandwidth for sending player attribute updates (you lost 20HP!), spawning effects/items/projectiles, moving NPC's etc... But this scenario still sounds feasible for clients on 56k modems and a server with a 100mbps backbone!

I don't see why it isn't possible on a correctly developed and tested protocol.

Let's take irc as a example. Freenode server can regularly have upwards of 10k clients on it connected using tcp and back in the day maybe ten years ago irc servers used to hold tens of thousands more (this is already less efficient than the udp protocols used by most games) that use maybe 512bps average. These servers are usually low end machines, bog standard dedicated servers with 100mbps Burstable... So in short... Why not?

Edit: also irc servers have to deal with regular and brutal denial of service attacks, connect floods etc much like game servers so this might be a good analogy... Food for thought...

He does it by keeping most of the traffic peer-to-peer.

Well with that logic the number of players is nearly infinite. You only need a very small amount of traffic to tell players about each other initially, and from that point on the players mainly communicate with each other. And if you aren't too concerned with cheating, you keep all the item/logic client side rather than server side. No item updates, no location updates, no nothing, just a lot of ip addresses and save/retrieve state.

Why do you think PirateBay was one of the world's biggest sites with the least amount of hardware. P2P tech rocks.

Another thing I forgot to mention - I have a friend making an MMO that can handle over 500000 players off of a single server (Intel i7 CPU, 30mbps connection)... He does it by keeping most of the traffic peer-to-peer. Players discover who is nearby in the world, and then those players communicate directly without need for the server.

That is very interesting way to reduce the bandwidth requirement for the server. I think for turn based MMO, or low level players per grid/area it would be great.

In case of real-time combat MMO with high number of players per grid, I think P2P will be way harder especially on maintaining 'chronology authority' ( e.g who attack/kill/loot first ) and also attractive for tampering since most data bypassing the main server.

I think Ragnarok Online client itself use standard client-server model and does not support P2P....thus the 'magic' most likely at server side.

I still lacking details of what type of internet connection used by his server, but he just told me he is using consumer level broadband. In 2005, typical broadband speed is only ~1-2Mbps in my country...

I don't see why it isn't possible on a correctly developed and tested protocol.

Let's take irc as a example. Freenode server can regularly have upwards of 10k clients on it connected using tcp and back in the day maybe ten years ago irc servers used to hold tens of thousands more (this is already less efficient than the udp protocols used by most games) that use maybe 512bps average. These servers are usually low end machines, bog standard dedicated servers with 100mbps Burstable... So in short... Why not?

Edit: also irc servers have to deal with regular and brutal denial of service attacks, connect floods etc much like game servers so this might be a good analogy... Food for thought...

I think that IRC servers are comparable somewhat ( based on my personal and not professional knowledge :D ). But in a way you making me doubt my friend even more. There is nothing remotely close to 100Mbps in 2005 in my country for non-enterprise user.. maybe around 2-5Mbps only

Bandwidth is very unlikely to be the constraining factor, actually.

CPU starvation would be much more probable. A game of any appreciable complexity takes serious computational overhead well beyond the network requirements. RO is a pretty simple game (no 3D physics, straightforward rules, etc.) but I'd be pretty surprised in a 2005-era P4 could simulate 30,000 simultaneous players.

Think of it this way: if it was that easy to host tens of thousands of players, why would MMOs typically sink huge amounts of money into datacenter hardware? It'd be stupid to build all the infrastructure to host your playerbase if you could do it on your average laptop.

Have in mind Ragnarok Online is a 2D game, so don't think "30k WoW players doing PvP" level of complexity. Its sprites on a 2.5D background.

You can get some info about the game and the engine on Wikipedia

http://en.wikipedia.org/wiki/Ragnarok_Online

http://en.wikipedia.org/wiki/AEGIS_(Ragnarok_Online)

Apparently, it wasn't very well designed.

And btw, if he is remembering stuff from 10 years back, definitively he is painting it a bit pink at least. Maybe it "worked", but it lagged like fucking hell.

Here's all the game logic you need:

auto newpos = player.pos + player.vel;
if (!map.blocked(newpos)) player.set_pos(newpos);

map.blocked() is a simple grid or hash table look-up.
player.set_pos() is a variable update and an update of the map -- either hash table delete/insert, or map grid write-write.
There's approximately three cache misses worst case (this includes dereferencing player, which can be pre-fetched if you loop through all players.)
Three cache misses at the time was about a microsecond total. That leaves a frame rate of 33 Hz for 30,000 players, minus whatever overhead for networking.
(My guess is it ran nowhere near 30 Hz on the server, though.)

On modern machines, there's some chance you can keep the map in L3 cache, which means only one cache miss per player -- the actual player record. And, as I said, that could be pre-fetched.

For networking, you'd likely want to use epoll, kevent, or I/O Completion Ports (depending on O/S).

Physical simulation makes for a significantly more CPU hungry implementation, especially if 3D geometry collision is involved.

For something round-based, this is pretty trivial, but for something kind of "action, PvP", I think the claim is a bit unreasonable for a single server (withough e.g. doing some P2P trickery -- which is not really "a single server" -- as suggested above).

Here's all the game logic you need

Well yes, almost. Though in reality it needs to be a little bit more complex, and the "little bits" are unluckily quite expensive. Something like:

for(;;)
{
while((result = system_poll()) != timer_object_id)
{
    if(is_connect(result))
        log(add_new_player_blah());
    else
        log(parse_command(decrypt_and_verify_auth_token(recvfrom(...))));
}

for(auto p : player)
{
    auto newpos = player.pos + player.vel;

    if (!map.blocked(newpos))
        player.set_pos(newpos)
    else          // a non-cheating client software should normally
        log(...); // not allow this move in the first place
}
for(auto c : clients)
    send_updates();
}

In particular, send_updates must figure out what to send to each client. Sending the complete map is out of the question (death by bandwidth) as is sending a fixed small subsection of the map (including tiles with no changes, but possibly excluding tiles outside your area which are however interesting). Deciding what to send is a O(N²) problem, and N² for large N is not precisely a trivial thing to cope with. Of course you can prune away some of those 30k (zones, spatial hashing, you name it), but

the OP makes explicit mention of "PvP", "siege" and "very few small areas", which suggests that no matter what you do, N will be considerably large. If only 1,000 of those 30,000 fight against each other in each siege, then collision/interest queries are 1 million operations (so, 30 million alltogether, for 30 such sieges). That's without actually creating a response, encrypting it, copying data to the network stack or processing it. 30k calls to send are not entirely free either (assuming most people are actually actively fighting, they will want updates every round!).

Also, system_poll() (that might be epoll_wait or whatever) and actually reading and processing the readied sockets will have considerable overhead which is more than just a few cache misses. The usual assumption is that connecting to 30k clients means most are inactive most of the time anyway, and epoll is O(1) in respect to that. However, it is of course still O(N) in respect of actual stuff that happens. There is no way it could be any different, if N events happen, you must process N events.

For "PvP in constrained areas" that "most are inactive anyway" assumption does not hold -- most are active most of the time.

A single call to any not-totally-trivial kernel function like epoll_wait or recvfrom costs a minimum of 5,000 clocks (I haven't actually measured this to be honest, but I think it's a rather conservative estimate for those two functions -- in fact, receiving a datagram may easily cost 4-5 times as much if you include the kernel overhead for interrupt, going through iptables, reassembly etc, and copying data on the user end), so doing maybe 5k calls to epoll_wait and 20k calls to recvfrom per second might already consume roughly 40 milliseconds, and there you haven't actually done anything useful yet (yes, 40ms does not look like a lot, there are 1,000ms in a second... but mind you, that's just for receiving stuff, not actually doing something yet -- plus you may want more than just one update per second, at 10 ticks per second, you only have 100ms available).

Especially in a PvP game, regular complaints about cheaters will inevitably come, and you will have to deal with them in a manner that satisfies the (anyway never satisfied) audience. Which means that as the most basic thing you need to log pretty much everything, and logging must be kind of reliable and failsafe, and in a format so your customer service / moderators / bot heuristics can easily and efficiently replay, parse, or query data. Which, of course, is possible but not at all a "free" operation.

RO-players are often bragging and exaggerating things - he was probably speaking of 300, not 30000.

The game got inefficient TCP networking and IIRC doesnt even set the nodelay option. Its well known that an official server (actually a server-cluster) would hold about 2000-3000 people, in exceptional case 8000. And if there were more than 200 on a single mapserver in WoE it started lagging, at about 250-300 moves could take a minute until the server reacted, skill commands would be ignored and people would get disconnected constantly from CPU-overload. I remember a company licensing the game apologised for the constant lag and thereby admitted the game was so inefficient that even though they bought the fastest machines available (although they got told their old machines were good enough which they coudnt believe) it did not help much.

I doubt the server emulator I heared the pservers were using was much more efficient and even more, there would never have been that many people on an illegal pserver.

hplus0603/samoth,
That is an interesting view on processing power needed to just update on player movement. My sentiment however that O(N^2) problem must also apply to his server bandwith requirement, considering the exceptional number of players in his claims... is this correct?

wintertime,
While I do personally believed that he is overexaggerating, I lacks the facts to refute his claims. Considering the huge advancement in mmorpg in recent years especially from networking perspective, I thought I might have missed something that might make it a possibility.

Another question though, in order to support such high number of players, are the consumer level networking devices (e. g router) able to cope with the load? Or do I need special switch/router/etc for the server?

My sentiment however that O(N^2) problem must also apply to his server bandwith requirement, considering the exceptional number of players in his claims... is this correct?

Not necessarily, no. Actually, not at all

You usually decide for some maximum bandwidth per client (in the simplest case) that you will allow, and then you figure out what you can send in that bandwidth, ordered by importance. So the per-client bandwidth would be "constant" and the total bandwidth requirement would be O(N). But you can of course do a less naive bandwidth management, so the total bandwidth is capped as well (not only limited by what can physically go over the cable).

Your character's position and life status would be among the most important things to transmit, as well as anything that happens very close to your character (other character swinging a sword at yours, etc). The farther away things are, the less important frequent updates are (usually, that is). Some events are more important than others, too (someone shooting a fireball at you vs. a team member dying vs. someone putting on a silly looking hat).

One common strategy would be to find the most important X events (final importance being a function of distance and event importance) and transmit those changes, presumably giving events that were not transmitted a little more weight for the next round. That's basically a nearest neighbour search which is O(N²) although with a structure like a kd-tree it would be O(log(n)) at the expense of extra memory and the cost to build and maintain the tree (which is a nuisance with moving/changing objects, rebuilding from scratch is most probably easier and faster than maintaining the tree). In any case it is not a totally trivial thing to get right, and not totally trivial on the CPU side.

If you have tiles, a different strategy could be to "walk" the tiles in a spiral-like manner starting from your character's position, and send updates from events that happen on tiles that you touch, until you have X events together. I'm not sure what complexity that is, but it might actually be O(1) in respect of the number of players, since nothing really changes when there are more players. However, this is certainly not the most "deterministic" runtime (you might visit 5 tiles or 50, depending on what happens when and where). Nor is it the most cache-friendly way possible to traverse data.

You guys are over complicating the problem. The problem isn't to create a "proper" MMO with proper validation, it's to create a ragnarok online server capable of having that many players...

AKA a server that takes game updates from the client (nonvalidated). Of course it's possible on a P4, but you'll get people teleport hacking/breaking collision.

Now creating a simulation with that many users where the client isn't trusted... That sounds very iffy. I suppose it depends on the game's physics rules. It might be possible to run a simulation with that many users... It also depends on how important the data is. If there's 50 characters entering a region do they all need data, or can it be sent gradually?

So if there's 50 players entering a region the player is going to, how much data needs to be sent to get the simulation accurate (Besides things like customizations etc).

Wait did he work on the client or just the server?

My sentiment however that O(N^2) problem must also apply to his server bandwith requirement

Bandwidth really isn't the problem. Gigabit interfaces were standard in 2005, and a gigabit commit of co-lo network traffic is about $2k/month (probably was somewhat higher then, but not enough to kill a game.)

Of course it's possible on a P4, but you'll get people teleport hacking/breaking collision.

And, for all we know, that's exactly the experience that was delivered.

Btw: If the map is a grid, you just need to send the occupied data of the WxH cells around the player with any frequency. I'd imagine that, the more occupied the grid is, the less frequently you'd get a full refresh.

The real outcome of this discussion is likely "you can push any one axis about as far as you want -- it's pushing on all axes at the same time that's hard."

The one thing I didn't hear anyone discuss is, where did he get 30k players? That many players at once implies a huge user-base.

I played RO for a long, long time, the most players online I have seen on a server was ~20k on iRO, near the launch.

If there is a private server with 30k players, I would play in it @.@

On topic.

Some very important information is missing in this topic:

- RO is a zone based game, to pass from one zone to the next you need to enter a warp. This makes away easier for the server to spread the players around.

- RO had several big cities, people would spread mostly based on their levels (I would pick mine based on the music, search aldebaran theme on youtube, you won't regret it).

- WoE was played in 20 different castles, each with more than one line of defense. I would say that 1k people in a single attack is possible, but very unlikely, heavy attacks would have 300 players involved (search "how to break an unbreakable defense" on youtube and you will see how massive the attacks can get).

From my experience I can say I hardly ever lagged, even on the most hard core invasions, sometimes I would take some times to be transfered from one zone to other, but rarely had a laggy experience from the server lagging (and I player with 128 ~ 256kb connection at that time).

The one thing I didn't hear anyone discuss is, where did he get 30k players? That many players at once implies a huge user-base.

Probably clones and duplicates