Some questions regarding mmorpg server code architecture

http://www.kegel.com/c10k.html

C-Junkie, thx for the link, that is planty of interesting materials and docs, but it is oriented on servers providing common internet services (http and ftp)... Those services are based on the following scheme: connect-request-reply-disconnect. As for the games, the connection lifetime can vary a lot, and the amount of data sent and received by both sides is undetermined... I would be glad if you post links to similar analisys and reviews of popular networking paradigms concerning game servers...

Games that I know of and their OS''s for a fact:

Asherons Call - Windows 2000
Everquest - Windows NT
Anarchy Online - Windows 2000
Shadowbane - Windows 2000
Dark Age of Camelot - Linux / Windows 2000 mix
Ultima Online - Windows NT
EVE Online - Windows 2000
Asherons Call 2 - Windows 2000


I can''t really think of any other major MMORPGs at the moment. I spoke with the majority of these development teams at one point or another, which is how I learned what OS / database servers they use.

check out Planeshift (http://www.planeshift.it)

You know if any of them use IOCP?

-=[ Megahertz ]=-

thx, Imperil

And what about question #6 ? TCP is perfect for online games, because it already has reliable delivery and connection management, but one of the most significant problems that rise when using TCP is packetizing the stream (games usually treat incoming and utcoming data as packets (commands, states, etc...), not streams). When using TCP with its stream-based nature, how to properly handle multiple buffers and data pending to be sent/received through multiple connections?

quote:

---

Original post by roookie
thx, Imperil

And what about question #6 ? TCP is perfect for online games, because it already has reliable delivery and connection management

---

It depends on the game. For turn based or non time critical things ( cards/checkers/etc ) TCP will be perfect. For games that require quick delivery, then TCP is evil. See this article on Gamasutra for some nice discussion about online protocols.




-------
Andrew
PlaneShift - A MMORPG in development.

acraig, mmorpgs are not first person shooters, there''s no fast action in there... Mmorpgs can afford lag of several hundreds milliseconds. All commercial mmorpgs i know use TCP. The question was not to point out advantages and disadvantages of certain protocols.. The question was: how to packetize the streams when serving multiple connections? (but still thx for answering)

Question #9
Does anybody know of any good complete step-by-step tutorials on IOCP?

It''s been a while since I brought out the packet sniffer but I''m fairly sure that both Everquest and Asheron''s Call 2 use UDP for the main gameplay data stream. Most other MMP''s probably do as well. Chat may be on a seperate server cluster that uses TCP.

I''m not sure what you mean by packetizing stream with respect to multiple connections. You break TCP streams into messages by either having a size indicator, by having fixed size messages with the size implied by the message id, or some combination of the two. That has nothing to do with managing multiple connections though.

Anon Mike, i mean application breaking TCP streams into packets, so that the main server logic could work with user commands, but not raw data streams. Can you describe the code logic which can do this (should be non-blocking)?

quote:

---

Original post by roookie
acraig, mmorpgs are not first person shooters, there's no fast action in there... Mmorpgs can afford lag of several hundreds milliseconds.

---

The lag for TCP can be several SECONDS, as many as 30 or higher in bad cases. A mmorpg can't deal with that sort of lag any better than can an FPS.

quote:

---

All commercial mmorpgs i know use TCP.

---

What are you basing that on? As far as I know, that is entirely opposite of being correct. All commercial mmorpgs you know use UDP. They MIGHT mix in a bit of TCP for the non-time critical stuff (like patch updates) but their main game protocols are UDP in almost every case. I believe AO was entirely TCP at first (and perhaps they still are, I dunno), but they are the exception, not the rule. And they had horrendous lag issues in the beginning (which is why I have a feeling they switched to UDP as well, though I've no reference for that, just a guess).

Maybe you have some reference that prooves me wrong, heh. But I think you might be kidding yourself. TCP is almost always a bad choice for non-turn based games. There are any number of references on gamesutra (the Tie Fighter vs. X-Wing post mortem on gamesutra being the most famous and probably most quoted) that bear this out.

If you want help with IOCP, I've written an IOCP server. It's exclusively UPD though But I will help where I can with advice and questions if you'd like it.

Ron

[edited by - RonHiler on January 30, 2004 6:16:41 PM]

RonHiler, well i was not speaking of all the commercial mmorpgs, i only mentioned the ones, that i tried... DAoC, AO use TCP (checked that with netstat, no UDP sockets were open)... UO also uses TCP (and its free emulators, like UOX, etc)...
UDP is good for its speed and its packet-oriented nature, but i''ll have to manage online presence of clients manually... Got any ideas on implementing that (logins, logouts, authorization, etc...)?

I know UO used TCP and it wouldnt suprrise me if AC and others used it too as its primary protocol. Think about it, it simiplifies the design and increases bandwith utilization, both goals far exceed the need to reduced latency for a MMORPG. Some players might get the occasional 30 sec lag burst, but their connection would have to be horrible or the server is overloaded, to be dropping that many packets to casue TCP to do that.

-ddn

Well, i was aiming on TCP, but now i''m in doubt what to use...

The two questions are:
1. TCP case: how do i packetize it?
2. UDP case: how do i manage connections, logins, logouts and other "pseudo-connection" related stuff?

To "packetize" TCP -

1. Have a big buffer, bigger then the biggest message you''re going to send.
2. Check to see if the buffer is full, if so you either have a logic error or the user is trying to hack you. Either way drop him.
3. Read some data into the buffer.
4. Check to see if you''ve received a full message yet, if not go to step 2
5. Act on the message in some appropriate way
6. Copy any remaining data to the front of the buffer and goto step 4 (the last read could have read the end of one message and the beginning of the next in one shot).

That''s it. Nothing special.


For UDP connection management -

Logins and logouts are just messages. There''s no connection related stuff to worry about. At most you may want to assume he disconnected if he logged out.
When you call recvfrom or equivalent you''ll get the ip/port of the sender of the packet along with the data.
Lookup the ip/port in a table somewhere. If you can''t find it this is a new connection. Add the ip/port to the table.
Note in the table the last time you''ve received data on that connection.
Every once in a while walk the whole table. If you find an entry that is to old assume he disconnected. Log him out if necessary and delete the entry.

We use Linux servers and prefer UDP connections, but if there''s a problem getting through firewalls and whatnot, we "change down" to TCP (and the user has a laggier experience). We find that about 0.9% of our users are connected through TCP on a typical day.

We use select(), which on our gateway/router machines (packet heavy) take as much as 9% of the CPU. I''d probably go with something like /dev/poll in a new generation, although the other implicit/asynchronous methods are probably good, too. On windows, you need something asynchronous; their select() is horrible (and can only deal with 64 sockets at a time). Multi-threading per connection will kill your server with synchronization overhead; state machine is the way to go. While we only use one socket for UDP connections, the servers talk to each other using other sockets, so we have a fair number of them open at a time (like, more than 100, and less than 10,000 :-).

Our cluster is mostly homogenous 2.4 GHz Pentium IV/DDR 1U machines, 100 MBit to a Foundry switch, and 300 Mbps to the ''net; the database servers are 2U and have dual RAID set-ups; one for database log, and one for database table space.

("We" == large-ish commercial MMOG with single shared world (no shards); not on your list)

quote:

---

Original post by RonHiler
The lag for TCP can be several SECONDS, as many as 30 or higher in bad cases. A mmorpg can''t deal with that sort of lag any better than can an FPS.

---

World of Warcraft (which is a MMORPG) currently uses TCP for all game communication. Which is something I find interesting.



--
Martin Piper
Network programming made easy with ReplicaNet
http://www.ReplicaNet.com/

Regarding "packetizing" TCP, because the stream is guaranteed in-order, it''s no harder than writing a size field before the actual data. Use scatter/gather to avoid copying.

int send_tcp_message( uchar code, usize size, void * data )
{
  iovec iov[ 3 ];
  iov[0].iov_base = &code
  iov[0].iov_len = sizeof( code );
  iov[1].iov_base = &size // warning! byte order!
  iov[1].iov_len = sizeof( size );
  iov[2].iov_base = data;
  iov[2].iov_len = size;
  return writev( mySocket, iov, 3 );
}

int receive_tcp_message( uchar * outCode, usize * outSize, void * outBuf, usize maxSize )
{
  iovec iov[ 2 ];
  iov[0].iov_base = outCode;
  iov[0].iov_len = sizeof( *outCode );
  iov[1].iov_base = outSize; // warning! byte order!
  iov[1].iov_len = sizeof( *outSize );
  int l = read( mySocket, iov, 2 );
  if( l != sizeof( *outCode ) + sizeof( *outSize ) ) {
    // stream is de-synched at this point!
    return (l < 0) > l : -1;
  }
  if( *outSize > maxSize ) {
    errno = ENOMEM;
    // stream is de-synched at this point!
    return -1;
  }
  return read( mySocket, outBuf, *outSize );
}
 

Note that, in a real app, you''d either want to drop the user when you go de-synch in the protocol, or write the data into a buffer, which you''d empty into the read queue the next time you''re reading.

Also note that synchronous reads are really bad, because a user could send you a single byte, and then nothing more, which would make you hang forever.

Last, TCP requires that each user has his/her own socket open on your machine, which causes more kernel stress. You''re better off with UDP if you can swing it.