Skip to main content
gamedev.net
🔒 Locked

MMO architecture

Started by Basiror Feb 27, 2009 at 8:37 AM 16 replies 7.3k views
Original Post
Basiror
Basiror
Hello, I have a basic question regarding the server architecture of MMOs Lets say the world is divided into zones, and each process/server takes care of one such zone. How would you implement the transition between zones? My current plan is to overlap the 8 neighbouring zones and to notify the neighbouring servers/processes about the transition There are 2 architectures I can think of a) - each zone has its own server - a central server which forwards all packets between clients&servers upon transition the zone server stores the information into the database server and sends a notification to the central server. The central server notifies the new owner of the client about the transition and retrieves the client's data from the database server. The client's packets are automatically redirected to the new zone server. This way the client does not realize the redirection and only has to maintain one connection to the central server. Of course you can handle multiple clients with multiple central servers, just for the case that the traffic throughput gets too high. b) Same setup as above, just without a central server the client connects to. Upon transition the client has to connect to the new zone server and retrieve the transition data/gamestate of the new zone. Which system would you use? I am in favor of a), it is just simpler to implement, but I have no experience with multiplayer programming, so any advice is greatly appreciated. thx in advance bye
http://www.8ung.at/basiror/theironcross.html
prashh
prashh

Most MMO game servers do not handle interaction between/among players in different zones. Zone by definition is mostly that.

In a, am not sure why need a central server to distribute messages between client and server, seems like a bottle neck as well as single point of failure, unless I misunderstood of what you mean by central server.

A does have advantage that the client does not have to reconect.

You could check out http://www.gamantra.com/DeploymentGuide.pdf get some ideas.

Be one.
DrHalan
DrHalan
Imo if you keep this central sever really light and hand all big operations to the zone /chat servers I would prefer a).
My Blog - http://www.freakybytes.org
hplus0603
hplus0603
We use a model where the client connects to all zone servers within view, and zone servers tell the client about objects within its borders only. (Technically, we have a view server for each simulation server, which actually tells the user about the objects)

An alternative is to make the client connect to only one zone server (except when transitioning), and make each zone server tell the client about all objects within client range (this means that zone servers need to forward their updates to their neighbors).

A third option is to go with the "central server" approach, but to have more than one. Simply build a separate map for updates, and the client will connect to the central server for its area. That server is connected to each simulation server within its purview, and may exchange information for objects close to it in other view sectors.
enum Bool { True, False, FileNotFound };
disks86
disks86
One problem with a & b is that each zone server would have to be aware of all ajoining zone servers correct? That sounds like a headache waiting to happen.

If I were you I would break the servers up like most mmorpg's do my realms, districts, or whatever your choice term would be. Then if they want to switch servers log them out and they can reconnect to the other server using your login logic.

As for getting a list of available realms that would be a job for a central server or possibly multiple servers that can be reached using the same dns name.

I guess what I'm trying to say is IMHO making each server it's own world is easier than trying to make each server be a part of a world.

Also if you need to have each zone/map processed independently I would use a thread. The overhead of spawning a new thread is less than spawning a new process.


hplus0603
hplus0603
Quote:
I guess what I'm trying to say is IMHO making each server it's own world is easier than trying to make each server be a part of a world.


Yes, it is! And it makes for easier-to-design gameplay, too! I've heard some MMO designers say, at GDCs past, that they built a seamless zoning world, but that their next game would go back to zones, for gameplay reasons.

That being said, if you're trying to mimic a real world for some reason, then you have to bite the bullet.

Quote:
Also if you need to have each zone/map processed independently I would use a thread. The overhead of spawning a new thread is less than spawning a new process.


The reason to split into zones (apart from gameplay reasons) is for scalability. If you put different zones into different threads in the same process, then you can't move different zones to different machines. While you can be an 8-thread CPU these days (and Sun even has something 64-way I think), the main bottleneck is the memory throughput, and having 8 different threads all slamming the same memory bus is no good. Better to put them each on a different process, so you can put them on a different machine.

Also, with processes, if one crashes, the others don't. And, finally, the difference between spawning a process and a thread, given that this is something that only happens once, on start-up, isn't worth talking about -- it's not significant at that point.
enum Bool { True, False, FileNotFound };
elFarto
elFarto
I've always wondered how Azul's machines would hold up to an MMO. Their biggest machine has 864 cores (16 processors, 54 real cores per processor) and
768 GB of RAM.

Regards
elFarto
hplus0603
hplus0603
How much cache and memory bandwidth? That's what you'll be limited on for an RPG style system. In general, if you have 54 cores on a single chip, the memory bus into and out of that chip will be the bottleneck (almost literally).
enum Bool { True, False, FileNotFound };
elFarto
elFarto
Some information here and here. In that they state:

"205 GBps aggregate memory bandwidth."

In the blog post he states:

"We have a lot of memory bandwidth - one of our bigger boxes is on the top-20 supercomputer bandwidth list."

Seems they have a 16K L1 cache, and I can't find it's L2 cache size. They have some other cool stuff like RTPM (which is like having a profiler permanently running).

The only problem I can see with the Azul boxes is it's floating point performance probably isn't that good (I don't know for sure). Oh, they also only run Java applications (which isn't really a problem).

Regards
elFarto
chapter78
chapter78
Do you have access to the ACM portal? If you do there is a great paper on designing the architecture of a MMORPG dealing with scalability issues and state management. I would share it but I think this is against the rules!
hplus0603
hplus0603
I bet that Azul box would do pretty well with Darkstar. I did read up on it, and it's an interesting concept: You install a JVM on your regular server, where that JVM re-vectors to the Azul boxes, where individual cores run the Java threads. Each core isn't anything special, but if your Java app is highly threaded (like a web service app) then they can accelerate it just fine (modulo locking).

I wonder what they do about high-bandwidth I/O, though -- they talked about Fibre Channel, but wouldn't the JVM model mean that you have to first slurp the I/O into the host machine, and then forward it to the JVM accelerator machine?


Yes, I have the ACM digital library. Please share the link.

Btw: if it's an article by someone who hasn't actually built and operated an MMO, I'd read it mostly for the entertainment value. What people think is important up front, vs what turns out to be important in operation, are usually very different.
enum Bool { True, False, FileNotFound };
chapter78
chapter78
It is probably just worth reading for the fun of it, it is fairly general. If you do a search for 'game architecture' on the ACM library there are a few which may interest you on a similar subject.

Here's the link: http://portal.acm.org/citation.cfm?id=989066.989081&coll=Portal&dl=GUIDE&CFID=25935922&CFTOKEN=49257567
Kylotan
Kylotan
Seemed pretty pointless to me to be frank; academic hand-waving at numerical solutions to a problem that is already well-explored empirically.
Basiror
Basiror
Quote:
Original post by hplus0603
We use a model where the client connects to all zone servers within view, and zone servers tell the client about objects within its borders only. (Technically, we have a view server for each simulation server, which actually tells the user about the objects)


Letting the client connect to the neighbouring zone servers sounds like an elegant solution.
First you don t have to wait for reconnecting to the new zone server upon transition and
it also makes handling interzone actions easier.

E.g.: The client is in zone A and casts a spell at an actor in zone B, you simply tell each of the connected zone servers that you are casting a spell and upon completion the server that actually holds the target actor resends the response to the client

time 0: Client begins to cast a fireball (takes 3 seconds) on actor in zone B
time 0-3: all servers receive the clients cast message and store it in a queue
time 3: the server that actually holds the target actor answers with a respones


Between frames each server has to update its gamestate with neighbouring zone servers.

e.g.:
time x: actor 1 transits to zone B (crossing some threshold to avoid flipping)
time x+1: zone B acknowledges actor 1's transition and tells its neighbours about its ownership


That should be easy to implement I think


Any further suggestions?

thx for your help
http://www.8ung.at/basiror/theironcross.html
kyoryu
kyoryu
It's a lot easier to have a single "connection server" that acts to forward messages to the appropriate zone server.
Drew_Benton
Drew_Benton
Quote:
Original post by Basiror
Quote:
Original post by hplus0603
We use a model where the client connects to all zone servers within view, and zone servers tell the client about objects within its borders only. (Technically, we have a view server for each simulation server, which actually tells the user about the objects)


Letting the client connect to the neighbouring zone servers sounds like an elegant solution.


How exactly do you do this though? Would you pretty much be limited to using UDP? If so, would a client be sent a packet that tells it the new zone server's address and port it should talk to and likewise on the server side, the zone servers have a copy of what port and address to send to?

How would you handle security on your packets in this event when working between so many servers? I would assume each server handles its own security, but the security protocol is the same across all servers.

I know a lot of these questions are dependent on the game and "how you want to implement it", but I'm having a hard time understanding that specific solution. I'm so used to connecting to one server via TCP or one server via UDP (FPS games shouldn't need this model). I do have a few threads on my to read list, such as Distributed server architecture for load balancing and a few others I don't have the links handy, but this thread has some interesting stuff in it too!
kyoryu
kyoryu
The client sends a message to the "connection server", including an identifier specifying the desired recipient. UDP or TCP doesn't matter.

The connection server has a lookup to what internal connection (probably TCP) the recipient is on, and sends that message to the destination.

The major benefit is that you lose a lot of handshaking complexities, dealing with connections that don't happen, etc. The client sends everything to one place, which then takes care of internal routing.

It also lowers your attack surface area, since only one process needs to be exposed (or possibly multiple copies of the same process for load reasons), while the majority of your processes are completely secured behind firewalls.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.