MMO server cluster architecture

Could you elaborate more on option 2, or provide a link to something short and readable? As I see it, pretty much all MMOs would have a server constantly serializing/deserializing to/from DB backend, for some definition of constant.

And I don't agree that option 1 is necessarily that complicated. Load balancing can be done quite easily, perhaps by having multiple zones per server, or by balancing characters rather than zones, etc. Once you have a reliable system for migrating a player from one server to another, you have pretty much everything you need. You do need to consider it a distributed system from the beginning however.

http://www.projectdarkstar.com/documentation.html has tons of highly-marketing info with some technical details

Quote:

pretty much all MMOs would have a server constantly serializing/deserializing to/from DB backend

yep, but darkstar uses DB backend for synchronization as I understood, so game state are stored in DB and not in memory, so no caching - no synchronization, only DB transactions. But what is latency?

Quote:

Load balancing can be done quite easily, perhaps by having multiple zones per server, or by balancing characters rather than zones

Zoning is described as approach #1 and seems to me to be much complicated than char balancing that is described as approach #2. If it's true so the main questions are: does #2 has acceptable performance and scalability and what are its main cons for MMO games?

brief and more technical then marketing Darkstar description: http://www.projectdarkstar.com/components/com_mambowiki/images/1/1b/Agcarch_v2.pdf


[Edited by - @ on August 29, 2008 3:03:15 AM]

Quote:

Please comment/correct/supplement this description or describe alternative approaches.

That would depend on your requirements.

How many actions/sec, how many users and how do they behave, how many mobs, complexity of physics, complexity of world, persistence requirements, accuracy and latency requirements, ....

Quote:

Original post by @ Quote: pretty much all MMOs would have a server constantly serializing/deserializing to/from DB backend yep, but darkstar uses DB backend for synchronization as I understood, so game state are stored in DB and not in memory, so no caching - no synchronization, only DB transactions. But what is latency?

The pdf you linked to doesn't say anything about there being no caching. It looks to me to be more of a distributed database system - useful in itself, but not really comparable to geographical partitioning of servers. It's comparing apples and oranges. You could have still have geographically partitioned zones with Project Darkstar, and I can see good reasons to do that.

Latency might be poor, depending on how it works. Hard to say without seeing more technical details. Obviously if 2 servers try committing to the same data object, one will succeed and one will fail, and if some other task needs to execute serially then that needs to be resolved first. That won't be a problem if a single server authoritatively owns any given data object - but that doesn't necessarily mean it's geographically partitioned, either. Apples and oranges.

Quote:

Zoning is described as approach #1 and seems to me to be much complicated than char balancing that is described as approach #2.

No, I don't see any reason why that would be the case.

Quote:

Original post by Antheus Quote: Please comment/correct/supplement this description or describe alternative approaches. That would depend on your requirements. How many actions/sec, how many users and how do they behave, how many mobs, complexity of physics, complexity of world, persistence requirements, accuracy and latency requirements, ....

I think this game can be described as a typical fantasy medieval MMORPG: swords, bows, dungeons etc. Number of players - high scalability is required, mobs, physics etc. - typical case, interactions - middle level.

Quote:

You could have still have geographically partitioned zones with Project Darkstar, and I can see good reasons to do that.

Yes, completely agree. I don't plan to use Darkstar itself. I just analyzing their approach. So I doubt what way to choose: typical zoning approach mainly described as #1 or to start from darkstar approach and only then add some zoning or any other techniques as performance improvements. Under Darkstar approach I mainly mean their transactional model. Of course DB performance requirements are very high in this case. They say something about performance tuning for 50% reads 50% writes. Still investigating this.

Zoning requires specific design from the beginning. First - static zoning, then "seamlessness", dynamic partitioning (seems very chalanging task for me not to implement but to choose an algorithm) etc. Transactional approach seems to me to have completely different way: DB, serialization, publishers/subscribers, interserver communication etc. And after that - mainly performance improvements since you already have good scalability.

What can I receive at the end of both ways? The same result or one of them is more promising? Which one is easier/cheaper?

sorry for so many questions ;)

hm, just found an interview with Darkstar's lead (http://java.sun.com/developer/technicalArticles/Interviews/kesselman_qa.html) where it says "Essentially, what we're building is a transactional database tuned for massive scalability at a 50 percent read-50 percent write data load".

Seems like this is not the simplest possible way regardless of many advantages it promises, especially taking into account their resources and that they do it already some time and still cannot show scalability.

There's an article in the latest CACM about Darkstar, too. There, we learn that, while they can add more machines to spread the load, they currently actually LOSE throughput when you do. Oops!

In general, though, I think the idea to have full consistency for each game object, and having each game object be able to decide what other objects it can interact with, is not going to scale sufficiently -- it simply over-delivers on guarantees, which ends up costing a lot and removing a lot of flexibility that you need to make the system perform well.

In essence, you'll have to remove certain often-changing properties (object location, say) from the database consistency guarantee, and once you do that, the benefit of using an object database like SGS instead of a regular database disappear. But you still have to pay the penalties, such as not being able to run ad-hoc queries.

I love the direction of this thread, dealing with OODMBS's, and have always felt the could be used in a massive game.

I have quite a bit of experience with them, and I do love them. But, they have their good sides and bad sides.

As hplus stated, by nature an OODBMS has no native data mining to support ad-hoc queries; however in the business logic of the MMO, the design will probably support a direct mapping of objects to the database schema, so there is no need (or mininal) for standard queries or ad-hoc queries. This is the blessing of the OODBMS, and this is the curse of the OODBMS.

The best performing solution on small gear using open source OODBMS I provided for a for an ISP could stream 1000 writes per second sustained, approaching 82 million records per day. That was balanced for 90% writes / 10% reads, and the reads were a bitch being truly impacted by the load.

I would pursue a backend for an MMO using object databases, but probably as a secondary project. It really comes down to experience for the development team, as most likely the pure dream of streaming objects that extend the core code would be overused until the learning curve is met where to best use the concept. If done correctly, I think the core performance would dance around a relational database core on the same level of hardware, and would probably meet it's performance using less gear.

It is vital to think through the design using streaming/persistent objects, as they are not query based but can actually support your real-time business logic (game logic) - but probably not all of it. I do not think I would count on the transaction model to solve all the concurrency issues (collisions, etc.,); I think that still needs to be designed in the main core; it could be done using a streaming cache off the OODBMS - which you would probably need to design into both the client and server.

A huge bonus to a good OODBMS structure is a 'divide and conquere' approach. Solutions perform fantastic using several core object databases, each designed for a particuliar task. Typical relational designs seem to favor a megalithic database that encompases everything; the best object database designs are like your code - small efficient libraries linked together, but with each 'library' an actual server core program handling requests to resolve the objects required. And this is usually a normal fallout that occurs with object systems once a comfort level is established with them (just like C++).

I have often wondered why they are not used in MMO's as the native mappings to the objects are typically C++. I have come to the realization it is as simple as people not being aware of them as they follow the cookie-cutter approach of what has gone before with the extreme duress of getting a title out, and it is difficult to overcome the momentum of the core 'relational' databases being used. They do work.

Note that there is no quicker one-to-one lookup in a database if you know what you need then is achievable when using an OODBMS with a good schema. They are used world-wide for 411 lookups by all the major telcom's.

I have been using a free open source solution called 'GOODS', which is rooted in a transaction based model. It performs fantastic for enterprise, and I would take a stab using it for an MMO. However, I have put thousands of hours into developing a framework around it - not touching the actual OODBMS code - but a transaction layer to pack/unpack objects, client and server cache beyond what the database provides (which would be critical for an MMO), automatic code generations, data binding, threading, networking, etc., into a useful framework. It is all more work then most realize. Sigh.

You need to know what your strength is; don't loose sight of the game over technology.

Best luck!

I think that I would suggest taking the custom approach: identify your game's particular needs and then tailor your server infrastructure to fit. As with any server system, it's essential to start with a strong networking subsystem upon which you can confidently build your application layers. Beyond that, how you choose to partition server responsibilities and distribute load depends on your specific game mechanics. Because databases are slow, you'll probably need to cache data in a server somewhere. And if you're intending to release your game on a global scale, then you'll need to geographically partition some login and game servers clusters (as you mention), and maybe local data caching servers as well. These are likely the crux architectural issues you'll face, though the issue of load balancing should be included in the set of difficult aspects. I'm avoiding it here because it can be very domain specific.

My thoughts on Darkstar, etc.: In the same way that generic game engines seem to take the uniqueness out of individual games, I fear that generic game server solutions may have a similar negative effect on MMO-style systems. Inevitably someone will get it right enough to be usable, but I think that for a significant time to come the truly unique MMOs will be ones that roll their own server systems. I believe that a market does exist for offloading MMO server infrastructure, which would likely require adoption of the service's load-balancing mechanism.

Anyway, good luck!

Thanks for all thoughts and advices. Seems like iterative advance to "dynamic zoning" is my way.

Quote:

Original post by humanapp Because databases are slow, you'll probably need to cache data in a server somewhere.

What is typical strategy to deal with DB? Does its clustering is required only for reliability or for performance improvement too? Only replication is used as a scaling strategy or some "zoning" here is possible too? How does it interfere with dynamic zoning of the virtual world?

Quote:

Original post by humanapp ... And if you're intending to release your game on a global scale, then you'll need to geographically partition some login and game servers clusters (as you mention), and maybe local data caching servers as well...

Do you mean to keep geographically (in real world) located servers to minimize latency? Doesn't it add new complexity level to virtual world zoning? Can "instance dungeon" be still acceptable approach in this case, e.g. in combination with localization etc.?

Does anybody can suggest some info about virtual world dynamic partitioning algorithms?

I've read an article "Improving the Performance of Distributed
Virtual Environment Systems" where in conclusion was said that

Quote:

The average system response (round trip-delay of messages notifying movements of avatars) remains practically invariant with the number of avatars in the system until the DVE system reaches a saturation point. This saturation point is given by the limit of 100% CPU utilization in any of the servers. ... The results show that workload balancing mainly has an effect on system throughput, while the amount of inter-server messages mainly has an effect on system latency. However, if none of the servers reaches saturation, then the amount of inter-server messages does not have any significant effect on system latency.

Maybe this was the main reason why I was so interested in SGS approach since seems it is more targeted to improving workload balancing and less to minimizing inter-server communication then traditional "zoning" approach. Does anybody can comment this research results?

Quote:

Original post by @ Quote: You could have still have geographically partitioned zones with Project Darkstar, and I can see good reasons to do that. Yes, completely agree. I don't plan to use Darkstar itself. I just analyzing their approach. So I doubt what way to choose: typical zoning approach mainly described as #1 or to start from darkstar approach and only then add some zoning or any other techniques as performance improvements.

To be honest, I think you'll find aspects of one bleed into the other. Can you definitely say that an actor on one server will never need information about an actor on another server, even if you operate a one-server-one-zone policy? As soon as you do that, you need to set up a system for making requests of other servers and receiving responses from them. From there, why not make it possible to act on those remote actors, since you already have a system for getting the request to them on the relevant server? And once you've got that, most of the rest follows naturally.

Quote:

Zoning requires specific design from the beginning. First - static zoning, then "seamlessness", dynamic partitioning (seems very chalanging task for me not to implement but to choose an algorithm) etc. Transactional approach seems to me to have completely different way: DB, serialization, publishers/subscribers, interserver communication etc. And after that - mainly performance improvements since you already have good scalability.

I disagree, as I think almost all non-trivial systems will have a bit of both. Even highly partitioned servers with no inter-zone communication may still share data, such as player accounts. And you probably still need transactions even if you only ever had one server, such as for ensuring consistency of trade. (Where 'trade' could be anything from NPC purchases, to PC swaps of items, to spending skill points on skills.)

Possibly, you could have small enough zones that each can run entirely in memory, with a fast enough file system that means you can save each character after every trade, and enough stability that consistency doesn't become a problem, and with a front end server that can handle all your account info atomically and separately from anything else. But I think most people wouldn't be able to do all of those.

Quote:

there is no need (or mininal) for standard queries or ad-hoc queries

Most well-run games run all kinds of interesting queries for business purposes; all the way from "what's the average level per class" to "show me the 100 richest players in the world." What those exact queries are will change daily. Sometimes, you can still do it, but extracting the runtime data from the OODBMS into a relational data warehouse, but then you suddenly have to worry about both how to create the OODBMS schema AND the relational schema for your data model.

In general, my money is on the federated data model -- not all data in a single database instance, but instead use reliable messages to make sure that cross-slice updates make it through. Only a very few kinds of transactions (mostly trade) truly require distributed transactions, in the sense of a transaction monitor.

Quote:

Original post by humanapp ...And if you're intending to release your game on a global scale, then you'll need to geographically partition some login and game servers clusters (as you mention), and maybe local data caching servers as well. These are likely the crux architectural issues you'll face, though the issue of load balancing should be included in the set of difficult aspects. I'm avoiding it here because it can be very domain specific.

Is there any techniques that can be used for most MMOGs? Seems this topic is the worst clarified one. The problem is to combine geographical deployment with zoning. Will geographical replication of zone server work fine? Especially taking into account how slow latency can be over an ocean. Should special prediction logic be used in this case or client's one already implemented to compensate latency of communication with local server (geographically) is enough?
Any alternatives?

Quote:

Is there any techniques that can be used for most MMOGs?

What most MMOs do is shrding. Their game world is only designed to handle 1,000 - 5,000 simultaneous online users, and gets too full if more people are playing. Thus, they replicate the same content across many shards, giving each one a separate name ("Darkbane," "Brell Serilis" etc).
For international players, it's then pretty easy to put some shard servers in Europe, some shard servers in Asia, and maybe even one or two down under if you're really lucky. Players are then recommended to choose a server close to themselves for ideal playing experience.
The only thing you need to distribute across the globe in such a set-up is the acount login/verification stuff, which is not real-time.

And, to answer your question, no, replicating a simulation server across an ocean is not trouble-free, so most games don't do it.

Well, since the technology is open source, why not just download it and try it for yourself? As for multi-node, we released the first version to show people where we thought the direction would go. We never claimed it would be performant as this is still in the research stage. At this time, we continue to make significant improvements across the board.

At the Austin Game Developers Conference this week, we demonstrated scalability of the single node server with our Project Snowman game, which was a CTF game all done in a 3D engine called jMonkeyEngine. We were running 400+ games that consisted of a total of 1000 players and 3000 bots at the same time on a U1, 4 core x86 server. The load monitor was showing almost even load across all four cores all of the time and only peaking the server at 35%. Oh, and the server was handing collision detection, logic and AI for all games.

The best way to see what a piece of tech can offer you is to try it. There is no risk as it is free, and the community building up there is starting to make some fantastic contributions. Hope you stop by and check it out!

www.projectdarkstar.com

-Chris

Quote:

Original post by CMelissinos Well, since the technology is open source, why not just download it and try it for yourself?..

the most important reason is time. Time required to investigate what your solution already provides and what is just marketing buzz, time to realize if provided functionality really costs dealing with 3rd-party solution. As I realized you just came up real clustering problems. But it's the most important problems that we're faced with. When do you plan to provide proven scalability across nodes, not only across cores?

As the last but not the least issue is Java implementation, nothing against Java itself, but I have a requirement to use C++. I think this is quite reasonable decision, only Erlang can look more promising but I don't have enough experience/knowledge for expert opinion here. Even Python microthreads looks very interesting, but I'm afraid only for massive clients simulation. Does anybody can comment this opinion?

Your game server is really interesting project because its architecture (as it's described in your marketing documents) is elegant and can simplify quite challenging tasks. But first you should prove its end performance characteristics.

Quote:

dealing with OODMBS's

I didn't catch this before, but I thought it important to mention. A "true" OODBMS will support various forms of attribute indexing and querying (compare for example Versant); however, the SGS model right now is just an ID->blob of serialized Java object.

Quote:

Original post by @ Quote: Original post by CMelissinos Well, since the technology is open source, why not just download it and try it for yourself?.. the most important reason is time. Time required to investigate what your solution already provides and what is just marketing buzz, time to realize if provided functionality really costs dealing with 3rd-party solution. As I realized you just came up real clustering problems. But it's the most important problems that we're faced with. When do you plan to provide proven scalability across nodes, not only across cores? As the last but not the least issue is Java implementation, nothing against Java itself, but I have a requirement to use C++. I think this is quite reasonable decision, only Erlang can look more promising but I don't have enough experience/knowledge for expert opinion here. Even Python microthreads looks very interesting, but I'm afraid only for massive clients simulation. Does anybody can comment this opinion? Your game server is really interesting project because its architecture (as it's described in your marketing documents) is elegant and can simplify quite challenging tasks. But first you should prove its end performance characteristics.

I understand time is an important consideration, but the reality is that, for any tech, you will never get the appropriate view of technology purely through demos, literature or benchmarks. Because each game type uses varied transactional models and each developer does things their own way, the best way to determine the technology impact is to trial the tech yourself. That does not pertain just to Darkstar, but to any tech you plan on using.

One suggestion is to post your inquiries to the forums at ProjectDarkstar.com and listen to what other developers are doing. As well, you can always PM me and we will help in answering as many questions as we can.