MMORPG Servers question

Thanks for feedback, here are my answers:

Quote:

How many units did your previous games sell? IF they were merely hobby projects that didn't sell then you still have too little experience to write, maintain, and run an MMOG.

Id definitely agree with Maintaining and running MMOG, though I disagree with Writing MMOG. As I said, its gonna be less feature-rich than e.g. Everquest2 (at least 10 times less, that is). This is accomplishable in 2 years with small team with graphics renderer (terrain/characters/animation/special effects/Collision Detection) already completed.

Quote:

Thousands. Any less and you've got a crap networking layuer / arch.

Sounds good considering wed be happy with 2.000 subscribers.

Quote:

Quote: I.e. How many servers/PCs would we need to serve those ~2.000 people ? Depends entirely on how complex your game is; if it's a real-time MMOFPS with lots of time-critical gameplay and large amounts of data pinging around then you'll might need as many as a server per couple of hundred players. However, IMHO that only happens if your architecture is crap, your network programmers are crap, and your game-design is crap. It takes considerable incompetence to be unable to manage 500 players per server, and in most games you should be able to manage 1k.

Well, its gonna be RPG, so movement isnt going to be that fast as in FPS, still I can imagine that submitting information about 2.000 players each second (or so) is going to be hard to achieve, but compression is my friend (or so I guess).
And 1000 people sounds good. That would mean Id need just 2 servers. Thats accomplishable.

Quote:

Quote: 2. Are there any commercial libraries (with support) for Multiplayer component of MMORPG (codebase) ? Plenty. There's even a website devoted to listing them (you'll have to google for it, something like "MMOG technology List" or something like that), although its' very basic info only. Shameless plug: http://grexengine.com (although the site is extremely plain. Just email sales @ grexengine.com and you can get the marketing info etc emailed to you)

Thanks for links, Ill check those out (some of them I already checked). Naturally I googled already, I just hoped someone will provide a real-world feedback for the library they work with (e.g. like you).

Quote:

Quote: 3. How many people would we need to make this all run 24/7 (i.e. support,shards,network specialists) ? For 2k players teh rule of thumb is you need approx 3-7 full-time staff (developers + support + technicians). I've seen 15k players supported, commercially, by a total staff fewer than 5, but that's exceptional (if you've got an excellent project manager on-board and a cunning game-desing, you could manage this too, but I wouldn't recommend it).

Sounds reasonable, actually thats what I found out elsewhere so far too. So it would be doable on $10k monthly revenue.

Questions :
1. Do you think that its impossible for one programmer to be adding new content and fixing bugs ?
2. How many PCs are required for one server ? I heard stories of 50 PCs per server. What`s that mean ?

you might be able to save on bandwith my doing something like what Shadowbane did, by using the mouse for movement. i thought it was a pretty good idea. Shadowbane required to use as little bandwith as possible, since it was sort of action based, and encouraged huge battles, Player vs Player battles where one guild would storm anothers castle.

a little OT but, the Shadowbane game design was amazing, but in reality it just didnt work. whenever a huge battle would take place, you and 50 people go to storm a castle, all hell (lag) broke lose. the whole castle storming thing was going on in _VERY_ slow motion.

however, that was years ago when the game first came out. id be very interested in knowing if they ever solved this problem.

sorry for slightly de-railing your thread.

I don't know what country and state you're in. In California, junior programmers cost at least $5k/month, and people who are experienced go for much more than that. Customer service might be cheaper, but there's still rent, and power, and cooling, and bandwidth, and hardware leasing... $10,000/month wouldn't work at all around here for a company size 3-7.

Regarding players per server, that depends on your degree of security, how much simulation you do, how interactive it is, and all that. If all you're doing is bouncing a few packets a second between players that are close to each other, you can easily go to 2,000 on a single piece of modern hardware. After all, most routers are just regular computers, and do many more packets than that. If you do secure server-authenticated simulation at high simulation rates where each tool, attachment, vehicle, weapon, etc is a fully simulated object, and fully replicated to neighboring servers and clients, and have expensive rules for who can see what, when, and where, then you can cut that in ten. If you're a typical RPG, you can design towards the high end of that. If you're a military simulation, you find yourself at the lower end (but then, your customer might have money to spend on hardware :-)

Quote:

For 10k players, you're aiming for a total staff of approx 5-15, for 50k players you want 8-20, all depending upon your game etc.

I can second these numbers.

There's also a question of man-hours versus hardware cost. If your people cost you $10k/month, and you can buy a $1k server to solve a problem that would take a person one week to solve, then buying the server makes more sense. I imagine that if your team is based in some country with cheap labor, the equation would look very different -- although what the size of the market would be is then less clear (trying to support the US market from abroad is a tough nut to crack).

Quote:

Original post by hplus0603 I don't know what country and state you're in. In California, junior programmers cost at least $5k/month, and people who are experienced go for much more than that. Customer service might be cheaper, but there's still rent, and power, and cooling, and bandwidth, and hardware leasing... $10,000/month wouldn't work at all around here for a company size 3-7.

Im from Slovakia as can be seen from the header of the thread. This means I can get junior programmers in my area for about $300 per month. Still, if I offer $500 per month, theyll be lining up in front of my door (Im in region with 23% unemployment rate (highest in whole country) where average salary is about $300). True, add 37% to that because of taxes&social insurance, but still its far from $5k as in California. Office holding 10 people costs around $300 per month.
Although, Id have to add about 20% to that because of inflation and the fact that it wouldnt start sooner that 2006/2007 IF we shall start the development later this year.

Quote:

Original post by hplus0603Regarding players per server, that depends on your degree of security, how much simulation you do, how interactive it is, and all that. If all you're doing is bouncing a few packets a second between players that are close to each other, you can easily go to 2,000 on a single piece of modern hardware.

Well, and what about if all 2000 players are close to each other (not necessarily on same screen), but close enough to send packets about all 2.000 players ? How do I scale this into more servers ?

Quote:

Original post by hplus0603After all, most routers are just regular computers, and do many more packets than that. If you do secure server-authenticated simulation at high simulation rates where each tool, attachment, vehicle, weapon, etc is a fully simulated object, and fully replicated to neighboring servers and clients, and have expensive rules for who can see what, when, and where, then you can cut that in ten. If you're a typical RPG, you can design towards the high end of that. If you're a military simulation, you find yourself at the lower end (but then, your customer might have money to spend on hardware :-)

I guess only time will tell. But I needed to know the worst scenario as to how many severs do I need to buy and setup and maintain.

Quote:

Original post by hplus0603There's also a question of man-hours versus hardware cost. If your people cost you $10k/month, and you can buy a $1k server to solve a problem that would take a person one week to solve, then buying the server makes more sense. I imagine that if your team is based in some country with cheap labor, the equation would look very different

Could you elaborate more about what do you mean by the example of $1k server ? Like it has software/configuration that prevents some type of crashes/bugs/issues that are otherwise handled only manually ?

Quote:

Original post by hplus0603although what the size of the market would be is then less clear

Did you mean to say that were just trying to adress home-country market ? LOL, I doubt Id find 5 people here, this is Slovakia with most of people having dial-up being paid by time (i.e. the more youre online the more you pay - pricing per minute). Those $5 theyd pay for 3 hrs of connection during day-time. Although there is pretty strong base of people having broadband (like me).
Naturally, since this is MMOG, well try to adress the world market where I think it is possible to find 2.000 subscribers paying $5 each month. I could be wrong though of course. And of course it would have to be new people, since our game would hardly have features that would cause EQ2 players to quit and start here.

Quote:

Original post by hplus0603(trying to support the US market from abroad is a tough nut to crack).

Well, I think that we could easily cover 18 hours per day of customer service, though 24/7 would be harder to achieve although most probably necessary. What do you mean by supporting US market from abroad ? Answering phone calls ? No problem with language here (except for salary these people require).

I still would like to know what do they usually mean when they say that 1 server is run by 50 PCs. Does it mean Ill have to buy 100 PCs for two servers if I want each server to be able to handle 1.000 players ?

Quote:

Original post by VladR Well, and what about if all 2000 players are close to each other (not necessarily on same screen), but close enough to send packets about all 2.000 players ? How do I scale this into more servers ?

my guess is simply the game will lag horribly. i have never even heard of (and definetly never played) a MMORPG which could support 2000 players in the same area. Shadowbane, a commercial MMORPG which was designed from the start to support many players in a small area (fighting each other no less) could not handle this. in fact, when i was playing Shadowbane, it could not even handle 1/10th of that many players. even with 200 players the game would lag horribly and be in slow motion. theres really not much you can do about something like this.

A game like EverQuest runs about 100-200 players per piece of hardware (at least at the time of shipping -- when a 1 GHz CPU was the top of the line). They have maybe 50 zones in the game. They could put very heavily loaded zones onto their own machines, but put sveral zones that seldom have that many people all on a single machine to save on hardware cost. Thus, an initial EverQuest "server" (really, a world instance, or "shard") would be run by something like 30 machines.

What redmilamber and I are saying is that, with the right networking structure and game design, and today's PC hardware, you can probably put 2,000 players on a single piece of hardware -- but it depends a lot on what you want to do with the game, and thus how that informs the networking and simulation infrastructure you need on the server side. If you want all of everything, at high simulation rates, you won't be able to do 2,000 entities on a single machine, for much of the same reasons that you don't have 2,000 simultaneous active NPCs in a single-player game of Half-Life.

The easiest way to load balance between physical servers within a single shard is to require the user to "zone" between areas of the game, and put different zones on different machines -- the EQ model. The harder way is to let the user walk freely, and use shadow/ghost objects to hand off simulation responsibility between different physical servers. Asheron's Call did this, and many modern systems (including ours) does this. Our web site is somewhat low on details, but take a look at Forterra Systems for a few more details.

Quote:

Original post by Anonymous Poster How many units did your previous games sell? ... So. You mean "none", yes?

Actually, I dont know. We do budget games at present which means we get a fixed fee for license of our game. I dont care if it sells 1000 or 10.000 pieces. I only care for the license fee which at present is enough for us.

True, itd a BIG jump IF we started development of MMOG (though a 10 times feature-less than competitors have).

Quote:

Original post by Anonymous Poster Sure, it's doable. But, in reality, not unless you have one or more MMOG experts on the team or you have some top 0.001% people in your team (hey, it's possible; just very unlikely). I don't know your situation, I don't know what mega-genius you have, I'm just pointing out my experiences based on having worked with a lot of people in your position and they all went bankrupt, wound up the company, or simply gave up. And these were the ones with probably the best chances of success, carefully selected from hundreds.

It is very unlikely (aka impossible) to find some MMOG expert in my area. So from this point of view its impossible.

But it was also said to be impossible to create a game by only one programmer and one artist (in spare time after daily job) and yet we did it twice and are finishing our third 3D game. Of course MMOG cant be compared with Single-player budget games, theyre probably 100 times tougher to finish. Still, I could try. And for this Im gathering as much info as possible as to whether Ill try or not.

Quote:

Original post by Anonymous Poster Well, obviously I'd say the one I know and use and helped write is "the best", although I'd also say that BigWorld's one is the only other good one (all the rest are moderate to mediocre (butterfly, zona) or just crap (being kind, I'll simply mention no names - it's pretty obvious who the useless ones are when you look at their tech docs). The other question is: how many of them can you afford? IIRC all the decent ones cost mahola...

Could you please be more specific and give me a link where pricing information could be found ? It probably is outside my range. But maybe I could buy it on leasing. Otherwise I wont know. Or just tell me the range in US$.

Quote:

Original post by Anonymous Poster Quote: Questions : 1. Do you think that it

s impossible for one programmer to be adding new content and fixing bugs ?

Possible, yes. I've seen it done. However, before you get to 2k people you absolutely MUST have dedicated "content" programmers and "bug-fixing" programmers. Otherwise what happens is one of those two jobs doesn't get done properly...it's jsut too hard, mentally, for one person to handle the workload without getting so depressed they get no work done at all, or simply ignore one aspect.If youre talking about development phase, thats natural to have these two programmers have separate roles. Or were you talking about launch phase ?

Quote:

Original post by Anonymous Poster Quote: 2. How many PCs are required for one server ? I heard stories of 50 PCs per server. Whats that mean ? We've been through this: in the above conversation, PC == server. What you're thikning about are clusters, and I wish everyone would call them that, because that's what they are, they are NOT servers. A cluster is a group of servers joined together to *simulate* a single server, but it is not (should not be) called "a server". Unfortuantely a significant nubmer of people insisit on muddying the waters :(.

Thanks for clarifying. Now it makes more sense. Actually thats what I was thinking about originally, but since I dont know much about networks from inner side, I thought thats a separate issue to clusters.

Quote:

Original post by Anonymous Poster By the way - does anyone think it's worth me writing an article on this stuff? I could do a followup to "How to build an MMOG" (I think it's in the FAQ, or else google for it as a Stratics article) which actually contains practical numbers (like the staffing estimates), actual links, resources, etc. A couple of people (including two publishers) asked me to write a *book* on this topic, but I simply don't have *that* much time at the moment...

A book is a mammoth task. Article is achievable more easily. It would be much appreciated.

Quote:

Original post by hplus0603A game like EverQuest runs about 100-200 players per piece of hardware (at least at the time of shipping -- when a 1 GHz CPU was the top of the line). They have maybe 50 zones in the game. They could put very heavily loaded zones onto their own machines, but put sveral zones that seldom have that many people all on a single machine to save on hardware cost. Thus, an initial EverQuest "server" (really, a world instance, or "shard") would be run by something like 30 machines.

Hm, I thought the main reason of slow-down (or low number of players handled) isnt the CPU but bandwidth. Or am I missing something here ?

Quote:

Original post by hplus0603What redmilamber and I are saying is that, with the right networking structure and game design, and today's PC hardware, you can probably put 2,000 players on a single piece of hardware -- but it depends a lot on what you want to do with the game, and thus how that informs the networking and simulation infrastructure you need on the server side. If you want all of everything, at high simulation rates, you won't be able to do 2,000 entities on a single machine, for much of the same reasons that you don't have 2,000 simultaneous active NPCs in a single-player game of Half-Life.

Itd be probably cheaper to have 2 different machines each handling ~1000 players. Besides, reliability&stability is very important here. So its better to have more hardware always. Because if one HW breaks, still at least half of people could play. But pissing off ALL customers is a Great NO-NO. But if your motherboard/CPU burns, bad luck anyway (and this happens a lot, at least to me in past-and no overclocking was present).

Quote:

Original post by hplus0603The easiest way to load balance between physical servers within a single shard is to require the user to "zone" between areas of the game, and put different zones on different machines -- the EQ model. The harder way is to let the user walk freely, and use shadow/ghost objects to hand off simulation responsibility between different physical servers. Asheron's Call did this, and many modern systems (including ours) does this. Our web site is somewhat low on details, but take a look at Forterra Systems for a few more details.

We definitely dont plan to let player go freely. If other commercial MMOGs get away with this, we wont be an exception complicating things for ourselves unnecesarilly. Though situation might be different in 2007. Still, itd be low-cost alternative to high-profile MMOGs.

"They have maybe 50 zones in the game...."

Everquest currently has 220+ zones in the game. Each zone is a separate instance running on a server. Some of the lower populated/vacant zones do run along side of other instances on the same physical machine.

Some of these zones are instanced so that more than one of them can be running at the same time (a feature of the Lost Dungeons of Norrath expansion). These instances are created and destroyed on demand by players. Get 3-72 (size varies depending on if its for a group or a raid) of your buddies together and talk to an NPC and the dungeon instance is started up. Once you're done the instance is shutdown and destroyed.

EQ also has 50 servers/shards that each run their own set of 220+ zones, so thats about ~11,000 zones running at any given time.


Here's a list of zones =)


Abysmal Sea
Acrylia Caverns
Ak'Anon
Akheva Ruins
Barindu - The Hanging Gardens
Befallen
Beholders Maze (Gorge of King Xorbb)
Black Burrow
Burning Woods
Butcherblock Mountains
Cabilis East
Cabilis West
Catacombs of Dranik
Cazic Thule
Charasis (Howling Stones)
Chardok
Chardok, Halls of Betrayal (Chardok B)
Citadel of Anguish
City of Mist
Cobalt Scar
Crushbone
Crypt of Dalnir
Crypt of Nadox
Crystal Caverns
Dagnors Cauldron
Dawnshroud Peaks
Deepest Guk
Doomfire, the Burning Lands
Dragon Necropolis
Dranik's Hollows
Dranik's Scar
Dreadlands
Drunder, Fortress of Zek (Plane of Tactics)
Dulak's Harbor
East Commonlands
Eastern Karana
Eastern Wastes
Echo Caverns
Emerald Jungle
Erudin
Erudin Palace
Eruds Crossing
Eryslai, the Kingdom of Wind
Everfrost Peaks
Feerrott
Felwithe
Ferubi - Forgotten Temple of Taelosia
Field of Bone
Firiona Vie
Freeport East
Freeport North
Freeport West
Frontier Mountains
Fungus Grove
Great Divide
Greater Faydark
Grieg's End
Grimling Forest
Gukta, Outpost of Marr
Gulf of Gunthak
Halas
Halls of Honor
Harbingers' Spire
Hate's Fury
Highpass Hold
Highpass Keep
Hollowshade Moor
Iceclad Ocean
Icewell Keep
Ikkinz - Antechamber of Destruction
Inktu'ta, the Unmasked Chapel
Innothule Swamp
Jaggedpine Forest
Kael Drakkel
Kaesora
Kaladim North
Kaladim South
Karnor's Castle
Katta Castellum
Kedge Keep
Kerafyrm's Lair (Sleeper's Tomb)
Kerra Island
Kithicor Forest
Kod'Taz - Broken Trial Grounds
Kurn's Tower
Lair of Terris Thule
Lake of Ill Omen
Lake Rathetear
Lavastorm Mountains
Lesser Faydark
Lower Guk
Marauder's Mire
Marus Seru
Mines of Gloomingdeep
Mines of Nurga
Miragul's Menagerie
Mistmoore Castle
Mistmoore Catacombs
Misty Thicket
Mons Letalis
Muramite Proving Grounds
Nagafen's Lair (Solusek B)
Najena
Natimbi - The Broken Shores
Nedaria's Landing
Nektulos Forest
Neriak 3rd Gate
Neriak Commons
Neriak Foreign Quarter
Netherbian Lair
Nobles' Causeway
Northern Desert of Ro
Northern Karana
Oasis of Marr
Ocean of Tears
Oggok
Old Sebilis
Paineel
Paludal Caverns
Permafrost
Plane of Disease
Plane of Fear
Plane of Growth
Plane of Hate
Plane of Innovation
Plane of Justice
Plane of Knowledge
Plane of Mischief
Plane of Nightmare
Plane of Sky
Plane of Storms
Plane of Time A
Plane of Time B
Plane of Torment
Plane of Tranquility
Plane of Valor
Qeynos Aqueducts
Qeynos Hills
Qeynos North
Qeynos South
Qinimi - Court of Nihilia
Qvic - Prayer Grounds of Calling
Ragrax, Stronghold of the Twelve
Rathe Mountains
Reef of Coirnav
Riftseekers' Sanctum
Rivervale
Riwwi - Coliseum of Games
Ruins of Lxanvom (Crypt of Decay)
Ruins of Old Paineel (The Hole)
Rujarkian Hills
Runnyeye
Sanctus Seru
Sewers of Dranik
Sewers of Nihilia - Emanating Crematory
Sewers of Nihilia - Lair of Trapped Ones
Sewers of Nihilia - Pool of Sludge
Sewers of Nihilia - Purifying Plant
Shadeweaver's Thicket
Shadow Haven
Shadowrest
Shar Vahl
Siren's Grotto
Skyfire Mountains
Skyshrine
Solusek's Eye (Solusek A)
Southern Desert of Ro
Southern Karana
Splitpaw Lair (Infected Paw)
Ssraeshza Temple
Steamfont
Stonebrunt Mountains
Surefall Glades
Swamp of No Hope
Tacvi, Seat of the Slaver
Takish-Hiz
Temple of Droga
Temple of Marr
Temple of Solusek Ro
Temple of Veeshan
The Arena
The Bazaar
The Bloodfields
The Caverns of Exile (Solusek C)
The Deep
The Forgotten Halls
The Grey
The Maiden's Eye
The Nexus
The Overthere
The Ruined City of Dranik
The Scarlet Desert
The Tenebrous Mountains
The Twilight Sea
The Umbral Plains
Thurgadin
Timorous Deep
Tipt - The Treacherous Crags
Torden, the Bastion of Thunder
Torgiran Mines
Tower of Frozen Shadow
Tower of Solusek Ro
Toxxulia Forest
Trakanon's Teeth
Txevu - Lair of the Elites
Unrest
Upper Guk
Uqua - the Ocean God Chantry
Veeshan's Peak
Vegarlson, the Earthen Badlands
Veksar
Velketor's Labyrinth
Vex Thal
Vxed - The Crumbling Caverns
Wakening Land
Wall of Slaughter
Warrens
Warsliks Woods
West Commonlands
Western Karana
Western Wastes
Yxtta - Pulpit of Exiles



-=[ Megahertz ]=-

Quote:

VladR: Because if one HW breaks, still at least half of people could play.

You need to figure out how you scale to multiple machines. For reasons of locality, you find that splitting by geographic area is usually the best performing. So, in this case, if one machine goes out, those parts of your world that are served by that machine, will be inaccessible. Unless you do like City of Heroes and instance the common world zones, too, based on load.

If you want something that's existing, and in production, there is the engine behind Ryzom: NeL. I'm not terribly impressed technically, but it's open source, has shipped one title, and you can start developing content right now.

Quote:

redmilamber Does anyone think it's worth me writing an article on this stuff?

I think that would be great. The more understanding there is for the range of options, how they're influenced by product design, and what they mean to your technology investment, the better.

Quote:

Megaherz Everquest currently has 220+ zones in the game.

I was talking about back when they started in 2000. I agree they have more zones now, and they probably are running it on more modern hardware than they did five years ago, too :-)

"I was talking about back when they started in 2000."


I had a feeling you were, but I threw the numbers out there anyway.

Let me also add a few more things. The most populated EQ zone is the Bazaar which allows for people to set themselves up as traders however they must stay logged in to sell their wares. This zone typically has 300+ people in it and I've seen as high as 500ish.

One optimization that was made to support this high volume of users in the zone was that if a player isn't moving they're not sending updates to the server. I'd be willing to bet that if everybody started moving at once, it would crash the zone.

Another factor is just how densely packed people are in a zone. Some raid zones in EQ would bog down significantly if over 120 people were in the same general area, which was common when more than 1 guild was raiding there. However were you to split these guilds up and have one on one end of the zone and one on the other, things would improve a bit.


-=[ Megahertz ]=-

Quote:

Or there's the grexengine way, which is a lot more efficient than ghosting (AC and most others of that ilk have massive synchronization overhead keeping the ghosts in synch). Maybe you use similar tricks to avoid the synch overhead, but still call it ghosting? In particular, there are some real nasty worst-case scenarios for ghosting/synch problems that I believe I've seen in action on several of the 50k+ MMORPGs causing massive (measurable in fractions of a minute or more!) lag.

Our system runs real newtonian physics for every object, at 30 steps per second, using 4th-order runge-kutta integration -- it's a "distributed simulation," not an "RPG". The networking protocol is tightly integrated with the physics simulation. This means that if anything actually fell behind, it plain wouldn't work. This in turn means we have to limit the absolute total density of objects in any one area, or step down to a simler model when the servers detect overload.

Because the system is designed to let you play over a 28.8 kbps modem, the networking consumption of a single object is miniscule.

The benefits we get include server-authoritative, zero-perceived-latency, low-bandwidth, high-fidelity, secure physics. We can guarantee that any two objects that the simulation says should be able to interact, CAN interact with function-call latency (i e, nanoseconds). The cost is a finite upper limit on population density. Other systems that don't limit density, instead scale back the simulation when things get too complex, or they disallow interaction between objects, neither of which is the right choice for our platform's needs.

I wrote a white paper on how it all works, which I wish I could post here, but marketing wants an NDA to release it :-(

You are correct: We are fully deterministic everywhere, and we have very fast (and fully predictable) hand-over. It's a hard problem to solve -- hence, we spent a lot of time on it, and charge money for our solution. Our platform solves a lot of other problems too -- it's a fully functional, modern, virtual world / distributed simulation toolkit, complete with rather long list of features deleted.

I think I can predict your next question, and will attempt to answer: There are, of course, cases where we are not 100% deterministic on all clients, such as under certain packet loss conditions, and when low-latency interaction results in your forward-predicted shadow copy of yourself being wrong. In the case of interaction with latency lower than one-way transmission latency, our solution makes sure that the actor (e g you shooting) sees the right thing; everyone spectating sees the same thing, and the victim (e g me getting shot) getting a correction which we design hiding mechanisms for depending on the actions. We find the ability to trade off latency versus accuracy for the "victim" being very useful, for example.

To the original poster. You might find these links handy. They are written by the creator of Eternal Lands MMORPG and have some nice hints of what can go wrong and what really paid off etc:

A Beginner's Guide to Creating a MMORPG
Eternal Lands' MMORPG Postmortem: Mistakes and Lessons, Part I
Eternal Lands' MMORPG Postmortem: Mistakes and Lessons, Part II

The thing that stands out from those links is this collection of important points:

1) you're going to get it wrong initially, so swallow your pride and promise that everything will probably change at some point
2) listening and reacting is the basic cycle of development, and absolutely necessary
3) that's especially true when it comes to things that detract from your ability to grow your subscriber base
4) some changes for the greater good are necessary, even if people complain, so you have to have the guts to make them

Quote:

EQ didn't use clustering

That would depend on how you define "cluster". You said something like "bunch of servers with LAN-based interconnect" which is a fine way to build a cluster. Do you have to use MPS, or Wolfpack, or any of the other OS-level clustering technologies or APIs, for it to count as a "cluster"? Not in my book. In fact, sometimes, it's more efficient not to.

Quote:

Plenty of games developers mis-name things because they failed to do adequate research

When you run these games operationally, you will notice that USERS name things FOR YOU. You have the option of trying to re-educate your users, which is futile (I know; I've tried :-) or just accept what your users call things to make communication possible. To a user, "www.amazon.com" is "a server," just like, say, "Brell Serilis" is "a server". Do we stay in the ivory tower, or do wo communicate the way our users do?

Quote:

Original post by hplus0603 When you run these games operationally, you will notice that USERS name things FOR YOU. You have the option of trying to re-educate your users, which is futile (I know; I've tried :-) or just accept what your users call things to make communication possible. To a user, "www.amazon.com" is "a server," just like, say, "Brell Serilis" is "a server". Do we stay in the ivory tower, or do wo communicate the way our users do?

I totally agree. Any non-elitist who has real-world experience would know and accept this. I work in an office of 115 computer-ignorant (more or less) people; all of whom use at least one piece of software I have written. I learned very quickly the importance of communication. Sure, there may be some lock-and-key developers who are assigned a task, go into hiding, then complete the project having never had to deal with a real user. I think these people are at a disadvantage to us accessable developers. Communication is paramount in the software development industry. There is no advantage in use confusing terms when a more simple one would suffice; other than to simply sound smart.

database is almost always a bottleneck of big applications with many users / conns. At low user / conn, you may not get aware of that problem, but if the usage becomes higher, then its almost always the db which first begins to cry.

The server for frontend (e.g. webserver) are much less critical, as they can be duplicated if the load becomes too high, and we went up from 5 synced server in last year up to 12 of em this year. And herefore you dont even need high end servers, just some duals with 8 or 16 gig memory and raid is always enough for each upgrade.

But the main database, where everything is stored, cannot be just "copied" to reduce its load, as you have then the problem to keep the databases synced, and thats very difficult. Therefore we use a sun cluster running solaris on it, and that currently fits our needs quite well with a average load of < 0.5 even at high peak times.

Sometimes if the database gets too slow, it helps to keep track of the database/tables which are mostly used, and move them to ramdiscs. With this we got some very significant performance boosts at our databases (e.g. the geo databases, which are quite small, but got a lot of entries, and are access much more often then others).

Caching is not usually the right solution for database load problems, because caching makes things like globally consistent commits much harder. Instead, you should design your system to not need to hit a single database for everything. Our system scales the database instances horizontally, just like simulation, so we add N SQL instances for M users, and K simulation servers for the same M users. If you didn't have this scalability, and put all the data in a single database, you would run into a bottleneck sooner or later. So the solution is not to :-)

When it comes to "servers" vs "clusters," I've given up on trying to correct that terminology. 90% of the professionals in the business, and 100% of the customers, say "server" when they mean mean "shard", and I just translate to "cluster" in my head. This has been established usage for over five years, so it's not going to go away in six months. It's actually fairly common for a new application of some old technology, to spawn new, slightly different, use of terminology. For example, game developers think "height map" when they hear the word "terrain"; military simulation types think "buildings, trees, and burned-out cars" (which game developers would likely call "levels").

You can feel free to use the words "cluster," "machine" and "shard", because those are unambiguous terms. You might also continue to say "server" when you mean "a single machine," but as I'm sure you notice, you'll end up spending more time than it's worth qualifying that statement. When it comes to language, it doesn't matter if you're "right" -- because there are many "rights." What I think matters is finding common ground for unambiguous communication.