Distributed Gaming

Hi Mister Abdulwahed

Great article, I hope to see you write another article.

bye

This article appears just on the right time. My friend and I were discussing whether we should include networking/multiplayer in our next game. This article is a great resource.

Maybe a tutorial in network programming would be nice too.

Glad you like the article, guys. Thanks!

Below, I am posting an e-mail I received with some very good questions about Meesha and Distributed Gaming. Thanks to Lloyd for his queries!

-Omar

-----Original Message-----
From: Abdelwahed, Omar
Sent: Monday, June 02, 2003 11:28 AM
To: ''Lloyd''
Subject: RE: Meesha Network

No problem, Lloyd. I appreciate the feedback!

Good questions... let''s see if I have decent answers:

1. I used the MMO example, because it is very topical today. However, I probably did a poor job of discussing the design considerations when making a game "Meesha-aware". One of the decisions is simply what resources does one identify as a candidate to distribute. Not absolutely everything has to be distributed. This is up to the developer(s). In addition, there''s the granularity of the resource to consider. To your point, I would not suggest to create a distributed resource that tracks a single NPC. However, a resource that tracks all NPCs would make sense. So, as an encapsulated resource, all redundant nodes would be responsible to maintain state with the peers they have demeaned (dynamically) most reliable. This state could include decisions around physics to whatever degree is considered reasonable. Note, Meesha does not dictate any of this. It merely allows the identification, publication, and subscription to these resources while maintaining quality checks. One still needs to make appropriate design decisions.

2. Propagation of state is something I''ve spent a great deal of time on... and continue to today! Currently, Meesha allows every peer to have it''s own unique view of the network. A peer can than switch to other peers that serve the same resource. Switching occurs based on certain metrics, for example, response time. If one peer starts to experience less performance than expected by another peer, it can choose to switch to a better service. I have yet to prove, though I believe, that state is more efficiently spread in this manner than in a client-server model. I have to admit that this is based more on intuition than anything else. However, I believe that there''s a connection with this model and the notion in Information Theory that deals with information as a type of energy and the Second Law of Thermodynamics (ie: Entropy). (Now that I''ve come off as completely crazy, I''ll give you these links from MIT to follow-thru on my thoughts:
Presentation on Information and Entropy: http://www-mtl.mit.edu/%7Epenfield/pubs/complex-99.html
Full MIT class offering, includes notes: http://www-mtl.mit.edu/Courses/6.095/

Thanks again for reading my article! I hope it helps and inspires!
-Omar

-----Original Message-----
From: Lloyd
Sent: Monday, June 02, 2003 6:49 AM
To: omar.abdelwahed@bestbuy.com
Subject: Meesha Network


Hey Omar

Just read your article on GameDev.Net about your Meesha network system.
You''ve definately got some interesting ideas there, but i''ve just got a
couple of questions if you don''t mind

Continuing your example of a MMO, if every node is equal how do you run the
physics/ai on a resource in this system?(assuming a resource is a single NPC
character?) surely one of your "nodes" still needs to be the authorative
node for a particular resource? otherwise who decides whether a bullet hit
an object or not? what happens if the authorative node loses connection with
the network?

What about propogation delay between nodes? i.e node 1 updates and objects
position, which propogates to node 2, to node 3, to node 4, etc etc, all the
way down to node 10. In a traditional client server architecture the update
would have reached node ten at a similar time as it reached node 2?

Thanks for your time
Lloyd

quote:

---

Original post by Anonymous Poster
Re-posting something I sent in e-mail:

While this article shares a lot of thoughts with an article series from
a few years ago by Crosbie Fitch (www.gamasutra.com might still
have it online), it seems to ignores the most important part of
distributed gaming: latency.

Take a game like Planetside for example. If player A drives a
tank, which player B rides gunner in, and shoots a bullet
entity (C) at a third player D, then the distribution mechanism
you suggest has very little chance of accurately or
authoritatively resolve the outcome. Because of the latencies
involved, all these entities will be running somewhat out of
sync with each other, and must constantly strive to get back in
sync.

Now, to this, you add a very specific action that has very
tight tolerances for position, aim, velocity, etc, and requires
messaging across all four entities (assuming the tank A must
agree that there''s ammo left). Draw it out on paper and try to
figure out if the current internet, or indeed any near-term
internet, is capable of serving this example.

Here''s a hint: I don''t think the answer is "yes" :-)



Another issue, which I didn''t bring up in e-mail, is the
consistency of authority. To make an authoritative decision,
the agent needs to have up-to-date information about its
surroundings. If you run a simulation which spreads different
entities that closely interact, far apart in geography, then
that seems to be a formidable challenge, and I don''t see
anything in this approach which tackles that? (in a clustered
server, this is hard enough already!)

---

I think what''s causing confusion is the assumed choice for redundant resources that are distributed. This is what I tried to clear up in my last post (and what was obviously unclear in my original article). Again, it is up to the development team to make appropriate choices for game resources.

In your example, you''re assuming that each player is a unique resource/peer that needs to synchronize state with all the other players/peers. (This would look much like the LAN peer games in Quake, for example.) As you point out, this would indeed cause many issues when used for a network game across the Internet. What I might suggest instead is making "player servers" as one type of resource where there are many distributed, redundant nodes. Or, perhaps, I might even make several sub-types of resources each of which handle various aspects of player state. Then I would scale each resource as necessary to handle issues of synchronization.

See, that''s the real key here. In Distributed Gaming, one is able to slice up resources as one see''s fit in order to meet the demands of the game. How one decides to implement application-level resources is not specified in any way by Distibuted Gaming or the Meesha implementation. This allows a very high degree of flexibility in design while still giving a core framework that even allows one to add new resources to existing games.

Another way of looking at this is that most networking solutions I''ve seen in gaming to date attempt to efficiently send packets of information given the nature of the Internet. Distributed Gaming does not attempt to re-hash this approach. Instead of dealing with packets of information, it deals with more efficient ways to manage state. There is a difference here.

I think what is very clear to me now is that I need to write a follow-up article!

And, please keep sending me feedback! It is most appreciated!

-Omar

I''d be interested to see how you work up your "trust metrics". From my experience with MMOs, there really isn''t anything to distribute, because you simply cannot trust clients to have authority in any single area. Distributed AI or NPC actions? No, it''s easily exploitable. Distributed Events system? Nah, you don''t want me injecting/modifying events do you? Also, how many network connections are you planning for each (client-side) peer server to support? You''re running into issues of trying to rely on the client to run a particular OS, with a particular level of performance. It just doesn''t seem feasible.

For LAN-based games peer to peer is great, as is this distributed model you describe. But for WAN games such as MMOs, I don''t think we''ll be seeing anything running on the client-side for a long time.

quote:

---

Original post by fingh
I''d be interested to see how you work up your "trust metrics". From my experience with MMOs, there really isn''t anything to distribute, because you simply cannot trust clients to have authority in any single area. Distributed AI or NPC actions? No, it''s easily exploitable. Distributed Events system? Nah, you don''t want me injecting/modifying events do you? Also, how many network connections are you planning for each (client-side) peer server to support? You''re running into issues of trying to rely on the client to run a particular OS, with a particular level of performance. It just doesn''t seem feasible.

For LAN-based games peer to peer is great, as is this distributed model you describe. But for WAN games such as MMOs, I don''t think we''ll be seeing anything running on the client-side for a long time.

---

Just to note, the "trust metrics" are not based on the type of information that makes up a particular resource. The metrics are completely independent of the actual data that is represented. We can build metrics that are generic to any information (latency threshold, for example) and use these metrics to measure the overall value of the information. Again, refer to this MIT link to see what I''m talking about (especially, if you don''t think I''ve made a good case for this):
http://www-mtl.mit.edu/Courses/6.095/

Also, whether or not you decide to create a resource to handle NPCs or events is one design decision. Whether or not you then allow your players to become nodes for these particular resources is yet another decision. If you''re worried about exploits, then don''t. You could just as easily manage all the nodes as independent machines running in various hosting facilities around the globe much like today''s MMOs. The difference will be that you can easily scale these individual resources whereas a typical MMO has to scale everything that resides on a given machine.

I think what is needed here is an actual demo for people to try out. I''m working on one right now as well as updating my web site with all pertinent information on Meesha and Distributed Gaming. Of course, this is all happening in my so-called "free time".

I''ll post an update soon! Keep the questions coming!
-Omar

quote:

---

Original post by Anonymous Poster

quote:

---

one is able to slice up resources as one see''s fit in order to meet the demands of the game

---

I''m seeing no evidence of this, though.

---

Yes, fair enough. I need to get my demo out to show people!

quote:

---

Note that for scalability, you HAVE to be able to split the SAME kind of resource over multiple machines, or you will be bottlenecked on whatever the limit is for a single machine to serve the most expensive atomic resource.

---

I totally agree! However, this is not the case of most (if not all) online games. "Asheron''s Call" seems to be the notable exception. I believed they originally used DCOM to solve some of their scaling needs. (Gamasutra has an excellent post-mortem on the original launch.)

Note that the way scalability is generally handled with most online games (and with a fair amount of websites and other Internet applications) is with redundant servers behind load-balancers. That is, they are handled not by intelligent resource handling via brokers or other software architecture, but by "big iron" and some damn expensive Cisco devices. This is not the same thing as I describe by any means.

quote:

---

Once you do that, latency and loss becomes real pains in the neck. You still would like everyone who did not suffer a lost packet to see the same thing happen in the world, and only bring the out-of-sync hosts in-sync. However, while a host is out of sync, what if it''s the "master" for some resource, so, by definition, everyone else is out of sync?

Also, splitting a single player in multiple nodes is very seldom the right solution. Suppose you had one "stat server" and one "position server" and the player pressed "jump". Instead of now involving the player machine and the single server for that player, the "position" server now needs to query the "stat server" for information to resolve how far/high the player jumped. As soon as you leave a single machine, your general cost (latency, overhead, etc) skyrockets with several orders of magnitude.

To provide a compelling service, you HAVE to synchronize players with some number of other players, and to scale well, you HAVE to support some of those other players being authoritative on some physically distinct server. Note that EverQuest doesnt'' scale well: all players in one Zone shares the resources of a single server, and thus it won''t scale a single Zone (which has static limits) further than what a single machine can get you. In a sense, EverQuest is not only sharded on a per-world-instance axis, but it''s also sharded on a per-location instance, with watertight boundaries between locations, which makes the design simple, but massive shader scaling impossible.

---

Hm... I swear we are arguing the same thing here.

But let me present a different analogy to see if that helps. Forget MMOs for a second. I argue in my article that the difference between E-Commerce and Online Gaming is strictly the type of transactions that take place and the granularity of time required to make a transaction successful. If you take this for granted, then we can see "massive, online" applications today that handle hundreds of thousands of simultaneous "players" using similar approaches to what I describe as Distributed Gaming. Indeed, if it were not for the fact that a game transaction needs to complete within milliseconds (rather than seconds in e-commerce space), I would have just utilized the very stable application servers that exist today to create an online game, MMO or otherwise. Not to gratuitously plug my employer, but you can take a look at BestBuy.com to see this in action today.

Moreover, if you take for granted that you can analyze data to determine its relative "value" without having to know what the data actually represents than the only difference between E-Commerce and Online Gaming is the relative transaction times. This would generally suggest the need of a more roboust approach to complete transactions quicker while maintaining quality. This is why I began designing my approach 3 years ago. Today''s app servers didn''t provide the efficiency I needed.

-Omar

[quote]Original post by cab
Hi Mister Abdulwahed

hai sir how are you, i need some help from you, i am writing an paper on distributed gamming software, i read your paper it is great. but i want complete details on distributed gaming software, can you please mail me all the web links and books where we can find the material on distribute gamming software.

[quote]Original post by cab
Hi Mister Abdulwahed

hai sir how are you, i need some help from you, i am writing an paper on distributed gamming software, i read your paper it is great. but i want complete details on distributed gaming software, can you please mail me all the web links and books where we can find the material on distribute gamming software.