P2P multiplayer on a massive scale.

Nothing new about trying to design P2P for something other than file sharing network control...but no one has implemented a good library that I know of. I would like to see one.
As far as ideas for P2P there are tons in the network literature. Not too sure about yours though. Bandwidth and processing power are not always the limiting factors. Usually most people optimize the delay or hop count because these are more important for realtime traffic. If you want good ideas look up overlay networks..that is really what you are forming when you do P2P...unless your network is real small then you could have a mesh.

This topic was real hot in the literature, but not much has come of it.

The main problem with P2P game setups (at least in my mind) is the issue of making sure that none of the clients are cheating. Usually in a server client setup this is done by having the server really do everything, always making sure that the client is passing good data and that the client only recieves the data it needs.

Once you have the data distributed across the network on clients, it becomes a lot more difficult (if not impossible) to do this. How do you make sure that the player isn;t magically teleporting across the part of land they are responsible for? Or that they didn;t just turn a rat pelt into a precious gem?

Of course, that doesn't mean this can;t work with a smaller scale, where everyone knows each other and trusts each other. But go beyond that and you need to secure your data somehow.

Isn't that kind of what GameSpy Arcade (Or Quakeworld) did?

Couldn't peers check each other periodically?

Synchronization would be your most difficult part. If you have several different "pseudo-servers", how do you keep the data between them even. It's difficult enough to do it when the servers are on the same rack. Doing it while they're 100 milliseconds apart would be bruital. Expecially if you hit high traffic and you get a netsplit of some sort.

Doom was actually a p2p mesh, but it didn't scale so well, which is why id uses the client server model now. That's the largest p2p game I know of.

Quote:

Original post by Grizwald Couldn't peers check each other periodically?

That is simple to say, but how? Are two clients computing the same part of the world to provide redundancy? If they don't agree, which one is corupt? Should you use 3 clients for each piece of data? What if two of the clients are cheating?

If I am a company who makes money by levelling and selling characters, what is keeping me from having all of my workers keep logging on and off until they have matching parts of the data? Then they could cheat in sync, and roll in the dough.

There are definately a lot of issues, and also a lot of possible solutions. All of the solutions I can think of though have quirks. For example, you could split the data up a lot, really fragment it, so that any single client doesn;t really have enough to cheat with. But then you have to create some way to access all that data from all of those clients, just so a client can sell a loaf of bread.

Hmm.. the main reason for introducing a p2p MMO system would be saving costs on a server. But what if a server would be used not for everything in the game, but just to in some way ensure that no-one is cheating (that'd be useful if it didn't take as much bandwidth as normal game server would). This way the server would be analogous to BitTorrent tracker - bittorrent is highly scalable without much tracker traffic.
(Not sure how the checking would work, though. Any ideas?)

How about some kind of random auditing to catch cheating. Each peer could at any time query the state of another peer and all of the transactions (ie. trading items with other peer and such) made by that peer. This would be done in a random fashion and cross checked against data from other peer for consistency. This would be best done by a server, but the load would be small. Potential cheaters would then get flagged and could be examined more closely.

If for example peer A is queried and claims to have gotten a Sword +10 from peer B, but B doesn't say that he gave a Sword +10 to A, then A is likely to be a cheater. This wouldn't really work for cheats that just enhance abilities, like speeding up movement.

Yeah... The internet, in its current state, is just not fast enough. And with P2P, there are too many considerations. You can't trust the hosts not to cheat, you can't trust them to be always fast, and you can't even trust them to stay online.

Honestly, I feel its just "not gonna happen". And when the internet is fast enough for this to be possible (everyone has a 100Mbps connection), then everyone will also have the bandwidth to run a central server.

Arena Net has a good article concerning P2P. And its actually been used before in "school project" type MMOs (look for a guy named Wei Xu, he's a guy from UPenn who's been in some groups that have dealt with implementing it). Speed, breaking up the server tasks, etc. are easy to implement (everyone pretty much becomes a mini server dealing with their own character). The hard part is cheating. The best solution I've seen are the ones similar to Guild Wars, you only play with the people you want, certain data is server centric while others are not, etc.

What if each server checked itself, say with a hash (to check crucial data and binaries) and then verified it with the central peer tracker?

Quote:

Original post by Grizwald What if each server checked itself, say with a hash (to check crucial data and binaries) and then verified it with the central peer tracker?

Checking your own binaries won't work - what prevents you from sending fake hash?

I myself gave thoughts to that idea some time ago. Superpig posted something similar one or two months ago in his journal, those interested can check it out.

One important point which i haven't seen mentionned, is (in addition to the problems already reported) that this P2P solution will not automagically reduce your bandwidth costs to zero.

Think about it:

1. Not all players are equal faced to bandwidth (56k anybody?).

2. The upload bandwidth of a player is generally very low compared to the download one. You might have an ADSL connection of 4 MBits+ download, but if your upload is limited to 256 KBits, the limiting factor will be this last one.

3. As such, this P2P model only works when you can find some groups of players, where all players in a group have good connection conditions.

4. And obviously, since the server is out of most transactions, the latency between the players become an important parameter. What if 3 players are interacting; two of them have a good latency (50 ms), but the latency to the third one is very high (350 ms) ?

So, you need to design your system to support extremely diverse bandwidth/latency combinations. If you thought it was hard with a centralized server, expect some real troubles now.

Yet, i do believe it can be done, and reduce a lot the bandwidth required by the server. I might try to implement it in the more or less distant future.

Y.

Quote:

Original post by FlowingOoze How about some kind of random auditing to catch cheating. Each peer could at any time query the state of another peer and all of the transactions (ie. trading items with other peer and such) made by that peer. This would be done in a random fashion and cross checked against data from other peer for consistency. This would be best done by a server, but the load would be small. Potential cheaters would then get flagged and could be examined more closely. If for example peer A is queried and claims to have gotten a Sword +10 from peer B, but B doesn't say that he gave a Sword +10 to A, then A is likely to be a cheater. This wouldn't really work for cheats that just enhance abilities, like speeding up movement.

Then all you need to defeat this system is for both A and B to be cheating.

Just a few ideas from ym thoughts on this:

1) When splitting up tasks, do so at a very fine grained level. For example, a client should not be "Inventory for 1000 Players"; ;instead it could be "Iventory for 10 players, commerce for the bar in Nagshead, and player movement on Acre G-43". That way any single player has difficulty doing large scale cheats because the client has control over only a small bit of data.

2) Never give a client control of its own player's data. Very obvious, but it instantly makes it much more difficult to cheat.

3) Distrtribute copies of the same data to IPs that appear to be geologically seperated. Although these IPs don't give location precisely away, it should allow you to put one copy of the data in Texas and the other in Virginia. This lowers the chances of the multiple clients collaborating.

4) Instead of having two clients have complete copies of each others data, also fragment who clones what. For example, if I have data A, F, and J don't protect it by giving another client A, F, and J. Instead, split it up among three clients that have (A, D, R), (C, F, K), and (I, J, Z). This way if clients are collaborating they still only have access to a small bit of the data on their machine.

5) In the case where a discrepency is found, check all clients affect by the discrepency by comparing each piece of their data against all copies of the data on all clients. Unless there is a very large and pervasive (50% of players) network of cheaters who work together, this will point at any client simply modifying lots of the data it has access to. Since the N data fragments are spread across at least N (or 2N if data is triple stored) clients it is more difficult to get the required number of collaborators with the same data.

It is technically impossible to have a large scale p2p (on the Internet) game, because of bandwidth and latency issues. Plus, it makes absolutely no sense, the client/server paradigm works great.

Quote:

Original post by Raduprv It is technically impossible to have a large scale p2p (on the Internet) game, because of bandwidth and latency issues. Plus, it makes absolutely no sense, the client/server paradigm works great.

Not at all. I know there has been at least one game that could count as an MMO that was essentially a P2P network. It involved every client have a patch of land that it owned, and releasing your own AIs into it. The AIs could then be teleported into other clients through a P2P network formed by them. As they spread through the network they reproduce and compete with the other AIs. Not your standard MMOG, but since it managed to have a 1000 players or so I think it counts.

Even for a more standard MMORPG, I don't think it is "technically impossible". At the very least, there are possible reqards to a hybrid type system. I heard that WoW streamed (a la BitTorrent) patches to clients from other clients, at least a proof of concept of P2P.

Once the security issues are dealt with there are certain cases where a hybrid system comes to mind. Any sort of operation that is a little more lag tolerant could be offloaded to the P2P network. For example, commerce systems can usually deal with a a little lag and can easily be "dealt with" since even a real simulation of any transaction would take longer than the transmission. As already mentioned, patches and levels can be distributed via P2P. All non combatant NPCs could probably be put on the P2P network, and so could all messages through the chat system. I think these are enough to make at least a dent in bandwidth, and they are all pretty easy once the network is there.

There are other more difficult possbilities too. If a player is off on their own, not engaged in battle, then it could make sense to manage that players movements and inventory on the P2P network. Crowded cities and combat situations would then be handled by the more reliable servers, which don't have to bother with lag tolerant situations.