Skip to main content
gamedev.net
🔒 Locked

Server Migration

Started by oliii Sep 13, 2006 at 4:23 AM 7 replies 2k views
Original Post
oliii
oliii
I'm about to start work on server migration for a game. It's all Socket stuff (plus some XBox Live I will worry later). so far, I have a P2P network of machines, and one of them acts as a 'server', mostly to synchronise game flow, and update game entities outside the players' control. A bit like a referee. That will possibly change and entity ownership will be spread out among clients. But it's another question for later. OK, so server migration... all players are at least connected to each other. Migration will occur when the server drops of quits the game. Assuming the server quits for now, all clients are obviously notified, and one of them will have to take control of the game flow. The problem is deciding which one, and all clients have to agree on who will be the next server. I've came up with this... at regular intervals (say, 10 seconds), the server sends a sorted list of the clients, in order of which they should be the server. With that list, also comes a sequence number, or a timestamp (a SqN should do). Say, SqN(0) : list is A, D, B, C. then 10 seconds later, SqN(1) : list is D, B, A. if server relinquishes his control, then client D should be the new server. If D drops or quits, it will be B. And so on, and so forth. Now, this list is sent (reliably or not, reliability dont seem that important), and each clients receives it. Supposing a couple of clients don't receive the list when the server drops or quits (communication cut short), this is where problems occur. They might have a deprecated list and think client A should be the server. ok, solving the problem... Once clients detected server drops or is no longer there, they go into 'migration state'. When entering the state, they broadcast their list to each other. By doing that, they can also check that everyone is connected to everyone else (no P2P clients have dropped). If it's the case, you can kick the incomplete ones (or sort them out). They do that until they all received the latest list, with the highest sequence number. All predicts that machine D is now the new server, machine D signals he is now in charge, all clients re-synchronise with machine D's state, and we carry on.... Also, C is no longer part of the network according from the last words from the server. C gets kicked out. This is not taking into account when clients start dropping and timing out. But all the rest is a regular P2P problem. So, any comments? Forseeing any problems? Anyone done server migration before? How did they tackled the problem overall? Hope explanations make a bit of sense. [grin]
Everything is better with Metal.
oliii
oliii
Also, note that it's not necessary for the server to quit the game to migrate. For example, if the server gets really laggy (downloading pron or something), it can migrates and join the clients queue.

Everything is better with Metal.
Dave Weinstein
Dave Weinstein
The best way to handle server migration is not to do it.

Either go for a true peer-to-peer architecture, in which case there is no server, or go for a client-server architecture in which if the server drops, the game is over.

Implementing server migration is a tricky, and incredibly error prone activity with a fairly low reward if you do manage to get it right.
oliii
oliii
I don't decide the game features and specs unfortunately. It's part of the U.S.P. of the game. God help me!

I am considering full P2P. That might an easier way out. Although there are some problems associated with it.
Everything is better with Metal.
Sneftel
Sneftel
Why would the migration order ever need to change?
hplus0603
hplus0603
Network connectivity changes over time, so migration order can change. The list might be sorted by lowest ping time from the old server, for example.

I think a simplified algorithm would be to send the server list every so ofter (with a sequence), and when a drop is detected, each client decides who they think is the server, and broadcast that machine's name, and the sequence number they got that idea from. The latest sequence number wins.

So, as client A, I might have a really old list saying "B" is the server, at tick 100. When I detect the server is gone, I broadcast "100:B" and I start believing that B is the server. However, someone else will have a newer list, saying "C" is the server at 101. When I receive that packet, I will update my idea of who is the server on A.

Note that, for server migration to work, all clients need to have the full game state (else they can't become servers after a break in communication).
enum Bool { True, False, FileNotFound };
oliii
oliii
yep that's the idea. I'll try that. Publishing a list instead of just the server would also insure that, if the new potential server drops (bored of waiting, ...), you can switch back to the first valid client in the list.
Everything is better with Metal.
Dave Weinstein
Dave Weinstein
More sand in the gears for this feature:

It is entirely possible that in a multiplayer game, everyone has a good connection to the original server, but there are machines which cannot directly contact each other. It is certainly often the case that while machines B and C have excellent connections to machine A, they have poor connections to each other.

It is also entirely possible for machines D and E to lose connection with server A, and decide that B is the new server, while B and C are talking to A just fine, and have no idea why this transition is happening. Also, what happens when suddenly D and E can talk to A again?

Server migration is hard enough when it is "This is Server A shutting down, everyone go to machine B as the new server". When you are designing it to deal with network fluctuations or a machine simply shutting down, you have a real headache on your hands.
oliii
oliii
Yep, I'm starting to feel the pain...

I'm always tryingto assume nothing, especially when it comes to network links.

There are three major problems I can see

1) Detecting when to migrate.
2) Who to migrate to.
3) the actual migration.

1: is relatively easy. When you loose the connection to the original server, you basically wait until a new server is ready to take over (or timeout).

2: this is the hardest part. Due to the crazy interconnectivity between peers, and who drops, who doesn't...

a) The server will broadcast the list of clients, sorted by connection quality (and game state, don't want to migrate to a host that is in the lobby when a game is in progress) every now and then.

[cpp]
struct C_ClientList
{
U32 m_uSqN; // sequence number
U16 m_uNumHosts; // number of hosts in list
C_HostGuid m_auHostGuids[eMaxHosts];
};
[/cpp]

b) When a client drops his connection to the server.
-> flagged as 'LostServer'.
-> Sends a packet 'MigrationData' packet. contains :
- latest Migration list received from the server.
- List of peers he is connected to.
- Who he thinks should be the server.

[cpp]
struct C_ClientMigrationData
{
C_ClientList m_xMigrationList; // sent by the server.
C_ClientList m_xPeersList; // who am I connected to.
C_HostGuid m_uServerGuid; // the machine I think should be the server.
};
[/cpp]

c) when other Peers receive the migration data packet from another peer, they update their migration list if it's older (SqN check).
d) if they are also in 'LostServer', they check the 'PeersList' and 'intersect' it with their own 'PeersList'. All peers not in both list have their connection severed. That 'PeerList' then gets broadcasted.
e) By process of elimination, all remaining peers should be eventually be connected to each other, have identical lists, and agree on who will be the next server.
f) That new server sees everyone is ready (everyone has the same data), and starts the migration process towards himself (XBox Live new session key, and so on...).


probably loads of potential problems, but i'll try that :)
Everything is better with Metal.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.