Recommendations for Authoritative Network Model

Those are actually pretty good resources.

What, in particular, do you want more information on? Is there something you're trying to do right now that you're having trouble with?

Those are actually pretty good resources.

What, in particular, do you want more information on? Is there something you're trying to do right now that you're having trouble with?

Thank you for your reply.
My issue is purely conceptual, and not in terms of implementation within a code framework. I am fairly used to socket programming, though I use Python.
I am not looking for anyone to start recommending libraries or to use another language. Not that I assume anyone will, but i find often on #Python people quote Twisted regularly, and the game engine I am using runs on Python.

Here is my documentation of progress:
http://blenderartists.org/forum/showthread.php?255181-Authoritative-Server-Architecture-Prototype-Do-not-comment&p=2120983#post2120983
It may prevent you from seeing the attached images, for which i apologise.

I would like if anyone could offer some advice if i have made any errors.
My main concerns relate to state synchronisation and lag compensation:

1) How would you perform delta updates? My current understanding is that you would determine the changes since the last update acknowledged. If this is the case, would you send the transformations of data between the latest gamestate and the client's data? I presume you only send full snapshots if severe packet loss occurs.
Or, would you send a list of gamestates which the client runs?
2) How would you identify delta updates / full game states? Would you assign IDs to the packets, or use time stamps? I presume the former, because server and client time may differ ... unless you used game time....

In Python you cannot serialise objects, so i would use a dictionary (which is kind of similar to an array in C++).
Sorry for the long post. I find it hard to find comprehensive examples / theory / papers that relate to Python because, it is at the end of the day, a different approach (higher level language), and I'm not studying computer science!

I hope I don't sound over-thinking or misdirected. I am painfully aware that it would be very easy, as a hobbyist, to produce a setup which didn't correctly account for latency and synchronisation and I would regret it in the long run (as well as be blissfully unaware of the synchronisation aspect to networking).

May thanks to everyone!

I appreciate I'm asking a little generically. Is there anywhere else that I could find information on this topic?

I think you'll have to read some code. The source code for Quake 3, Doom 3, and many other fine games is publicly available. There are also various "MMO server emulator" projects in various states of completion around the web.
Another option is just to try it out. Write a game. Stick networking in it. Try different things, and see how it works out.
There is no "right" or "wrong" answer, just various trade-offs for meeting various design goals.

Thank you for your advice hplus!
So, I've decided that yes, i shall continue through trial and error. at the moment, I'm working on full world snapshots.
I have a few questions before i progress;

1. As i will eventually include lag prediction and so forth, how should i synchronise the ticks between the server and client? If the server keeps a record of the current game tick, how should the client synchronise itself so it can map itself to the client and run backwards prediction.
   
2. As the avatar objects that represent other clients must be stored, I was thinking of creating a separate class that manages the users on the client. This made me realise that I may as well keep some form of game state local to the client so that i can update more than just positions and orientations of the clients. Is this a bad method?
   
3. A small question; what are the typical transmit rates for the server game state, client events and the update rate for the server game state?
   

Typically, each "step" has a number, assigned by the server. The client will notice what "step" the time is when it receives data for step X. Also, the client can measure how "distant" the server is in time by measuring round trip time.

Typically, the clients and servers both keep copies of the (publicly known) game state. The role of the server is to keep the clients as up-to-date as possible with the observable state of the game.

Typically, you only send full game states every once in a while, as "baselines." Then you just forward command inputs from all the players to all the players, which lets you run the simulation in parallel on all the machines. This saves a LOT of bandwidth. The inputs per player per tick is just a few bytes; the game state might be a hundred bytes per player. For a 20-Hz game with 8 players, that would be about 16 kB/sec of "full state" data, which is more than you'd want to use. FWIW, Xbox Live games are supposed to work fine with less than 8 kB/sec of total bandwidth (64 kbps) available.

Thanks hplus. I think what i need to do is to avoid processing game state on the client until it receives a full snapshot from the server, and takes the server's current tick from that, therefore it can keep track of the ticks on the client. Should the client operate at the same tick rate as the server? Or at least be aware of the tick difference between the server and client?
Also, what are typical tick rates in the industry for FPSs?

1. Server world update (thinking about 60)
   
2. Server send world update to clients (30 times a second?)
   
3. Client sends inputs (Not sure, 30 again?)
   

Should the client operate at the same tick rate as the server? Or at least be aware of the tick difference between the server and client?[/quote]

Yes, and yes.

Your simulation "should" use a fixed step time, which means a fixed tick rate. Anything else will cause lots of problems, and not really give you anything that's worth anything in my experience (and that of many others on the interwebs.)

Your client will have communication latency to the server. The client, to provide the best experience, wants to be aware of how much the round-trip is, to be able to do some kind of latency hiding. Exactly what you do, depends on the game mechanics, and how you choose to hide that latency -- pretty much 90% of the good sauce of network game programming goes into that part.

Typical rates for physics are from 30 Hz up to 120+ Hz. Typical rates for network packets are from 10 Hz to 30 Hz. Typical rates for "full state dumps" are from "never" to "occasionally," whereas rates for "command/input updates" are typically "every frame, batched into every network tick."

Thank you again Hplus for your support.
So, I beleive I should first start "batching" packets.
I plan to store the input dumps each frame, and sending the batched packets every 2 / 6 ticks(depending on the frequency).
However, this does somewhat reduce the simplicity of the design.
At the moment, the server receives user inputs in single packets, and stores them on a class (object). The game state simply reads these and performs logic on the client each tick.
Should the server receive these batches, and spread them over each frame, so in the same way they were read over a few frames they are counted as new inputs over a few frames, OR should the server apply these inputs during the same tick - iterating over each packet, and updating the game state in turn? It seems to me that the former may be more logical to save processing on the server, but I would love a second opinion.

Latency hiding sounds like the main task, but obviously it requires a position whereby the client is aware of the age of its received game states.

With the client, I expect to keep track of the first received game state tick, then it will relate its own tick to that.
I beleive that by getting the tick rate from the server, and working out the RTT time, i could find out the current tick on the server, so that the client would set itself to the same tick number. Because they operate on the same tick rate they shouldn't fall out of synch too quickly. If they do, i could work out the rtt time, get the tick rate again etc.
Unless I should just get the tick from the server and treat that as the current tick, regardless of latency?

Then, My next question (Sorry for asking, you're awfully helpful). When the client receives the confirmation server packet of its actions, it will be old data, by at least a few ticks. Therefore, when i add in prediction etc, the game state on the client is newer (for the client) than the server packet it receives. Therefore, should i keep some form of log of all the game states per tick? Or, should i keep track of different changes in game states (e.g a direction change of a client, not just a position change if it continues along that vector), and the ticks upon which they occured, so i can use interpolation to determine what the client predicted that change to be? I can't see how else to confirm the prediction as vaild.

Thanks again Hplus, this forum is great.

should i keep some form of log of all the game states per tick? Or, should i keep track of different changes in game states (e.g a direction change of a client, not just a position change if it continues along that vector), and the ticks upon which they occured, so i can use interpolation to determine what the client predicted that change to be? I can't see how else to confirm the prediction as vaild.[/quote]

There are different ways of doing this. Logs of events is one. Continually "drifting" the client towards the "correct" position is another. Delaying client feedback until you get the confirmed events back is a third. Check out the EPIC (entity position interpolation code) for an example of how to do the "drifting" part.

Wow! I didn't realise that you wrote that!
I had seen this before when i was trawling for interpolation (yes, the concept was new!)
This looks to be interesting. So what you're suggesting is that if the client prediction is "wrong" , you simply attempt to "push" the client towards the correct data? If so that is a nice notion.

Just to clarify, i need to be able to determine if the client's prediction is wrong. However i cannot just compare the existing game state on the client, because it will be further ahead in tick time than the server (due to the rtt of the client packet). So i need to be able to look at past packets.

This is where i get a little hazy. Firstly, your pushing toward the correct data is a good idea, but;
The idea is that a client predicts the result of user input. It plays the game state based upon that. Let's say that it was walking forward, however on the server an obstacle suddenly appeared in front of the client entity. Because this wasn't present in the client's gamestate, it is a valid action, but the server denies it and returns a corrected position packet where the client bounces backwards. Assuming an average 100ms delay from client to server, it will be 200ms before the correction is received.

At this point, the client must correct itself. However, does it push the client towards the position from the server packet, which is 200ms old? Or does it use the difference between the client's predicted position 200ms ago with the server position just received(that is 200ms old) to extrapolate?

I may as well continue to apologise because i am all too aware of the irritations that come with begginners who appear to ask countless question after quesiton.

The easiest thing to do is probably to keep older states (rather than packets -- those are different things.) Keep the last state received from the server. When you receive another state, extrapolate out to "now" based on the states in those two packets (EPIC helps with this.) Now, the position you extrapolated to from those packets may be different from where the player is. At this point, you can "drift" the client towards the extrapolated state. This is simple, but it also makes players with higher ping times less able to turn quickly, as a function of the math involved.

The next step would be to remember simulated state for each tick, and when you receive a packet from the server, compare the server based state with the state you simulated for that tick, and apply the delta between those two to the current player position, and to each remembered position after the state you received. You can then discard any remembered state older than that time step. Repeat for each state received from the server. This may "snap" the player a little bit, but as long as the forward simulation and the server agree, there will be no difference.

So, here's my current understanding:

Take the server packets and store them (assume they're world snapshots for now).
When i have stored two packets, i can start to extrapolate. Before then just use client predicted.
Use epic to push the client towards the current state. (not snap)

Store the above extrapolated states, storing the tick times, (The server and client ticks are synced).
When you receive a server update, apply the delta to all the packets that were simulated after the update, and delete previous packets.
This means that when the future server gamestate is received, it "should" be close to the client game state.

That sounds like one way of doing it.

If the game state contains position and velocity, you can extrapolate based on a single state if needed. However, extrapolating based on two states does give you possibly better extrapolation quality.

Thanks Hplus. I've already got the clock synching, it doesn't use RTT. It gets the time difference between 2 packets that were sent one tick after the other, removes the time between ticks and that is the time between Server and Client. Then, it works out how long a tick lasts from the frame rate, and finds how many whole ticks are required to create a length of time just larger than the transmission delay, gets the delta (maxticdelay - connectionlatency) and waits for that duration (time.sleep). then if adds the whole tick number aforementioned to the last tick received to get the current server tick.

Good, I have also batched packets. However, should it matter to the server which tick the inputs occurred on?
Now I'm going to look at delta updates, however I need to consider velocities, so this may be a little tricky to work into.
As a side note, I'm very happy to work in this manner. It is much more flexible / efficient than client side!

I have another question! (Also, I am documenting my progress on the forum I posted a while ago so you are also helping others in the process!).
I am now working on delta updates rather than world snapshots.
I have read that one could apply deltas as the difference between the last acknowledged packet - sending a delta vector would be smaller than a world vector.
However, how would one determine the last received gamestate from the client? I read that you confirm the last received packet and get the delta between that and the gamestate. However, by acknowledging every single packet the client received with the gamestate delta, it would create more bandwidth upstream, and it would incur a delay for the RTT.
Is this true? Any ideas?

The client sends inputs to the server every network tick. Adding a byte for "the packet ID of the last received packet" is not any amount of bandwidth to worry about. The important thing is to pack all messages for a particular network tick into a single packet for that tick; a few more bytes in that packet is not going to hurt anything. And, because it lets you optimize the downstream, which is where the big bandwidth usage is, it's actually going to help things overall.

Ok, so I've been thinking a little more about this. I was thinking about optimisation. Firstly, the less the client knows (that is up to date and relavant) the better. Also, i want to reduce downstream packet size.
So, it would make sense to omit telling clientA about clientCs new position if a lineOfSight test returns false between the two - that way the clients won't have to update each other about positions when they won't affect directly the user interactions (e.g shooting).
However, this means the following:
If i send deltas between the current game state and the client's last acknowledged, BUT omit irrrelvant data, the client isn't actually fully up to date. This means that in the future, the client will not have full game information. Therefore, I have two options;

1. Ignore this tomfoolery and just send all deltas
   
2. Store the client's current interpretation of the game, and use that for deltas.
   

Option 2 would make more sense. It would get the deltas between the last sent state and the current state, ommiting any invisible data (like hidden clients).
If a clientX was visible in this packet when it wasnt before, it would not fail the LineOfSight test, and the server would send the delta between the client's state (where the clientX in question was idle and unmoving) and the current state. In order to achieve this, after the delta is worked out, it is added to the existing game state and stored on the client. Therefore the client class on the server will represent the client's interpretation of the game, which in this example would have the idle X player behind a wall - thus the delta would tell the client to move clientX to its new spot.
This does mean more overhead though.
Does this make sense?