I understand how client side prediction works. I have some questions which I need to clarify:

1. What exactly do we send in the userCmd? In my case, what are the information I should be sending for a fps shooter?
2. When do we send it? In the main animation loop of the client?
3. If the server runs the game at 60 fps and the client runs the game at 30 fps, the client will always snap back even after server replaying the stored commands. To fix this we will need to update the game based on deltaTime right?
4. If the server runs the game at 60fps does that mean it should send the updates at that rate too?
5. In a first person shooter, should I correct the yaw and pitch of the player after receiving the update from the server too?
6. In the history (on the client) what should be good to store? delta positions or user commands? The player will have multiple forces acting on it. Should the server update also send current velocity in this case too? I don't think it needs to do that. I can go about with the current velocity even after I have received an update from the server. Is that correct?

Lag compensation: This is something I don't fully understand. If the player shoots another player, the player's yaw and pitch are obviously different on the server. How should I register a hit in here?

@undefined what should i send for player yaw and pitch? Mouse movement or the local yaw and pitch?

I would recommend sending whatever you get from your game engine.

If your game engine translates key presses to commands, and mouse movements to yaw and pitch, like many standard setups in Unreal Engine do, for example, then sending those is the right thing to do. Also, sending yaw/pitch makes sense, when you let the user control “mouse speed” – if you send mouse movement, and the server is supposed to derive yaw/pitch from that, then the server has to also know the mouse acceleration/speed settings for each player, which is cumbersome.