Skip to main content
gamedev.net
🔒 Locked

Splitting packets

Started by WebsiteWill May 11, 2003 at 5:22 PM 8 replies 750+ views
Original Post
WebsiteWill
WebsiteWill
While reading through the forums earlier I came across something that I''m having trouble figuring out. Basically what was said is this. When sending packets to a computer it can be a good idea to number the packets sequentially if you need to know if a packet arrived out of order. I can see needing to do this in order to send chat messages in a game or for file transfers. They said that basically Client sends packet 1 and server expects packet 1. If the server receives a packet earlier than what was expected then it was a duplicate and can be thrown away. If a packet later than the one expected arrives then it is put into a buffer for holding until the time when it is used. So based on this, both machines would need to keep a backlog of packets sent so that if for instance the server recieves packets 1,2,3 and 5 then it puts 5 into holding until 4 arrives. I''m guessing that if packet 4 doesn''t arrive then the server sends a message telling client to send packet 4 again because it never arrived. A question on this is, how many packets should we prepare for. Should I have a buffer for 3 packets, 5, 20? Or is it simply that the server will demand packet 4 before it will do anything with packet 5 and won''t accept any new packets until packet 4 arrives? Also, this is easy to visualize for a single client. But when using UDP where you have no direct connection I will have to examine each packet to see where it came from then based on where it''s from I examine the packet number to make sure it is in order. So potentially I am going to need a buffer for every single client to handle. Is this correct? So now let''s say that my program sends 4 different types of packets. 1) Non critical movement requests. If we lose one then we don''t really care as we''ll just adjust with the next one. No buffer needed for these. 2) Chat packets. We care about order here and we care that they all succeed so that''s one buffer. If I limit the number of maximum characters a person can type before sending then I will only need a buffer of at most 2 times the size of the largest possible chat packet. Chat packets can be varying length but you have to plan for the worst right? 3) Will need critical packets that are sent when the client makes requests that MUST reach the server. Things like request to change armor and such. These must arrive but do I need to care what order they are received in? Or can I just handle these in whatever order they arrive in? Also monster.died packets are critical I would think as well as a few others that fit into #3. Then 4) packets for file transfer are critical and have to arrive in order or they are useless. This is the hardest one to visualize as I would potentially either need a buffer large enough to hold the entire file or I could restart the entire transfer from packet one if it happens that more than two packets in a row are skipped. Or some other system to guarantee we get all the packets. Just seems like a lot of memory to keep in ram for every connection on the computer. At this point I am assuming 200-300 people physically connected (er sending to a certain port) to any server at a time if I get lucky. I''m definitely going to go UDP with this. Exactly how I dunno yet. I still haven''t found much useful stuff on IOCP. I''ll be writting my on guarantee system on top of UDP for when I need it. I''m just trying to figure out how all of these connections (well not exactly connected with UDP but you know what I mean)are managed. If I have 300 clients talking to a server with UDP then each packet will have 1 of 300 unique name identifiers with it. MAP works well here for locating the buffer any given packet belongs to. It also makes it easy to send system wide messages as I just cycle through the map keys sending whatever I need to them all. 300 clients doesn''t seem like too many to handle as long as the info sent is kept to a minimum. So my whole process would be something like this if I am correct. Server binds and listens on a certain port. Client binds and listens on it''s port. No real connections are made, it''s simply a matter of client sends a packet request to server who then accepts or not. Now the client has permission to trade info with the server. Server simply moniters to see if any incoming info is available. if so then I grab the packet, see who it came from and stick it into the correct place. If I have 300 clients then i can potentially have 300 packets waiting to be received every time the program loops so what do I o, call recv() until nothing else is there? Call it once per loop? A certain number of times? Just really confusing once you break away from a single client and server. I assume this has to take into account the actual game loop unless the send and recv is done in it''s own thread and the game loop simply does what it does with any information that might be in a TOBEPROCESSED buffer. Maybe I''m just confusing this more than it has to be.Sorry for such longwindedness here. Just a lot of stuff I''m not clear on yet. Thanks, Webby
martinpiper
martinpiper
You are right. Things do get more complicated when you depart from the single server and client model.
Amongst one of the many things you should remember is that if you opt for a multi-threaded approach, then when you are getting packets try to move the packets in to the relevant buffer as soon as possible. If you access the buffer from another thread you will need to use some sort of mutex/semaphore/critical section. This brings you on to the second thing to remember which is you shouldn''t block for too long when adding or removing items from the buffer of packets.
If you are receiving packet data faster than your PC can cope with processing the packets then you are in big trouble. (And probably have a slow PC or a very fast network card. )
Things do get difficult when you are rolling your own reliable/semi-reliable UDP solution and you start to handle more than about one hundred connections. I know that it is possible to re-write this area of code at least three or fours times before it is anywhere near fast enough. Just accepting new connections, responding to them, checking for duplicated packets and making new connections available to your "pending packets queue" is time consuming.
What I did in the end was to create my own database class which used a tree and node arrangement with a hashing algorithm that enabled a much faster match and lookup to be done on an IP address and packet sequence number to a relevant buffer or queue. This is 40% slower than a standard link list for queries up to about 24 random queries then after that the database starts to get quicker. When you get up to 3000 simultaneous random queries then the database works out to be over 1000 times faster than a linear link list search.
Regarding the buffering of UDP packets, so that they can be resent. Yes you do need to keep them available for a certain amount of time. Usually you can say that after 30/60 seconds if the server has not received any ACKs from a client then you can assume that connection to be dead and forget the whole queue. Or you can implement some kind roll back or consistency checking solution to the server.
The possibilities are endless.


--
Martin Piper
Network programming made easy with ReplicaNet
http://www.ReplicaNet.com/
-- Martin PiperNetwork programming made easy with ReplicaNethttp://www.ReplicaNet.com/
WebsiteWill
WebsiteWill
Thanks Martin.

I''ve been looking still for some stuff on IOCP but since it''s Windows only I really would rather have something as I plan to host on Unix if indeed it is ever hosted on anything.

I keep looking for IOCP stuff but mostly in hopes of stumbling across a good solution for Unix sockets. I''m developing it on windows with Winsock but it''s ported to Unix easily enough from there as long as I stay away from Windows specific functionality. Just the sheer issue of how to handle all of those connections is daunting. Guess I should just break down and write a few different methods and stress test it by spawning a bunch of local clients just to see which of them hold up the best. That''s probably what happens in every development house across the globe when they first decide to take on a multiplayer game

Mayhaps I''ll get good enough at this to be able to share so decent knowledge with this crowd pretty soon. Maybe in the form of tutorials. Being the comment and documentation stickler that I am, writing up a decent tutorial for it won''t be hard at all if I can ever get so solid code designed and written.

Wish me luck,
Webby
WebsiteWill
WebsiteWill
For IOCP I''ve read that you generally would create worker threads to handle everything. The general number of threads seems to be 1 for ever processor on the server. So what I gather is happening is that potentially each thread could recieve a message (packet) to work on. Given that packet it would then do what? Perform one iteration of the game loop on it''s own? If this is the case then given a 4 processor system with 4 helper threads then at any given time 4 frames of game can be getting processed at the same time? Or is game processing data handled seperately and the threads simply gleam the information from the actual game loop that it needs to process information requested or sent with the packet?

Also, what would be the Unix equivalent to IOCP. I''ve read about forking child processes and multiplexing sockets and multithreading and so far it seems like forking a child process would be the way to go. Only problem with that is what i have read entails forking a child process forbasically every client and that seems like it would quickly eat up system resources. Not to mention, it confuses me (as far as actual game logic goes) just as much as the threads needed for IOCP.

I can''t see each thread running it''s own game loop as that would be disasterous. Maybe I should go get the Quake2 source code out and just stare at it until something clicks. The more I read the more confused I get at times and I am generally able to learn stuff like this very quickly but not this time. Mostly because of the lack of concrete examples to learn from due to the tight secrecy kept when someone does finally get it right.

That and the fact that neither my local Barnes and Noble nor Books A Million supply anything near decent in the realm of network programming books. Not even the acclaimed Steven''s book!

Webby
wrathgame
wrathgame
Hi,

cant say much because my dissertation report is due in 7 hours (agh) .. paper on UNIX IOCP

Banga Gaurav, Mogul Jeffrey, Druschel Peter (1999). A Scalable and Explicit Event Delivery Mechanism for {UNIX}: {USENIX} Annual Technical Conference pp 253-265
http://citeseer.nj.nec.com/banga99scalable.html

And.. you point is very valid about multi processors.. basically when making a server which runs on multiple CPUs.. its almost like making a server which runs on multiple seperate machines (except the line of communication between them is much faster )

You would have to do a hell of a lot of management of resources.. making sure you lock access to memory each time a CPU is using it.. this could be bad cuz it may stop the other CPUs and make them pointless..

Another important note about IOCP on multiple processors, or even multiple threads... potentially with TCP a message can be sent in 1 chunk and recieved in 2 (in order of course) .. but if thread A starts to recieve the 1st chunk and then thread 2 interrupts and gets chunk 2 and stores chunk 2 to a buffer.. then Thread A kicks back in and stored its recieved data after chunk B... ops :/ so there are a lot of pitfals.. i would say on a multi processor computer just have 1 thread still for the IOCP and let 1 CPU manage that cuz it performs amazingly well anyway on a single CPU... then have other thread(s) processing the data by reading it of a lockable buffer which the IOCP thread writes to.. sorry dont have time to write this down well

hope that helps


~ Tim
~ Tim
WebsiteWill
WebsiteWill
Judging by the title of your dissertation I am guessing IOCP is a viable solution for Unix as well. The info you posted in regards to TCP I was aware of but not sure how to handle it. Although I am going for UDP. I need to go read to make sure UDP wil work with IOCP.

Another thing that I have read is that IOCP (on whatever system) is extremely scalable. They tout the ability to have as many as 50,000 clients on a single server with IOCP. In reality, a game server is not going to have more than probably 500 clients each on todays machines (I''m guessing here). So, that kind of scalability might not be worth the effort. However, finding a definitive answer on which type of setup would work well for a 3D game server (or hell, just a slightly educated opinion even ) is very difficult.

It''s not really a matter of implementation that''s hassling me. It''s purely a matter of which approach to take and an understanding of the fundamentals.

Back to the search function and web browser
Thanks,
Webby
wrathgame
wrathgame
The scalability is amazing with IOCP (i have just tested my one at 30000 connections and it only used 1% - 2% CPU)

Yes UDP is possible with IOCP check www.sourceforge.com

and concider the fact that with blocking sockets with one thread per client, after you connect 1000 clients to it the server alone will be context switching so much that it causes whats called thrashing, which is when the OS keeps having to swap data between the swap disk and RAM. yous server alone will be using 100% CPU and causing your whole PC to start lagging.. now add some game server code

with IOCP, no matter if you have 10 or 1000 or 10000 clients, you will not be waisting any processing power which could be used to serve more clients.

max number of clients is only dependent on how much bandwidth each server has, and how much processing each client will use up

ps. i havent been to bed for >30 hours now so my sentances are loosing their will for grammer


~ Tim
~ Tim
WebsiteWill
WebsiteWill
Wrath

Thanks a bunch. I''ll look more thoroughly into IOCP this weekend after finals! That''s when I hope to begin collecting all information I need to begin some rudimentary programming tests. It certainly sounds liko IOCP will be the way to go.

Know any good books or websites on the subject or at least with a chapter here or there on it?

Thanks,
Webby
wrathgame
wrathgame
See my comments on this post
http://www.gamedev.net/community/forums/topic.asp?whichpage=2&pagesize=20&topic_id=152387


~ Tim
~ Tim
WebsiteWill
WebsiteWill
Yep, I''ve read that thread before. I must say that my mind is made up. I''m off on a crusade to learn a whole lot of stuff about IOCP... Just after I take a Physics test and Theory of Computation test of Thursday. Psychology test on Friday and Compilers test on Saturday. Only one more semester till graduation. Then I''ll be another of those guys in the street chanting "Will trade degree for food"

Webby

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.