I am working on the design of a fantasy mmorpg that is set in Feudal Japan. Currently this is a "garage game" development. I plan to develop a good working demo for this game, and then seek investment to bring it to the masses. The ideal goal is to support 100K simultaneous players with 2.5K players per shard. I am concerned with making it very scaleable. I would ideally like to get a bare bones version of an expandable architecture running on a small test cluster of linux boxen. In order to meet this goal, I would like to break the game down into a set of processes (yes, in the unix sense). The list below indentifies the various processes of the game as I see it: * firewalling * authorization * patching * AI(s) * static game content database(s) * dynamic game database(s) * login lobby * landblock game/physics rules management processes * master control process * physical server control process * web portal process The basic client server flow would be: 1) client connects to the auth process and logs in (SSL) 2) a client patcher process starts, and commmunicates with the patcher process to update any out of date client files 3) the client is moved to the game login lobby, where they may choose a game shard to play 4) client''s last logout location/char profile is loaded from the dynamic game database and the character is spawned on the landblock process that is managing the login location 5) player plays, and is passed from landblock to landblock as he/she traverses the game world. Character data is backed up periodically to the dynamic game database. 6) player logs out, the character is persisted again at logout, and the player is moved to the login lobby 7) player can log out at this point or elect to join a shard and resume playing =================================================================== My first concern in tying all of this together is ip spoof proofing. I want to employ a challenge/auth technique. In other words, the very first packet from the client to a server machine will be to request a 32-bit anti-spoofing connection key. Every subsequent packet that contains this key must be from the initial IP. This verifies that the client really is at the IP they claim to be. My first question is would it be advisable to change this key periodically? If so, how often? Also, how do I maintain this key if it changes; in other words would I need another server just for the maintenance of this? Would it be advisable to build the spoofing protection into a "smart" firewall (but not very standard)? Is 32 bit enough? Perhaps I should use 64? =================================================================== A second question gets into how does the client ip data get routed to the right landblock server? Would it be inadvisable to tell the client the port:ip address of the landblock server directly? I am thinking I almost need a "smart" firewall/router process for this application too. As in, would this be advisable: * The landblock management process when it spawns a player object, registers with the smart router that gameplay packets for a particular connection key should now be routed to this landblock * during a handoff of a player from one landblock to another, one of the two landblocks will send a transaction message to the smart router to route gameplay packets to the new landblock My biggest concern is if I do follow this method, there is no easy way for the smart router to scale if it becomes a bottleneck. Maybe as part of the architecture I should include the ability to support multiple smart routers who autobalance the connection loads between them? =================================================================== Is any of this way off base? Please if you need more implementation details to address these questions, let me know. Any feedback is much appreciated!

You''re ideas look good. Have you started coding yet or are you still just designing? I''m just curious since I''m working on something similar. 100k players? I like it. Very ambitious! I''ll be happy to get 50 people on at the same time.

You''re anti-spoofing technique will protect you from someone randomly pretending to be someone else, but you''ll still be susceptible to attacks from someone intercepting your message and then spoofing the whole packet (key and all). The only way around this is to come up with some kind of private/public key type encryption scheme where the server changes the key periodically in synchronization with the client. That''s probably more trouble than it''s worth and it''s still susceptible to reverse engineering.

As for the routing of messages to the landblock server. I am planning to just give the client the address/port when they sign in. That''s what most MMOG''s that I''ve seen do. The only reason I can think of to not do this is that it puts other open ports on the servers at risk of attack. I''m not planning on running anything else on my servers so it''s not an issue for me. If you have a single server setup as a router, it makes it easier to bring your entire system to its knees with a single DOS attack.

bpopp (bpopp.net)

"You''re ideas look good. Have you started coding yet or are you still just designing? I''m just curious since I''m working on something similar. 100k players? I like it. Very ambitious! I''ll be happy to get 50 people on at the same time."
Heyas bpopp!
I''m still mostly in the design phase. I started this little project about a month ago. I''m working on the game design doc still (its VERY rough), but I wanted to verify my clustering architecture so I''m writing a test version of this in python.

I really want to get a working frame that goes from a client, through the login process, patching, all the way to firing up a simple engine that lets you move around a simple cube in the environment. From there it will be time to go back to working the design doc and adding the real content in this framework.

BTW, I live in California, but I noticed you''re from Tennessee. I am from TN originally, in the Murfreesboro area.


"You''re anti-spoofing technique will protect you from someone randomly pretending to be someone else, but you''ll still be susceptible to attacks from someone intercepting your message and then spoofing the whole packet (key and all). The only way around this is to come up with some kind of private/public key type encryption scheme where the server changes the key periodically in synchronization with the client. That''s probably more trouble than it''s worth and it''s still susceptible to reverse engineering."
I am just going for spoof protection to add a first line of security. Login/password and subscription information will be passed with a secure socket which I believe uses the dual-key system you mentioned.

"As for the routing of messages to the landblock server. I am planning to just give the client the address/port when they sign in. That''s what most MMOG''s that I''ve seen do. The only reason I can think of to not do this is that it puts other open ports on the servers at risk of attack. I''m not planning on running anything else on my servers so it''s not an issue for me. If you have a single server setup as a router, it makes it easier to bring your entire system to its knees with a single DOS attack."

I''ve since given this more thought, and here''s what I''m going to do:

There will be multiple smart routers. The client program will have a list of these addresses and pick one at random. If loading becomes an issue, I could perhaps have the routers communicate and dynamically hand off clients to maintain load balance.

The nice thing about this is it gives a good security control point; people outside will not be able to spam the landblock/patch/login directly.

Another good thing is it gives an access point for the spoofing key. The combination of the firewall ip address and an *internal* client connection id gives a unique way of identing each client connection. This connection key can be used on the insulated servers to associate game messages with a client. Also the firewall servers can periodically change the *external* anti-spoof key and this is transparent to the backend servers.

In other words, the firewall server will take the external key and associate it with an internal connection ID. It receives external packets, strips off the external key, then adds the internal key and forwards the data to the game servers. It can change the external keys at will and the internal servers will not have to deal with key changes.

quote:

---

I really want to get a working frame that goes from a client, through the login process, patching, all the way to firing up a simple engine that lets you move around a simple cube in the environment.

---

That''s similar to me. My first goal for this prototype I''m working on is to have multiple users login, get connected, and to be able to walk their avatars around my simple terrain. I''ll also handle some basic messages such as chatting, emotes, etc. I''m already a month behind schedule, but if I can keep up my current pace I should finish before Thanksgiving (or shortly after). I have all of the client stuff built (MD2 models, terrain, skydome, console, etc.) and the server is handling three messages (disconnect, chat, and connect). Once I finish I''m planning on posting an invitation here on Gamedev to see if I can get some testers.

If you''re curious you can take a look at my project page (see my signature). Unfortunately, the CPU melted on my linux server this morning so my site is down. Hopefully I''ll have it back up tomorrow afternoon sometime.

quote:

---

I live in California, but I noticed you''re from Tennessee.

---

Yeah, I''m from Memphis. Not much happens here (especially compared to CA), but it''s ok.

quote:

---

I am just going for spoof protection to add a first line of security.

---

Yeah, I imagine that will work. It should at least prevent a majority of the kiddie stuff. I think the only way it would fail is if someone happened to be on the same subnet as one of your customers and they intercepted a message that was being sent using a packet sniffer. Even then, all they could do is spoof bogus messages to the server which the server would most likely reject. I''m sure you''ve read this, but just make sure you put ALL your logic on the server. The client should have absolutely no authority over its state (or anyone elses).


bpopp (bpopp.net)

heh small world.

I used to live in TN, but now live in Cali as well and I too am working on a project similar to what you guys are doing.

I decided to mimic the server architechure that Everquest uses where theres a login server (possibly additional ones to support load), several different world servers, and each world server has numerous zone servers connected to it.

Most of the network framework is done and I''ve gotten the project to the point where the client can log into the login server via a username and pw and then get a list of worldservers presented to them along with their status. Users can then pick a world server and connect to it, at this point they are presented with a list of characters (or the option to create a new character) to choose from. Once they select a character the player connects the appropriate zone server (whatever one is hosting the zone the player was last in when they quit the game) and can move around.

Ive got a basic spawn system in place and mobs basically just spawn and despawn as if they were being killed while the player moves around. Other players can log in and see either other moving around. Although with only sending positional updates out to all the clients every 500ms the dead reckoning kinda sux and the movement can be jerky sometimes.

Data persistence is handled using MySQL DB. Characters and user accounts are stored here and each world that is created gets its own database for storing its relevant information. These can be hosted on either the same machine or their own machines.

I''m currently working on a GUI system so users can type in messages and commands to the server and getting the player to the point where he can zone into other servers hosting other areas of the game world.

Once I''ve added a few more things to the code, I''m going to start looking into creating some content creation tools and hopefully some content to go with it. =)


-=[ Megahertz ]=-

Sounds awesome Megahertz. I envy you. I''ve come so far in the last 10 months, but I still feel like I have such a long way to go. Is this something you plan to try and market or are you just doing it to prove you can (like me). Any chance I could take a look at a demo?

bpopp (bpopp.net)

quote:

---

Original post by Megahertz
Although with only sending positional updates out to all the clients every 500ms the dead reckoning kinda sux and the movement can be jerky sometimes.

---

Every 500 ms??? Doesn''t that cause a bunch of stress on your server? If you had on 1000 clients, you''d be sending out 500 updates per second per client. That''s 500,000 packets per second (presuming you don''t limit the updates to those close by, which of course you probably are But even so, you must be sending out a *ton* of position updates). That''s going to bring your server to it''s knees, isn''t it?

I''m only sending out updates if the last positional update was > 5 seconds ago on the client end. To smooth out the in-between time, I also send the velocity data along with it and let the client calculate where other avatars are.

I also plan on sending out an update when a client changes velocity to reduce the popping, but I haven''t gotten that far yet Once I do that, I figure the popping should be pretty minimal at that point, and I can correct an unsynched position over a few frames rather than all at once.

I''m also finding that my average player has between 200-1000 ms of round trip lag time (depending on their connection). This is another reason I went with a much larger update time.

So I''m curious to know how you''re managing with the 500 ms updates. Do you find that''s working okay?

just curious MHz, how long did it take you to get to that point? That''s awesome you have that much working. I am also planning to use mysql for data persistence. Also I have a *bunch* of questions:
1) did you use udp or tcp?
2) do you do any spoof protection?
3) do you use SSL for the login/password traffic?
4) Do you plan to store credit card numbers, and if so, how would you securely store it in your mysql database?
5) how do all the various servers know which server is at what address? Is there a database for this or is it handled in startup scripts, or what?
6) what platform are your servers running on? Linux/X86?
7) are you leveraging beyond2, nebula, nevrax nel, or twisted?
8) what languages are you leveraging for development? All c++ or mixing in python, etc?
9) what algorithm are you using for collision detection?
10) is your client directx or opengl or what?
11) (Related to collision detection) How are you determining viewability for a client? As in, how do you decide what things the client can see from the server perspective?
12) Are you using the observer/observable model for figuring out client updates?
13) does the client send update positions once every 500 ms too? If so, how does this figure into determining if someone is hacking the client to walk through walls?
14) Is your client engine BSP based or outdoor roamish/portals based?



Here are my answers to the above questions, for what it is worth:

1) did you use udp or tcp?
I plan to use udp for almost everything, with some (TBD) amount of TCP used for the login/patch process

2) do you do any spoof protection?
Yes, as above. I''m writing my smart firewall now.

3) do you use SSL for the login/password traffic?
I plan to, but my smart firewall only handles UDP, and handing off connection IDs is not trivial, but maybe I will have this process manage a SSL socket in addition to the UDP.

4) Do you plan to store credit card numbers, and if so, how would you securely store it in your mysql database?
I do not plan to store credit card numbers as yet. I plan to pop up a resubscribe when the user closes in on the end of a subscription period.

5) how do all the various servers know which server is at what address? Is there a database for this or is it handled in startup scripts, or what?
I''m planning on having the various processes be configured from a master control progam(MCR)at boot up, and the MCP persists the config in a mysql database.

6) what platform are your servers running on? Linux/X86?
Linux servers, windows dx9 client

7) are you leveraging beyond2, nebula, nevrax nel, or twisted?
I haven''t yet but when I get into the backend game servers, I might use twisted (www.twistedmatrix.com)

8) what languages are you leveraging for development? All c++ or mixing in python, etc?
Planning to use python for almost everything for my alpha demo. Moving to the release I plan to rewrite the bottlenecks in c++.

9) what algorithm are you using for collision detection?
I am planning on using a mix of http://www.gamasutra.com/features/20020118/vandenhuevel_01.htm

and

a great little sphere tree algorthm from the game gems 2 book, and I think EQ is using sphere tree.

10) is your client directx or opengl or what?
planning on a DX9+ client. Realistically, by the time this is released (2-3 years), DX9+ pcs will be the norm, IMO.

11) (Related to collision detection) How are you determining viewability for a client? As in, how do you decide what things the client can see from the server perspective?
Planning on using the sphere trees as talked about in 9.

12) Are you using the observer/observable model for figuring out client updates?
I am going to try to use the observable/observer model.

13) does the client send update positions once every 500 ms too? If so, how does this figure into determining if someone is hacking the client to walk through walls?
I am thinking client updates just have to occur every 200ms or so to do proper server side collision testing. Maybe I''m wrong here though? Perhaps if the client sent along a spline curve approximating its path, you could get away with fewer updates.

14) Is your client engine BSP based or outdoor roamish/portals based?
outdoor engine, no bsping planned. I''m still fuzzy on how this will all play out.


As an aside on updates, I''ve seen the following anti-lag measures:
1) change update frequencies of objects based on how "important" they are. If its hostile NPC it is important. If its a loot bag or friendly npc, it may not be as important
2) decrease update rate for more distant objects. I.e., the farther away it is, the less frequently you update it
3) prediction: instead of just positions, send velocities as well, so the client can approximate the object positions
4) Run the simulation some amount of time lagged so the future is to some degree already known (implies you can interpolate more rather than extrapolate)
5) prediction: instead of just putting the objects at positions, use the positions to plot out a smoother curve like a spline
6) Change update frequency based on how well your (probably linear) prediction model is handling the object. For example if someone is running in a dead straight line for a long period, the velocity prediction model works well and you can slow the update rate. If they are moving erratically, the update rate may need to increase

Well, I'm not Megahertz, but if you care, here are my answers

>1) did you use udp or tcp?
UDP, exclusively. I understand the lure of using a hybrid UDP/TCP approach (for file updates and the like), but I had already implemented a reliability and ordering protocol for UDP that I could apply to selected packet types at will, so there was no need to use TCP in my program (besides which it would have vastly complicated my server network code, which is IOCP anyway so therefore pretty complex as it is )

>2) do you do any spoof protection?
Yes. My users are required to prove they are the person they say they are. This involves IP/Port checks as well as encoded ID values.

>3) do you use SSL for the login/password traffic?
No. I use MD5 encryption and the plain text version *never ever* leaves the client machine. Since MD5 is irreversable, I think this is secure (someone tell me if I'm wrong, heh).

>4) Do you plan to store credit card numbers, and if so, how would you securely store it in your mysql database?
Yes I do. I haven't yet looked into this, but my understanding is that PayPal has a system whereby they hold all that info and deal with the billing process (for a transaction fee) and in return I get something that plugs into my database allowing me to know at any given moment who does and doesn't have a current account. That's about all the details I kn0w about it, like I said, I've not researched it at *all*

>5) how do all the various servers know which server is at what address? Is there a database for this or is it handled in startup scripts, or what?
Not applicable. I have one server

>6) what platform are your servers running on? Linux/X86?
WinXP Pro.

>7) are you leveraging beyond2, nebula, nevrax nel, or twisted?
No.

>8) what languages are you leveraging for development? All c++ or mixing in python, etc?
C++, exclusivly. Of course, I talk to my database (MySQL) using the normal SQL language, but even then, in the server code, I use the mysql.h header so I can access it via C++.

>9) what algorithm are you using for collision detection?
Undecided. Collision detection hasn't been implemented in my game yet. I'm open to suggestions (I'll check out your links).

>10) is your client directx or opengl or what?
DirectX (D3D more to the point of you question, though I also use DInput and DSound).

>11) (Related to collision detection) How are you determining viewability for a client? As in, how do you decide what things the client can see from the server perspective?
I'm using a landblock based system. If an item is in the same or an adjacent landblock to the player, it's data gets transmitted. Yeah, it's a bit inefficient (at initialization, you are sending up to 9 landblocks worth of data, while during play at worst case you could send 5 landblocks if the player crosses a boundary just right), but any other way I've seen requires major computations on the server end that I don't want the server spending time on. My server is based on the idea of processing incoming packets and getting onto the next one as fast as possible. It's up to the client to decide what is viewable from the data given.

>12) Are you using the observer/observable model for figuring out client updates?
No, if I understand your question properly.

>13) does the client send update positions once every 500 ms too? If so, how does this figure into determining if someone is hacking the client to walk through walls?
My clients send velocity and position change updates whenever a velocity changes (forward, sidestep, or angular, in other words, whenever the user stops or starts pressing a movement key), or every 2 seconds, whichever comes first. The server returns an ack to the client when it gets an update. Also all my clients send out 3rd party position update requests if the last one recieved for that avatar is more than 5 seconds old (as I said before). The server sends out a 3rd party position update upon request, or when that party changes one of their velocities (the second part of that hasn't yet been implemented, but that's the plan).

14) Is your client engine BSP based or outdoor roamish/portals based?
ROAM, portals indoor (though only ROAM is implemented as yet, there are no indoor sections to my game at this point ).

-Ron

[edited by - RonHiler on November 16, 2003 1:24:51 AM]

Hey Ron,

Thanks for the informative response!

Your all UDP implementation inspires me to do the same. You''re right, as long as you can apply a message sequencing/resend on top of udp, why bother with tcp?



">2) do you do any spoof protection?
Yes. My users are required to prove they are the person they say they are. This involves IP/Port checks as well as encoded ID values."

That is exactly what I am doing. Just curious, how big is your ID value? 16, 32, 64 bit?


">3) do you use SSL for the login/password traffic?
No. I use MD5 encryption and the plain text version *never ever* leaves the client machine. Since MD5 is irreversable, I think this is secure (someone tell me if I''m wrong, heh)."

I don''t think this is secure. What prevents someone intercepting a packet and reusing the hash to gain someone''s login? True they won''t have the plaintext password, but just resending the hashed to the server will work.

The only way to really do this securely that I know of is with a public/private key encryption system. In these systems only the private key can unlock what the public key has encoded, and vice versa. The usage would be:

1) server sends the client the public key
2) then the client can encrypt with the public key and send data, and only the private key on the server can decode this data.

The above is probably secure enough for login/passwords for non-financial data. If credit cards numbers are being send you probably need a bit more...

A problem that still remains is if someone can intercept packets and impersonate the server. To prevent this, people use digital signatures from a trusted, guaranteed secure third party. I''m a little fuzzy on how you avoid this but I think people just do something like store the public key with the trusted 3rd party.


">4) Do you plan to store credit card numbers, and if so, how would you securely store it in your mysql database?
Yes I do. I haven''t yet looked into this, but my understanding is that PayPal has a system whereby they hold all that info and deal with the billing process (for a transaction fee) and in return I get something that plugs into my database allowing me to know at any given moment who does and doesn''t have a current account. That''s about all the details I kn0w about it, like I said, I''ve not researched it at *all*"
Paypal is a good idea, I need to look into how they operate.


">9) what algorithm are you using for collision detection?
Undecided. Collision detection hasn''t been implemented in my game yet. I''m open to suggestions (I''ll check out your links)."

The sphere tree algorithm shows up in the books Game Programming Gems 2 and Game Programming Gems 3, and it looks very powerful, but i haven''t tried implementing it yet.

">11) (Related to collision detection) How are you determining viewability for a client? As in, how do you decide what things the client can see from the server perspective?
I''m using a landblock based system. If an item is in the same or an adjacent landblock to the player, it''s data gets transmitted. Yeah, it''s a bit inefficient (at initialization, you are sending up to 9 landblocks worth of data, while during play at worst case you could send 5 landblocks if the player crosses a boundary just right), but any other way I''ve seen requires major computations on the server end that I don''t want the server spending time on. My server is based on the idea of processing incoming packets and getting onto the next one as fast as possible. It''s up to the client to decide what is viewable from the data given."

Just curious, how big are your landblocks, and how is the scale setup with respect to your player?

">13) does the client send update positions once every 500 ms too? If so, how does this figure into determining if someone is hacking the client to walk through walls?
My clients send velocity and position change updates whenever a velocity changes (forward, sidestep, or angular, in other words, whenever the user stops or starts pressing a movement key), or every 2 seconds, whichever comes first. The server returns an ack to the client when it gets an update. Also all my clients send out 3rd party position update requests if the last one recieved for that avatar is more than 5 seconds old (as I said before). The server sends out a 3rd party position update upon request, or when that party changes one of their velocities (the second part of that hasn''t yet been implemented, but that''s the plan)."
hmm client driven updates. Thats interesting; it seems like a good idea.