Networked Game Structure

Referencing WoW or even EQ, you used to be able to connect and if the login server crashed, you would be fine long as you didn't log out.[/quote]
WoW used two different sockets to connect to it's client on two different ports.. Essentially you would send handshake and auth info to a login server, which would process your build info and details, send back a result packet and a realm list. You would then choose which realm to connect to - a new connection would be opened to the realm server and the login one would be dropped immediately after connecting.

In other words, in WoW at least, you would connect on two different sockets. Basically, yeah, I hate how you have to do it too, but it seems creating multiple sockets is the only way. Pseudo code:

Socket s = new Socket(serverAddress);
s.send(AuthRequest);
..wait until recv.
if(success)
Socket r = new Socket(realmAddress);
else
Error to base screen


So my question is how did they do it? Without exposing too many sub servers, what would be the best solution?

Your servers need to be flatter. It could look something like:


Subscriber/User Database
^ ^
| |
Subscriber Login Realm Character Database
^ | ^
| | |
| +---+--------+---+
| | |
| Realm Login Zone Server
| ^ ^
| | |
+--------------+----------------+
|
Client



The client connects to the subscriber login server, which uses the subscriber database to verify subscription.
This also returns a list of realms that are available. The client chooses one, and connects to the realm login server, which verifies the login credentials, and shows a list of characters.
The client selects a character, and the realm server tells the client which zone it's currently in.
Client connects to the zone server, giving a character and user reference, which the zone server verifies with the realm server, and then instantiates the character in the zone.
At that point, the client can talk to the zone server for all zone gameplay and chat, the realm login server for guild chat and cross-zone chat, and the subscriber server only for specific-player chat (if you support that -- most systems don't).

You probably have many realms that all share a set of subscribers, and many zones within a single realm. Similarly, you probably use Kerberos-style authentication (signed, datestamped tickets) for the hand-off from subscriber to realm, and realm to zone.

Neglected, thank you for your response I figured something like that would be needed due to the desire to move from a "single gateway" to a secondary sever, to avoid "login server/gateway" crashing and the connected clients dropping as well.



hplus0603, first thank you for the detailed post. You gave me more ideas about how to add/implement things that I have done differently, thank you for that. But my question is not about verifying, but how to route a new socket to said "sub-servers." But correct me if I'm wrong; from your post I've come to the conclusion that I will need to expose at least two/three servers. The login/gateway, the realm server, and the chat(global chat e.g. guild /global channels) server. (could combine the two into one but that's a minor detail for now.)


And to do all of this (again please correct me if I'm wrong) but I will need to close and recreate sockets to connect to said servers. e.g.

First Socket: Connect to login server, verify account. Obtain a Kerberos-style Ticket if the account is valid.
Server sends the ticket to the client verifying the user name/password is valid, and a list of realms.

Second Socket: Client closes first socket, recreates a new one that connects to the selected realm.
Client sends the Kerberos-style ticket for validation.
Server verifies. If valid, returns a list of all the characters.


Client is now connected to [Selected Realm] and now all messages can be routed through the [Selected Realm] server to zones/global chats (guild, whispers, etc), no need to connect to specific zone servers as all messages can be routed to the proper Port a zone server is listening on (assuming all zones of a specfic realm are on one machine and the same machine as the realm server itself).


So if the above is correct, would this mean that the login server exposes PORT addresses for all realms? e.g. Login_Server: mygame.com:7501 Realm_Sever: mygame.com:7502 Specific_Realm: mygame.com:7503-7600 (assuming 97 realms ... yeah I know just an example.) and to move from one to another I'd just recreate a socket and connect to the proper port address (with verification happening obviously) With this method wouldn't exposing the realm ports be a bit dangerous in so much that someone can blast the realms with all types of attacks? I'd like to keep the specific realms unexposed .. but I suppose this means using a gateway such as the "Realm Server" which would then be the target of said attacks if they were to happen anyways?
_

hplus0603, first thank you for the detailed post. You gave me more ideas about how to add/implement things that I have done differently, thank you for that. But my question is not about verifying, but how to route a new socket to said "sub-servers." But correct me if I'm wrong; from your post I've come to the conclusion that I will need to expose at least two/three servers. The login/gateway, the realm server, and the chat(global chat e.g. guild /global channels) server. (could combine the two into one but that's a minor detail for now.)

If you want to use a gateway, then you can build a separate gateway (taking the role of "client" in the diagram) and build a network protocol that adds an additional level of framing that the client uses to talk to the gateway. It really doesn't change the architecture much, except you need to expose fewer physical machines directly to the internet. (If you have full control over the services exposed, you can make that as secure as a system with a gateway)

So, instead of "client" in the diagram, it would say "gateway," and the client would connect to a gateway. I would probably make the gateway dumb, so the system would tell the clients "you need to talk to server 3722" and the clients would then ask the gateway to establish a new connection to 3722. To the client, it would probably look a lot like establishing new socket connections, except the 'address' in question is just "server 1234" in a way that the gateway knows to resolve to a particular host.

Note that in a real data center, you typically have a hardware load balancer of some sort (Juniper, Brocade, F5, whatever) which has the public IP/IPs of your service/s, and it does reverse NAT for connections to your array of servers. In some sense, this serves as a "transparent gateway." You'd typically want to allocate one port on the outside for each service/machine combination on the inside in that case.

[quote name='grazing' timestamp='1321273199' post='4883741']
...

... [/quote]



hplus0603, don't take this the wrong way but I love you. A million thank yous!



I'm sure we've all been at a point like this but I've been stumped while attempting to process this situation in my brain for a few days and couldn't quite wrap my brain around it until now.

Thank you again.

Sorry for reviving old post.
For the "traditional" gateway approach the problem I see is that if gateway falls, all clients fall as well.
If you have a list of gateways would be a matter to let the client know IP/Ports and if losing connection just trying to connect to any other from the list without user even noticing.

So the process would be:
1. Client login -> receives list of gateways
2. connect/validates to gateway
3. client start sending messages to gateway, it will redirect to proper internal "sub-server"
...

An image to ilustrate it better:
[attachment=6895:Dibujo.jpg]

*Note Im calling a gateway what in fact is a "home-made" server with functionality reduced to receive messages from client, parse/validate and resend. Gateways would be the only accesible by Internet and gateways and subservers are in same LAN.
** Gateways and subservers can easily scale.

Why bother with all the relay servers? I don't see what it gains you, aside from a lot of extra hardware to purchase and maintain. After all, there's no point running your "gateways" (relays?) on the same boxes as the "subservers" because that just defeats the purpose of redundancy.

I'm personally more a fan of connecting to a point of authority for login, then reconnecting directly to what you call "subservers" for service (be it gameplay, web shopping, or whatever else). The login system gives you a token which can validate your connection to the actual services, and you're done. The authentication system can easily be scaled horizontally for the capacity and fail-safety required, and you're free to scale the back end services as necessary without any intermediate infrastructure.

In other words, you get a (more or less) maximum ratio of fault tolerance to requisite hardware.

Hi,
I will explain before how I have in mind doing it:

All communications are limited to text, what client-network exchange are just text-strings with a specifiq format to identify the command and params, something like:
command&param1&param2&...
In the case a client want to send a message to a chat room, it would send something like:
"chatmessage&GeneralChatroom&Hello all"

Gateway would receive, decrypt and validate the string and depending of the command identify what server is in charge to process it, so it ends resending the string to the proper server in the network.

*Note what I call gateway is a server with traffic redirection based on message content as main duty.

Benefits of the gateway:

1. One IP per gateway, no gateway means you would need 1 IP per machine in cluster?
2. Frees of some network programming aspects client and "sub-servers", they just handle 1 connection for everything.

The login system gives you a token which can validate your connection to the actual services[/quote]
Yes, login is before all this. client and gateway obtain a token per session which is used also as key for message encryption.

Replace "gateway" by "proxy" and this is almost what I mean:
http://realmcrafter.com/store/pages.php?pageid=12

If you get to the point of engineering some kind of gateway server IP fall-back, then you should take a step back. You're doing it wrong.

Buy a hardware load balancer. In fact, buy two. Link them in "high-availability/hot-standby" mode. If one dies, the other can immediately jump in and take over.
This hardware load balancer is the interface between the internet and all your servers. It also serves as a firewall of sorts. You can use a single IP address on the outside, and map a number of ports (say, any port between 1 and 65535) to a different host and port on the inside.
An alternative is to use the load balancer to do round-robin balancing. It will poll your web servers (or gateways) for up status, and send each new incoming connection to one of the available, up, target hosts.

Yes, you may pay many thousand dollars for this hardware, but when you're at the point where you need gateways and forwarders and whatnot, the leasing cost of this hardware is a very very small part of your budget.

Yes, you may pay many thousand dollars for this hardware, but when you're at the point where you need gateways and forwarders and whatnot, the leasing cost of this hardware is a very very small part of your budget.

I am thinking in the amazon web services, do you have a favorable opinion about it?
http://aws.amazon.com/ec2/
http://aws.amazon.co...cloadbalancing/

Thank you hplus and ApochPiQ for the time to response, got it clear now.

Designing scalable infrastructures - in pretty much any field, but particularly in networking - is all about contextual details.

What kind of service are you trying to provide? How are requests from clients fulfilled by servers? How frequently do requests come in? What is the ratio of servers to clients expected to look like? What kind of resources does it take to fulfill the average request? Can you segregate request types based on their resource requirements, i.e. put lots of cheap requests on one server and spread out the expensive requests among multiple load-balanced servers?

All these kinds of details are important. I could probably write a huge list of such considerations that go into designing the typical high-scale internet service, and I'm far from an expert on the different models in use in real life systems (my own experience in high-scale systems is limited to a particular subset of technologies).

Without considering all these factors, it's hard to accurately judge the merits of some arbitrary architecture. What you have diagrammed may work excellently for certain styles of service, and completely fall apart at high loads for other types of service. There's a reason why services like elastic cloud computing are middle-of-the-road solutions; they will neither give you optimal performance at load, nor optimal price efficiency on the low end of the scale. (Though to be fair, cloud services are getting pretty darn affordable for most applications these days.)

Services like EC2 have to try and strike a balance between effective use of their own hardware resources and effective scalability for their clients. Someone designing their own network back-end infrastructure is free of such constraints, and can focus on building something that is optimal for their particular needs. And the solutions they deploy are likely to look very, very different.

I am thinking in the amazon web services, do you have a favorable opinion about it?
http://aws.amazon.com/ec2/
http://aws.amazon.co...cloadbalancing/

Thank you hplus and ApochPiQ for the time to response, got it clear now.

Yes, Amazon Elastic Load Balancer is one way to achieve load balancing, if you use their Elastic Compute Cloud infrastructure.