Original Post
Hey guys! I am working on the architecture/design of a distributed system and I am not 100% sure of how to attack some of my problems. So I figured out I wanted to make a post to this great community to hopefully get some nice input :) And gosh, this will be a long post so I really hope someone has patient to read it =/ Basically, the system I am working on is a system for providing highly interactive and some non-interactive services for mobile phone users, in addition to provide billing services for external systems (like external streaming servers etc). I have already implemented the system today and it has been running for soon 2 years, with a few quirks here and there but in overall it is quite stable. However, today, the system is implemented as one application with the exception of inbound MMS receivers/connectors which are implemented as servlets, running under Tomcat. The current system is event based, i.e. when an inbound SMS is received on one of the inbound SMS connectors, the message is first converted to an internal message format and then an InboundSMSEvent is posted to the system and whatever module that is interested in inbound SMS pick the event up. Basically, this is the service subsystem that does and it is the service manager that listens for inbound SMS events and dispatches the message further to correct service based on keywords written in the text message (SMS) etc. The connector subsystem (or tier if you want to) is responsible for communicating with all external systems. The definition of a connector is basically a component or module that handles the inbound or outbound communication with the different operators, whether it is messages like SMS, MMS or system specific message formats etc. The protocols or message services supported are SMPP, JMS, HTTP/HTTPS (SOAP, XML) and maybe others as new protocols or message services must be supported. The connector tier is also responsible for translating between operator specific message formats or protocol specific message formats and the internal message format used by the system. A service is a component or module that does something for a user of the service like a chat service, auction service etc. Even though the system works very well today, there are many drawbacks. The system does not scale very well, if one thing crashes it might crash the whole system and whenever a new update is needed for example for service A, the whole system must be restarted which resets the current state of all services and the rest of the system (the latter can easily be solved by persistence of course but you see my point). So a new architecture is needed to make the system more scalable and get rid of some of the other issues too. The system will be distributed. The system I am designing now shall be the “core platform” where new connectors and services will be easily be added as new telecom operators shall be supported or new services will be launched. The new system will follow very much of the previous system design, except many parts will be made distributed. The system is implemented in JAVA and I have decided to go for RMI but any other suggestions is welcome But my main problem in this all is to actually figure out how to make the system distributed to make it as scaleable as possible, flexible, good performance and stable and avoid single point of failure and bottlenecks. The message flow is pretty easy; a message originates as an SMS or MMS from a mobile phone to the operators and then to our system or the message originates from another external system and then is received by the connector tier, which converts it to internal message format and passes it further to the service tier. And vica versa, the message is generated by one of the services, is passed through the system to the connector tier which is then responsible to send it further using the correct connector (i.e. using the correct operator). For telecom operators’ connectors, there can be only one inbound connector but in some cases it can be 1 or many outbound connectors. Today, the connector manager is responsible for starting up each connector which in turn registerate itself with the connector manger using the country code + a unique id for the country as its id. The same for service system, the service manager is responsible for starting up each service which in turn registerate itself with the service manager using a unique service/channel id. There are other requirements to consider too, which is today handled individually by each service which is far from ideal. And that is for example special rules each operator might have. For instance, the operators in Norway have rules that if a user receives 20 premium messages (MT’s) from the system without sending a message back (MO), he shall be logged out of the service/system. Another rule is that a user can totally only be billed for max 5000 NOK pr month so each service must be aware of how much other services has billed. A own subsystem tier will handle this operator rules in the new architecture. My main problem is that I can clearly see how to make this distributed and yet avoid bottlenecks, single point of failure, good performance and stability. On top of that, the services must know where to find the connector manager or connector managers, the connectors must know where to find the service manager or service managers, the connector manager must know where to find all outbound connectors and service manager must know where to find all services. But this post already starts to get extremely long, so let me try to explain a specific scenario. Let’s say a service shall send the same message to 1500 users. The service can either generate 1500 messages; post them to the connector manager which dispatches the message further to correct connector. This might or might not put unnecessary load on the system. Instead, the service can generate 1 message containing 1500 recipients and pass it to the connector manager which in turn will can either sort the recipients based on country and operator for each user, generate 1 message for each combination of country+operator and add multiple recipients if the operator supports that or break it up in single message if the operator does not support multiple recipients in the message formats. Though sending 1 message to multiple recipients (if operator supports it) creates new problems; message logging. But that is of course also solvable. Another way is that the connector manager itself is responsible for creating 1 message for each recipient and passes it on to the correct connector. At least, this takes away load from service to connector manager. But then, what if the connector manager gets overloaded or crashes? Well, then we can run several instances of a connector manager. But how will a service know if a connector manager is down or overloaded and how will it find next connector manager to use etc Will it for example be ok to use a system where one connector manager is assigned as master and if master is overloaded it will direct request from service to next slave connector manager will such redirection create extra load on the system? Extra load it will put but will it affect the system too much? I really cant see any other solution to it…… Or if master has crashed, another connector manager takes over as master? Well, if you have read all the way down to here, I am happy And hopefully you get the idea of my problems and hopefully someone can give me some good input Thank you for your time!!
