Defending Direct Messaging

There are two major messaging paradigms that are widely used in distributed infrastructures: direct, also known as "brokerless", and decoupled, which is based on some central service that is responsible for transmitting messages between involved components.

The direct paradigm is a natural distributed extension of local subprogram call or a method call in object-oriented programming. In this scheme there is a calling or requesting entity, some payload associated with call parameters and the call target, which is either a subprogram or a method. Often, there is also a payload coming back as a response - such a scheme is perfectly appropriate for command-response type of interactions and is presented on the following figure.

The direct interaction approach is very natural for object-oriented systems and can be easily understood without taking into account the distribution aspect - in fact, some of the distributed frameworks work very hard to support the notion of "location transparency" where the user code is not affected by whether the caller and target are two entities in the same process or maybe in distinct processes in the distributed system.

There are many protocols that are based on this interaction paradigm - and the fact that many of them were built to reflect some client-server relationship (or request-response processing) shows that there is a design continuity that leads from local calls to this form of messaging.

The following are examples of distributed frameworks that are based on the direct interaction paradigm:

• Remote Procedure Calls (RPC), widely used on Unix systems in the days of procedural programming,
• CORBA as a standardization effort for distributed object-oriented systems,
• COM and DCOM used for component invocations on the Windows platform (also ported to other systems),
• Java RMI for homogeneous, but distributed systems built for JVM,
• HTTP with its GET and POST commands,
• All protocols used by SQL implementations,
• ... and just about any other client-server protocol in existence.

In all these examples the most important common property is that both communicating entities need to be involved at the same time and for the time of the interaction they are coupled - usually this coupling is reflected in the session-based connection mechanism, but is also visible in the fact that the communicating entities need to know each other's locations.

Even though the direct approach is very natural for command-response (or client-server) interactions, it might not be the most effective or even appropriate for systems where the data flows in only one direction. Popular examples of such installations are broadcasting systems, like TV and radio stations. Thanks to the fact that data flows only from producers to consumers it is possible to decouple them in the sense that producers can be independent from who and when is listening, as shown on the following diagram:

The broadcast approach is technically possible only if there is a medium that allows this kind of communication. Electromagnetic waves are appropriate for TV and radio stations. Sound waves are appropriate for public announcements in shopping centers or trains stations. Wall-mounted boards are appropriate for local notifications in the office.

The problem is that there is no easily accessible analogy in the computer networking world, since multicasting is usually forbidden across network segments due to its destructive potential.

In order to overcome the lack of proper broadcasting medium a new entity can be introduced with the purpose of relaying messages between all participants and thus playing the role of the missing medium. Such an entity is called a broker, a message service or a message bus and its role is shown on the following diagram:

There are many existing solutions based on this idea, with CORBA Event Service, Java Message Service (JMS), some implementations of Data Distribution Service (DDS) and AMQP being among the most well known.

This approach has the following advantages:

• The participating components are decoupled in space in the sense that they do not need to know each other's locations. Another aspect of this decoupling is the fact that producers are not particularly affected by the number of consumers that are receiving their messages - and conversely, the consumers are not affected by lifetime patterns of corresponding producers. This would not be possible with session-based direct protocols.
• The participants can be decoupled in time, so that it is not necessary for producers and consumers to be active at the same time, if the message service is able to hold messages for some time until they are delivered.
• It is possible to shift the interaction paradigm from target-centric (like in the object-oriented case) to data-centric, where the communicating parties need not be concerned by who they talk to, but rather what they talk about.

These advantages were recently appreciated by the distributed systems community, which has led to the higher popularity of such solutions and the common perception that this is the step forward in the evolution of distributed systems with the promise to solve all the problems of direct communication.

Still, notwithstanding the advantages, the above diagram clearly shows that the broker-based communication raises some important questions:

• Reliability. If the whole system is based on some central service, how much does it affect its reliability? Of course, there are ways to address this question with clusters and other redundancy-based architectures, but this will always have the higher price in terms of administration, maintenance and user training.
• Scalability. Again, if the whole system is based on a single central service, how does it affect the scalability and the freedom to add more communicating components? No matter how fast is the given implementation, there is always a limit that will become a showstopper when the number of communicating components, the frequency of messages and their volume will grow beyond the capacity of a single installation. Similarly to the reliability concerns, there are ways to address scalability with clustered installation or partitioning at the price of added complexity.
• Latency. Obviously, the indirect interaction involves more "hops" for every single message than the direct one and that, together with the inevitable processing overhead on the broker itself, surely add to the total latency that is observed in the system.
• Ability to target command-response interactions. Many message-service architectures are clearly focused on systems where the data flows in one direction and where communicating participants can be clearly defined as "producers" or "consumers". This is rarely the case and in practical systems the nodes have both roles - in particular, there are logical channels that have data flowing in both directions. A thermostat that reports its current temperature but also allows the setting of requested value is tutorial-level example of a system where the publisher is also a receiver of what is essentially the same data entity. A photo camera that sends its capture frame as a response to the explicit trigger is an example of a system where command-response interaction is more appropriate than fully decoupled stream of updates. It seems that message-service architectures lose some of the ability to handle these interactions in a natural way.

In addition to these concerns, the indirect architecture cannot really solve all the problems of direct approach, simply because instead of offering a complete replacement strategy, the indirect interaction is built in terms of direct connections between the message service and the participating components. This means that the potential problems - like the coupling between components - are not really solved, they are barely moved to some other part of the system. In practical terms, the problems of service location, session retention and flow control between participants are replaced by exactly the same kind of problems, but between participants and the broker.

What does it all mean to the software engineer that plans the architecture for new distributed systems? Which of the two approaches is the right one?

The point is, neither one is the most appropriate as a general answer - but it can be argued that the direct approach is more fundamental, as it can be used either alone, or as a building block for the broker-based solution. In other words, it is always possible to build a broker-based system with a solution that handles direct interactions, but it might not be easy to recover the advantages of direct approach if the given middleware solution targets only broker-based architectures.

This is exactly the reason why it is worthwhile to invest in direct messaging solutions. Even though they might look like things of the past, they are still defending themselves as foundation building blocks for modern architectures and as such certainly have their place in the world of distributed systems.