Programming Distributed Systems with YAMI4
8.2 Message Ordering
The subject of message ordering is of particular importance in many systems and the complete understanding of what factors can influence the ordering of messages is necessary for proper system design at the high level.
A simple client-server system where the client sends a single message and waits for server response (note that the calculator example belongs to this category) need not be concerned with message ordering issues - at any given point in time there is at most one outgoing message being handled by the agent, so the ordering is obvious.
Things become more complicated when there are many messages being processed at the same time.
The following are important facts that influence the ordering of messages:
- Each distinct target, as seen by the sending agent, has a separate communication channel with a single outgoing queue. This queue keeps messages according to their priorities (FIFO within priorities). The priority information is only used to manage the outgoing queue and is not propagated across the wire.
- Many channels can be handled at a time by the I/O worker thread, if the channels are non-blocking (which is the default setting).
- No fairness is guaranteed with regard to the throughput of each channel - that is, there is no guarantee that channels will transmit data with the same speed and there is no relative ordering between separate channels even in the presence of different priority values.
- Each received message is appended to the common incoming queue when it is assembled. Messages coming from separate channels have no relative ordering and the priority information as used by the sender is no longer visible.
- One or more dispatching threads can consume messages from the incoming queue and deliver them to the application code, where they are processed. No fairness is guaranteed for those dispatching threads, so after picking messages from the queue they can continue their work with no relation to other threads.
Taking into account the above factors, the following conditions can be used to define the ordering guarantees for messages in YAMI4:
- IF the sender (client) application has a single user thread that produces messages
- AND those messages have equal priorities
- AND those messages are sent to the same remote target
- AND the receiver (server) application has a single dispatcher thread that delivers messages to the application code.
If all conditions above are met, then the sender and the receiver will see messages in the same order. Any form of concurrency or processing or transport parallelism can introduce hazards that might make the relative message ordering impossible to predict. In particular, multithreading on any side or the use of several communication paths can introduce opportunities for the messages to lose their original or intended ordering.
