ARTICLE | Messaging Services: what are they and why are they an important part of modern systems?
In this article, I am going to look at what messaging services are and why they have become an important part of designing distributed systems that are resilient and scalable. First, I will focus on the modern approaches for application design and where modern messaging services fit into this. Then I will be focusing on the different messaging patterns that can be used to build modern systems and finally I will be looking at the different offerings in Microsoft Azure that can be used by software developers to build a modern message-based architecture.
Modern Application Architectures
Modern architectural approaches for designing enterprise systems are geared towards building separate services which interact with each other typically using network messaging. This is known as a Service-Orientated Architecture (SOA). This term has been around for a while and is commonly used to describe the logical separation of business functions into separate services so that they can be reused by different applications and managed independently.
With the rapid migration from on-premises hosting towards cloud hosting, the concept of SOA has become more important and widely used by companies due to the distributed nature of cloud computing and due to a company’s systems potentially running across both on-premises servers and in the cloud.
Further to this, companies have started to look at how their individual applications can be split up further using horizontal slicing to separate out the discreet pieces of functionality into small, autonomous services known as Microservices. In the past, this approach has been typically used because separate teams look after different aspects of the application, such as payments, notifications, UI management, and billing. More recently, with businesses rapidly moving their systems to the cloud, there has been a strong desire to utilize a Microservices approach due to other factors, such as reducing the overall impact of live releases or to better support elastic scaling of application components.
When designing a modern application, some of the main concerns for a developer/architect are:
To design an application that fulfils the requirements for each of these points, we need to consider the communication flow within the application and how resiliency can be built in so that the application can cope with peaks in activity. This activity could be attributed to the number of users using the application at the same time, automation factors, or the impact from external systems or services. This is where modern messaging services are used to help control the message flow in an application and decouple message senders from message receivers, therefore removing the sender’s performance dependency on the receiver. This approach can be especially important when delivering a Microservices based solution.
I should take a step back here and explain what I actually mean by a ‘messaging service’. Typically, you might think of a messaging service as something like Slack with its instant messaging or something like WhatsApp. The difference here is rather than person-to-person messaging, we are talking about software component-to-component messaging which happens in the background and may have no end-user involvement.
Common Messaging Patterns
Modern applications need to use messaging patterns to support loose coupling of components to maximize scalability and fault tolerance. This can be achieved using various messaging techniques such as the ones listed below:
- Asynchronous Request-Reply – helps to decouple the front-end implementation from the back end where the front-end needs a response and the processing needs to be asynchronous.
- Publisher-Subscriber – allows a publisher component (sender) to send messages to multiple subscribers (receivers) in a one-to-many fashion.
- Queue-Based Load Leveling – a message queue that acts as a buffer between two components to smooth intermittent peaks in traffic. The queue can hold several messages so if the incoming messages exceed the receiving components message processing speed, then the messages are not lost, they are held in the queue until they can be processed.
With each of these patterns, the sending component can be fully decoupled from the receivers using a messaging service that sits in-between the different software components, providing an abstract layer of separation. This is known as a Service Broker.
Messages vs Events
There are actually two types of messaging services that offer data transmission in this way, message-driven services, and event-driven services. Both approaches have a lot of similarities in architecture and implementation but with a fundamental difference in the data that is being sent between the components. They are commonly used together when building modern systems, especially when hosted in the cloud.
Message-driven services stream raw data (or messages) that needs to be sent and processed elsewhere. This data can be many things including an invoice, or a payment request that is sent to a back-end component for processing.
Microsoft Azure Supports the following message-driven services:
Azure Storage Queues
- A cloud queuing service built into Azure Storage Accounts
- Supports a large storage capacity (greater than 80 GB)
- Message order is not guaranteed
- At least once delivery guarantee (can deliver more than one copy)
- Maximum message size – 64 KB
Azure Service Bus
- A fully managed cloud messaging service
- Supports message queues
- Supports message topics and subscriptions (publish/subscribe)
- Supports topic and queue partitioning
- Supports up to 80 GB of storage capacity
- Message order is guaranteed First-In-First-Out (FIFO)
- At least once and at most once delivery guarantee
- Maximum message size – 256 KB to 1 MB (depending on Azure pricing tier)
Event-driven services stream notifications of a change in state between components. The actual events sent over an event-driven service only contain information about the event that took place on a system. Events do not contain any raw data such as in the case of a message.
Some basic examples of events are:
- A file is uploaded to Azure Storage
- An Azure resource being deleted
- A thermostat Internet of Things (IoT) device detecting an anomaly in temperature
Microsoft Azure Supports the following event-driven services:
Azure Event Grid
- A serverless fully managed cloud event service
- Has built-in support for events from a wide range of Azure services
- Supports custom events
- Used to incorporate an event-based architecture into applications
Azure Event Hubs
- An enterprise-level event processing service
- Can ingest millions of events per second with low latency
- Distributes events to different partitions
- Can typically create between 2 and 32 partitions
- Can be used as the event source of an Azure Event Grid Service.
There are various scenarios where message-based and event-based architectures would be a good option. For instance, you may want to build in some extra resiliency into the order processing module of your application. This can be achieved by separating out the order processing logic into separate Microservices and streaming all orders (or messages) over Azure Service Bus between the components. Another example of a message-driven architecture is for fire-and-forget messages, where no response is required. For example, when an application sends an email or when an application stores diagnostic data.
A good example for using an event-driven architecture could be for reacting to data files being uploaded into Azure Storage. When a new file is uploaded, an event is published over an event-driven service which then triggers a subscriber component to extract the data from the data file, transform the data and send it to another component for processing.
Whatever a company’s needs are in terms of system scalability, performance, interoperability, extensibility and resiliency, messaging services can be used to fulfil each of these requirements with Microsoft Azure providing some managed options for software developers to utilize. With the direction of architectural approaches moving towards smaller, independently deployable services, the need to support a reliable messaging solution has become an important consideration when designing new systems.
Further information about the messaging services on offer in Azure can be found at the following link: https://azure.microsoft.com/en-us/solutions/messaging-services/