Public Cloud – Flexible Engine
Distributed Message Service (DMS) – Message queuing service
Fully hosted on the cloud computer platform
DMS supports large-scale and high-concurrency access with the use of distributed cluster technology
This service decouples components of a cloud application, delivering substantial cost benefits.
DMS provides an easy-to-use web-based console for managing message queues and application programming interfaces (APIs) for accessing messages. Using the DMS console, tenants can quickly create queues and perform message production and consumption tests. User applications can then directly call RESTful APIs, making the DMS service immediately available for applications.
An all-round monitoring and maintenance system has been launched to ensure reliable running of DMS.
The following figure shows the architecture of DMS. Tenants have the choice to access a message service cluster, either using the DMS console to operate in website or by having business system make API calls to the message service cluster. All messages stored in isolated message service area are secured against unauthorized access.
Multi-tenant sharing
As a cloud-native service, DMS is directly available for use without deploying any hardware or software.
Message queue management
DMS provides a web-based console for tenants, allowing them to manage message queues.
Multiple queue type
Support standard queue, FIFO queue. FIFO queue belong to strict message order-preserving queue.
Simple APIs
DMS provides simple RESTful APIs for accessing messages on standard queue and FIFO queue.
High-security access control
Integrating with the IAM system, DMS provides certificate-based unified access authentication, allowing only authorized tenants to access appropriate message queues. DMS security mechanism determine the rights of tenants by default, tenants can only access and operate their own created queue.
High-reliability storage
With built-in reliable mechanisms, DMS stores queue messages in different segment files including message attribute and body content. Each message has three copies in the DMS storage. It also supports fault tolerance, fault discovery, and online data migration.
Fault proofing
DMS has a built-in optimized processing mechanism to ensure reliable and correct message consumption.
High scalability
DMS has its nodes deployed on distributed clusters, making it highly scalable.
Cloud Eye Service (CES)
CES, integrated into DMS, monitors the usage of message queues. It reports an alarm if CES detects that too many messages is waiting to be processed.
Benefits
Easy to use
After you buy the DMS service, you can use DMS to create queues, and send and receive messages via the web-based console using only three APIs.
High-security
With the identity and access management (IAM) system integrated, DMS provides an identity authentication mechanism. This mechanism prevents unauthorized users from accessing queue messages, giving tight protection for messages.
Support for massive queues
The number of queues supported by DMS is scalable, satisfying the ever-increasing requirements for message queues.
Flexible access
DMS supports access inside the cloud, making it applicable to a variety of service scenarios.
High-reliability
Technologies used by DMS have been maturely and widely used in the cloud service systems of carrier-grade customers across the globe, ensuring reliable, round-the-clock running of the DMS service.
Multiple communication modes
DMS supports point-to-point (P2P) and subscribe/publish communication modes, meeting the requirements of diversified services.
Transparent operation and maintenance (O&M)
The entire O&M process of DMS is transparent to users without affecting user experience.
Use Cases
DMS can be applied in a wide variety of service scenarios, including but not limited to:
- Service decoupling
- Inter-system eventual consistency
- Off-peak traffic control
- Log synchronization
- System communication
Service Decoupling
Scenario: e-commerce system
Description: In an e-commerce system, DMS can be used to provide message notifications for unimportant services that are dependent on other systems. This allows key services to proceed without waiting for other systems, decoupling them from unimportant services. For example, the order processing (OP) system puts order information in message queues of DMS, and then the inventory and delivery management systems read order information from message queues.
Inter-System Eventual Consistency
Scenario: trading or payment system
Description: In a trading or payment system, transaction status (transaction succeeded or failed) must be the same in different subsystems/modules. Therefore, reliable message transmission channels (without data loss) are required between subsystems/modules to ensure service continuity. DMS can provide high-reliability data transmission between subsystems/modules, keeping transaction status in-between synchronized in a simple and cost-efficient way.
Off-Peak Traffic Control
Scenario: e-commerce system or large website
Description: In an e-commerce system or large website, there is a capability gap between upstream and downstream systems. Traffic bursts of upstream systems with a high processing capability may have a huge impact on downstream systems with lower processing capability. For example, online sales promotions involve tremendous traffic flooding into e-commerce systems. DMS can be used to buffer order information, relieving pressure on downstream systems. It provides 3-day buffer for hundreds of millions of messages, allowing message consumer systems to process messages during off-peak periods.
Log Synchronization
Scenario: log analysis system
Description: A log analysis system needs to collect, process, and analyze logs. For such a log analysis system, applications send log messages to DMS through reliable asynchronous transmission channels. Then, other components can read log messages from message queues for further analysis, either in real time or offline. In addition, DMS can be used to collect key log information required for monitoring.
System Communication
Scenario: System requiring message exchanges
Description: Services load different requirements on communication between systems. These include the queue communication mode between two systems and publish/subscribe communication mode among multiple systems. DMS can be used to achieve P2P message communication between two systems and publish/subscribe message communication among multiple systems. One system publishes messages, while other systems can subscribe to and consume the same message.
Usage suggestion
Following the suggestions provided can help tenants use DMS service more efficiently and reasonably, that is a supplemental description of the previous section, to provide the best practices for using DMS.
Guidance for using DMS service
Before messages are processed, DMS allows them to be stored in queues. Users do not need to worry about message storage or management. All they need to do is to focus on developing applications.
Users can use DMS in their applications through the following methods:
- Integrating DMS into the web services of other enterprise cloud infrastructures, making applications more reliable, flexible, and scalable.
- Using DMS to create a queue. In this queue, every message is regarded as a task and a process is required to complete such tasks. One or more processes can read and execute tasks in the queue.
- Connecting message queues to micro services to build the micro service architecture.
- Storing notifications of important events of services in DMS message queues. Each event has a matching message in the queue, and applications that need to know the event information can read and process the message.
When using DMS, users can take some optimization measures to improve the API call efficiency, as well as message sending and consumption efficiency.
To improve the message sending and consumption efficiency, users are advised to use the batch message sending and consumption mode, which can effectively lower the number of API calls and minimize service fees.
Best Practices
During message sending and consumption, DMS service, producers, and consumers collaborate to ensure service reliability. Best practice for producers and consumers of DMS include the following:
- After messages are sent, the producer checks for message sent confirmation returned from DMS. If message sending failed, the producer re-sends the messages.
- After messages are produced, the producer waits for the message sending API to send an acknowledgement (ACK) to determine whether messages were sent successfully. If any abnormality occurs during message sending, the producer does not receive an ACK. In this case, the producer determines whether to re-send messages. If the producer receives a successful ACK, the messages have been reliably stored by DMS.
- During message consumption, the consumer consumes messages and then confirms whether messages have been consumed successfully.
- Produced messages are stored in the DMS service storage medium. During message consumption, the consumer obtains messages stored in the DMS service. Next, the consumer consumes the messages, records message consumption status as successful or failed. The consumer then submits the consumption status to the DMS service. Based on the consumption status, the DMS service determines whether to consume the next batch of messages or re-consume the unsuccessfully consumed messages.
- During the message consumption process, the message consumption status may not be successfully submitted due to some abnormalities. In this case, the corresponding batch of messages will be re-obtained by the consumer in the next message consumption request.
- Based on the preceding mechanism, messages may be sent or consumed repeatedly.
- DMS support the first in first out (FIFO) message delivery queue mode. The standard type queue cannot guarantee that messages are strictly ordered, but FIFO queue is strictly ordered queue. So please select the appropriate queue type based on the business scenario.
Functions
The following table describes the functions visible to tenants. These functions do not include O&M features.
| Category | Function | Description |
|---|---|---|
Basic functions | Creating a queue | Creates a queue for receiving and sending messages. Two options are provided at the same time: one is selecting queue type for standard or FIFO. |
Viewing queues | Lists the brief information about all queues of a tenant. | |
Viewing queue details | Lists the details of a queue, and the details of the groups that consume messages in this queue. | |
Deleting queues | Deletes one or multiple specified queues. | |
Sending messages to a specified queue | Sends messages (including message content and user-defined message attributes) to a specified queue. | |
Creating a message consumer group for a specified queue | Creates a message consumer group with a specified name. Application can consume message from a queue using a message consumer group. | |
Consuming messages in a specified queue | With this function, you can choose the type of message, including normal, the number of automatically consumed messages and the interval between consuming messages. You can also view the content, size, and sending time of each message, and stop message consumption. | |
Confirming message consumption | Submits message consumption confirmation to a message queue after each instance of message consumption. | |
Managing queues | Manages message queues through the web-based DMS console. | |
Accessing message queues | Accesses message queues through RESTful APIs for standard queue and FIFO queue. | |
Monitoring | CES | For details, see the CES Product Description. |