Public Cloud – Flexible Engine
Distributed Message Service for Kafka
A message queuing service based on Apache Kafka and more specifically Kafka premium instances
Distributed Message Service (DMS) for Kafka is a message queuing service based on Apache Kafka and more specifically Kafka premium instances. The computing, storage, and bandwidth resources used by an instance are exclusively occupied by the user.
Apache Kafka is distributed message middleware that features high throughput, data persistence, horizontal scalability, and stream data processing. It adopts the publish-subscribe pattern and is widely used for log collection, data streaming, online/offline system analytics, and real-time monitoring.
Scenarios 1: Asynchronous Communication
Non-core or less important messages are sent asynchronously to receiving systems, so that the main service process is not kept waiting for the results of other systems, allowing for faster responses.
For example, Kafka can be used to send a notification email and SMS message after a user has registered with a website, providing fast responses throughout the registration process.
Figure 1 Serial registration and notification
Figure 2 Asynchronous registration and notification using message queues
Scenario 2: Log Synchronization
Applications asynchronously send log messages to message queues over reliable transmission channels. Other components can read the log messages from message queues for further analysis, either in real time or offline. In addition, Kafka can collect key log information to monitor applications.
Log synchronization process :
Benefits
Rapid deployment
Simply set instance information on the DMS for Kafka console, submit your order, and a complete Kafka premium instance will be automatically created and deployed.
Service migration without modifications
DMS for Kafka is compatible with open-source Kafka APIs and supports all message processing functions of open-source Kafka. If your application services are developed based on open-source Kafka, you can easily migrate them to DMS for Kafka after specifying a few authentication configurations.
Security
Operations on Kafka premium instances are recorded and can be audited. Messages can be encrypted before storage.In addition to SASL, Virtual Private Clouds (VPCs) and security groups also provide security controls on network access.
Data reliability
Kafka premium instances support data persistence and replication. Messages can be replicated synchronously or asynchronously between replicas.
High availability
Kafka runs in clusters, enabling failover and fault tolerance so that services can run smoothly. Kafka premium instances can be deployed across AZs to further enhance service availability.
Simple O&M
The public cloud provides a whole set of monitoring and alarm services, eliminating the need for 24/7 attendance. A set of Kafka premium instance metrics are monitored and reported, including the number of partitions, topics, and accumulated messages. You can configure alarm rules and receive SMS or email notifications on how your services are running in real time.
Flexible specifications
You can customize the bandwidth and storage space for the instance and the number of partitions and replicas for topics in the instance.
Functions
The following table describes the functions visible to tenants. These functions do not include O&M features.
| Category | Function | Description |
|---|---|---|
Basic functions | Creating a queue | Creates a queue for receiving and sending messages. Two options are provided at the same time: one is selecting queue type for standard or FIFO. |
Viewing queues | Lists the brief information about all queues of a tenant. | |
Viewing queue details | Lists the details of a queue, and the details of the groups that consume messages in this queue. | |
Deleting queues | Deletes one or multiple specified queues. | |
Sending messages to a specified queue | Sends messages (including message content and user-defined message attributes) to a specified queue. | |
Creating a message consumer group for a specified queue | Creates a message consumer group with a specified name. Application can consume message from a queue using a message consumer group. | |
Consuming messages in a specified queue | With this function, you can choose the type of message, including normal, the number of automatically consumed messages and the interval between consuming messages. You can also view the content, size, and sending time of each message, and stop message consumption. | |
Confirming message consumption | Submits message consumption confirmation to a message queue after each instance of message consumption. | |
Managing queues | Manages message queues through the web-based DMS console. | |
Accessing message queues | Accesses message queues through RESTful APIs for standard queue and FIFO queue. | |
Monitoring | CES | For details, see the CES Product Description. |