使用mongodb、Kafka儲存mqtt訊息

一、引言

隨著物聯網技術的迅猛發展，大量的裝置和感測器產生了海量的資料。本文利用了 MQTT、Kafka 和 MongoDB 各自的優點，滿足實時資料處理和大規模資料儲存的需求。

如圖：

二、總結

優點：

1. 可靠和解耦：

Kafka的複製機制和持久化儲存確保了資料在傳輸過程中的可靠性，即使某個節點發生故障，也不會導致資料丟失，將資料生產者和消費者解耦，各模組可以獨立擴充套件和最佳化，減少了相互影響。
2. 高可用和靈活性：

MongoDB的複製集和分片機制提供了資料的高可用性和容錯能力，保證了資料儲存的可靠性和靈活性。

缺點：

1. 複雜度高：

包含多個元件（MQTT、Kafka、MongoDB）配置、部署和維護、各元件之間的協調和整合也增加了實現的複雜性。
2. 延遲：

資料從裝置上傳到最終儲存在MongoDB之間經過多個處理環節，每個環節都可能增加一些延遲。
3. 一致性：

資料在Kafka和MongoDB之間傳遞時可能需要額外的處理機制來確保一致性。

三、實現

準備工作

使用docker-compose.yml建立Kafka服務和MongoDB，簡易程式碼如下：

version: '3.8'

networks:
  app-tier:
    driver: bridge

services:
  kafka:
    image: 'bitnami/kafka:latest'
    networks:
      - app-tier
    ports:
      - "9092:9092"
    environment:
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
    volumes:
      - kafka-data:/bitnami/kafka

  mongodb:
    image: 'mongo:latest'
    networks:
      - app-tier
    container_name: mongodb
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db

volumes:
  kafka-data:
    driver: local
  mongo-data:
    driver: local

實現步驟

1. 裝置資料上傳：

服務端程式碼

var mqttFactory = new MqttFactory();
            var mqttServerOptions = new MqttServerOptionsBuilder()
                .WithDefaultEndpointPort(1883)//監聽的埠
                .WithDefaultEndpoint()
                .WithoutEncryptedEndpoint()// 不啟用tls
                .WithDefaultCommunicationTimeout(TimeSpan.FromSeconds(10 * 1000))//10秒超時
                .WithPersistentSessions(true)//啟用session
                .WithConnectionBacklog(1000)//積累的最大連線請求數
                .Build();
            using (var mqttServer = mqttFactory.CreateMqttServer(mqttServerOptions))
            {
                AddMqttEvents(mqttServer);

                await mqttServer.StartAsync();
                Console.WriteLine("Press Enter Ctrl+C to exit.");
                Console.ReadLine();
                Console.CancelKeyPress += async (sender, e) =>
                {
                    e.Cancel = true; // 防止程序直接終止
                    await mqttServer.StopAsync();
                    Environment.Exit(0);
                };
            }

private static void AddMqttEvents(MqttServer mqttServer)
        {
            MqttServerEvents mqttEvents = new MqttServerEvents();
            mqttServer.ClientConnectedAsync += mqttEvents.Server_ClientConnectedAsync;
            mqttServer.StartedAsync += mqttEvents.Server_StartedAsync;
            mqttServer.StoppedAsync += mqttEvents.Server_StoppedAsync;
            mqttServer.ClientSubscribedTopicAsync += mqttEvents.Server_ClientSubscribedTopicAsync;
            mqttServer.ClientUnsubscribedTopicAsync += mqttEvents.Server_ClientUnsubscribedTopicAsync;
            mqttServer.ValidatingConnectionAsync += mqttEvents.Server_ValidatingConnectionAsync;
            mqttServer.ClientDisconnectedAsync += mqttEvents.Server_ClientDisconnectedAsync;
            mqttServer.InterceptingInboundPacketAsync += mqttEvents.Server_InterceptingInboundPacketAsync;
            mqttServer.InterceptingOutboundPacketAsync += mqttEvents.Server_InterceptingOutboundPacketAsync;
            mqttServer.InterceptingPublishAsync += mqttEvents.Server_InterceptingPublishAsync;
            mqttServer.ApplicationMessageNotConsumedAsync += mqttEvents.Server_ApplicationMessageNotConsumedAsync;
            mqttServer.ClientAcknowledgedPublishPacketAsync += mqttEvents.Server_ClientAcknowledgedPublishPacketAsync;
        }

客戶端程式碼

 var mqttFactory = new MqttFactory();
            var mqttClient = mqttFactory.CreateMqttClient();

            var mqttOptions = new MqttClientOptionsBuilder()
                .WithClientId("MqttServiceClient")
                .WithTcpServer("127.0.0.1", 1883)
                .Build();
            mqttClient.ConnectedAsync+=(e =>
            {
                Console.WriteLine("MQTT連線成功");
                return Task.CompletedTask;
            });

            mqttClient.DisconnectedAsync+=(e =>
            {
                Console.WriteLine("MQTT連線斷開");
                return Task.CompletedTask;
            });
            await mqttClient.ConnectAsync(mqttOptions, CancellationToken.None);
    //傳送訊息
MqttApplicationMessage applicationMessage = new MqttApplicationMessage
                    {
                        Topic = "mqtttest",
                        PayloadSegment = new ArraySegment<byte>(System.Text.Encoding.UTF8.GetBytes(input))
                    };

      var res = await mqttClient.PublishAsync(applicationMessage);

2. Kafka訊息處理：

生產者程式碼

        var config = new ProducerConfig
            {
                BootstrapServers = "localhost:9092"
            };
            using var producer = new ProducerBuilder<string, string>(config).Build();
            try
            {
                var message = new Message<string, string>
                {
                    Key = e.ClientId,
                    Value = JsonConvert.SerializeObject(e.Packet)
                };
                var deliveryResult = await producer.ProduceAsync("mqttMsg-topic", message);
                Console.WriteLine($"Delivered '{deliveryResult.Value}' to '{deliveryResult.TopicPartitionOffset}'");
            }
            catch (ProduceException<string, string> ke)
            {
                Console.WriteLine($"Delivery failed: {ke.Error.Reason}");
            }    

消費者程式碼

var config = new ConsumerConfig
            {
                GroupId = "my-consumer-group",
                BootstrapServers = "127.0.0.1:9092",
                AutoOffsetReset = AutoOffsetReset.Earliest
            };
            using var consumer = new ConsumerBuilder<string, string>(config).Build();
            consumer.Subscribe("mqttMsg-topic");
//消費訊息並儲存到mongodb
 var client = new MongoClient("mongodb://127.0.0.1:27017");
                var collection = client.GetDatabase("mqtttest").GetCollection<BsonDocument>($"history_{DateTime.UtcNow.Year}_{DateTime.UtcNow.Month}");
                while (true)
                {
                    try
                    {
                        var consumeResult = consumer.Consume(cancellationToken.Token);
                        Console.WriteLine($"收到Kafka訊息 '{consumeResult.Message.Value}' at: '{consumeResult.TopicPartitionOffset}'.");
                        var document = new BsonDocument
                        {
                            { "clientId", consumeResult.Message.Key },
                            { "JsonData", MongoDB.Bson.Serialization.BsonSerializer.Deserialize<BsonDocument>(consumeResult.Message.Value) },//不同裝置上報資料格式不一定一樣
                            { "created", DateTime.UtcNow }
                        };
                        await collection.InsertOneAsync(document);
                    }
                    catch (ConsumeException e)
                    {
                        Console.WriteLine($"處理Kafka訊息異常: {e.Error.Reason}");
                    }
                }

原始碼地址：https://github.com/jclown/MqttPersistence