alpakka-kafka(8)-kafka資料消費模式實現

雪川大蟲發表於2021-08-15

上篇介紹了kafka at-least-once消費模式。kafka消費模式以commit-offset的時間節點代表不同的消費模式，分別是：at-least-once, at-most-once, exactly-once。上篇介紹的at-least-once消費模式是通過kafka自身的auto-commit實現的。事後想了想，這個應該算是at-most-once模式，因為消費過程不會影響auto-commit，kafka在每個設定的間隔都會自動進行offset-commit。如果這個間隔夠短，比整個消費過程短，那麼在完成消費過程前就已經儲存了offset，所以是at-most-once模式。不過，如果確定這個間隔一定大於消費過程，那麼又變成了at-least-once模式。具體能實現什麼消費模式並不能明確，因為auto-commit是無法從外部進行控制的。看來實現正真意義上的at-least-once消費模式還必須取得offset-commit的控制權才行。

alpakka-kafka提供了一種CommittableSource：

  def committableSource[K, V](settings: ConsumerSettings[K, V],
                              subscription: Subscription): Source[CommittableMessage[K, V], Control] {...}

從這個CommittableSource輸出的元素是CommittableMessage[K,V]：

  final case class CommittableMessage[K, V](
      record: ConsumerRecord[K, V],
      committableOffset: CommittableOffset
  )

這個CommittableMessage除原始訊息之外還提供了CommittableOffset。通過Flow或Sink都可以進行offset-commit。alpakka-kafka提供了Committer，通過Committer.sink, Committer.Flow幫助實現offset-commit，Committer.flow如下：

    Consumer
      .committableSource(consumerSettings, Subscriptions.topics(topic))
      .mapAsync(1) { msg =>
        updateStock.map(_ => msg.committableOffset)
      }
      .via(Committer.flow(committerDefaults.withMaxBatch(1)))
      .to(Sink.seq)
      .run()

或Committer.sink：

   Consumer
      .committableSource(consumerSettings, Subscriptions.topics(topic))
      .mapAsync(1) { msg =>
        updateStock.map(_ => msg.committableOffset)
      }
      .toMat(Committer.sink(committerSettings))(Keep.left)
      .run()

下面是一個具體的at-least-once示範：

  val committerSettings = CommitterSettings(sys).withMaxBatch(commitMaxBatch)


  val stkTxns = new DocToStkTxns(trace)
  val curStk = new CurStk(trace)
  val pcmTxns = new PcmTxns(trace)


  val commitableSource = Consumer
    .committableSource(consumerSettings, subscription)

  def start =
    (1 to numReaders).toList.map { _ =>
      RestartSource
        .onFailuresWithBackoff(restartSource) { () => commitableSource }
        //      .viaMat(KillSwitches.single)(Keep.right)
        .async.mapAsync(1) { msg =>
        for {
          _ <- FastFuture.successful {
            log.step(s"AtLeastOnceReaderGroup-msg: ${msg.record}")(Messages.MachineId("", ""))
          }
          _ <- stkTxns.docToStkTxns(msg.record.value())
          pmsg <- FastFuture.successful {
            log.step(s"AtLeastOnceReaderGroup-docToStkTxns: ${msg.record}")(Messages.MachineId("", ""))
            msg
          }
        } yield pmsg
      }
        .async.mapAsync(1) { msg =>
        for {
          curstks <- curStk.updateStk(msg.record.value())
          pmsg <- FastFuture.successful {
            log.step(s"AtLeastOnceReaderGroup-updateStk: curstks-$curstks")(Messages.MachineId("", ""))
            msg
          }
        } yield pmsg
      }
        .async.mapAsync(1) { msg =>
        for {
          pcm <- pcmTxns.writePcmTxn(msg.record.value())
          pmsg <- FastFuture.successful {
            log.step(s"AtLeastOnceReaderGroup-updateStk: writePcmTxn-$pcm")(Messages.MachineId("", ""))
            msg
          }
        } yield pmsg
      }
        .async.mapAsync(1) { msg =>
        for {
          _ <- pcmTxns.updatePcm(msg.record.value())
        } yield "Completed"
        FastFuture.successful(msg.committableOffset)
      }
      .toMat(Committer.sink(committerSettings))(Keep.left)
      .run()
    }

消費過程其它部分的設計考慮和實現，如多執行緒、異常處理等可參考上篇討論。

對於at-most-once消費模式的實現，alpakka-kafka提供了atMostOnceSource：

  def atMostOnceSource[K, V](settings: ConsumerSettings[K, V],
                             subscription: Subscription): Source[ConsumerRecord[K, V], Control] = {...}

下面是用這個Source實現at-most-once的示範：

 val atmostonceSource = Consumer
    .atMostOnceSource(consumerSettings, subscription)

  def start =
    (1 to numReaders).toList.map { _ =>
      RestartSource
        .onFailuresWithBackoff(restartSource) { () => atmostonceSource }
        //       .viaMat(KillSwitches.single)(Keep.right)
        .async.mapAsync(1) { msg =>
        for {
          _ <- FastFuture.successful {
            log.step(s"AtMostOnceReaderGroup-msg: $msg")(Messages.MachineId("", ""))
          }
          _ <- stkTxns.docToStkTxns(msg.value())
          pmsg <- FastFuture.successful {
            log.step(s"AtMostOnceReaderGroup-docToStkTxns: $msg")(Messages.MachineId("", ""))
            msg
          }
        } yield pmsg
      }
        .async.mapAsync(1) { msg =>
        for {
          _ <- FastFuture.successful {
            log.step(s"AtMostOnceReaderGroup-updateStk: msg: $msg")(Messages.MachineId("", ""))
          }
          curstks <- curStk.updateStk(msg.value())
          pmsg<- FastFuture.successful {
            log.step(s"AtMostOnceReaderGroup-updateStk: curstks-$curstks")(Messages.MachineId("", ""))
            msg
          }
        } yield pmsg
      }
        .async.mapAsync(1) { msg =>
        for {
          _ <- FastFuture.successful {
            log.step(s"AtMostOnceReaderGroup-writePcmTxn: msg: $msg")(Messages.MachineId("", ""))
          }
          pcm <- pcmTxns.writePcmTxn(msg.value())
          pmsg <- FastFuture.successful {
            log.step(s"AtMostOnceReaderGroup-updateStk: writePcmTxn-$pcm")(Messages.MachineId("", ""))
            msg
          }
        } yield pmsg
      }
        .async.mapAsync(1) { msg =>
        for {
          _ <- FastFuture.successful {
            log.step(s"AtMostOnceReaderGroup-updatePcm: msg: $msg")(Messages.MachineId("", ""))
          }
          _ <- pcmTxns.updatePcm(msg.value())
          _ <- FastFuture.successful {
            log.step(s"AtMostOnceReaderGroup-updateStk: updatePcm-$msg")(Messages.MachineId("", ""))
          }
        } yield "Completed"
      }
      .toMat(Sink.seq)(Keep.left)
      .run()
    }

由於offset-commit和訊息消費是兩個獨立的過程，無論如何努力都無法保證只讀一次，必須把這兩個過程合併成一個才有可能實現。所以，exactly-once可以通過資料庫系統的事務處理transaction-processing來實現，就是把offset-commit和資料更新兩個動作放到同一個事務transaction裡，通過事務處理的ACID原子特性保證兩個動作同進同退的一致性。這也意味著這個exactly-once消費模式必須在一個提供事務處理功能的資料庫系統裡實現，也代表kafka-offset必須和其它交易資料一起存放在同一種資料庫裡。mongodb4.0以上支援事務處理，可以用來作示範。

首先，先研究一下exactly-once模式的框架：

  val mergedSource = Consumer
    .plainPartitionedManualOffsetSource(consumerSettings,subscription,
      loadOffsets)
    .flatMapMerge(maxReaders, _._2)
    .async.mapAsync(1) { msg =>
          for {
            cmt <- stkTxns.stkTxnsWithRetry(msg.value(), msg.partition(), msg.offset()).toFuture().map(_ => "Completed")
            pmsg <- FastFuture.successful {
              log.step(s"ExactlyOnceReaderGroup-stkTxnsWithRetry: committed transaction-$cmt")(Messages.MachineId("", ""))
              msg
            }
          } yield pmsg
        }
        .mapAsync(1) { msg =>
          for {
            curstks <- curStk.updateStk(msg.value())
            pmsg <- FastFuture.successful {
              log.step(s"AtMostOnceReaderGroup-updateStk: curstks-$curstks")(Messages.MachineId("", ""))
              msg
            }
          } yield pmsg
        }
        .toMat(Sink.seq)(Keep.left)
        .run()
    }
  }

在上面的例子裡使用了plainPartitionedManualOffsetSource：

def plainPartitionedManualOffsetSource[K, V](
      settings: ConsumerSettings[K, V],
      subscription: AutoSubscription,
      getOffsetsOnAssign: Set[TopicPartition] => Future[Map[TopicPartition, Long]],
      onRevoke: Set[TopicPartition] => Unit = _ => ()
  ): Source[(TopicPartition, Source[ConsumerRecord[K, V], NotUsed]), Control] = {...}

getOffsetsOnAssign提供指定partition的offset（從資料庫裡讀出指定partition的offset值），如下：

  private def loadOffsets(partitions: Set[TopicPartition]): Future[Map[TopicPartition,Long]] = {
    offsetStore.getOffsets(partitions)
  }

 def getOffsets(partitions: Set[TopicPartition])(
    implicit ec: ExecutionContext) = {
    log.step(s"OffsetStore-getOffsets: ($partitions)")(Messages.MachineId("", ""))

      def getOffset(tp: TopicPartition) = {
      val query =  and(equal(KfkModels.SCHEMA.TOPIC, tp.topic()),
                         equal(KfkModels.SCHEMA.PARTITION,tp.partition()))
      def offset: Future[Seq[Document]] = colOffset.find(query).toFuture()
      for {
          docs <- offset
          ofs <- FastFuture.successful(if(docs.isEmpty) None
          else Some(Offsets.fromDocument(docs.head)))
        } yield ofs
      }
      val listFut = partitions.toList.map(getOffset)
      val futList: Future[List[Option[KfkModels.Offsets]]] = FastFuture.sequence(listFut)
      futList.map { oofs =>
        oofs.foldRight(Map[TopicPartition,Long]()){(oof,m) =>
          oof match {
            case None => m
            case ofs => m + (new TopicPartition(ofs.get.topic,ofs.get.partition) -> ofs.get.offset)
          }
        }
      }
  }

注意loadOffset的函式型別： Set[TopicPartition] => Future[Map[TopicPartition, Long]]，返回的是個Map[partition,offset]。

另外，plainPartitionedManualSource返回Source[...Source[ConsumerRecord[K, V]]，要用flatMapMerge打平：

  /**
   * Transform each input element into a `Source` of output elements that is
   * then flattened into the output stream by merging, where at most `breadth`
   * substreams are being consumed at any given time.
   *
   * '''Emits when''' a currently consumed substream has an element available
   *
   * '''Backpressures when''' downstream backpressures
   *
   * '''Completes when''' upstream completes and all consumed substreams complete
   *
   * '''Cancels when''' downstream cancels
   */
  def flatMapMerge[T, M](breadth: Int, f: Out => Graph[SourceShape[T], M]): Repr[T] =
    map(f).via(new FlattenMerge[T, M](breadth))

引數breadth代表需合併的source數量。

還有，saveOffset和writeStkTxns在同一個事務處理裡：

 def docToStkTxns(jsonDoc: String, partition: Int, offset: Long, observable: SingleObservable[ClientSession]) = {
    val bizDoc = fromJson[BizDoc](jsonDoc)
    log.step(s"TxnalDocToStkTxns-docToStkTxns: $bizDoc")(Messages.MachineId("", ""))

    observable.map(clientSession => {
      val transactionOptions = TransactionOptions.builder()
        .readPreference(ReadPreference.primary())
        .readConcern(ReadConcern.SNAPSHOT)
        .writeConcern(WriteConcern.MAJORITY)
        .build()
      clientSession.startTransaction(transactionOptions)
      val txns = StkTxns.docToTxns(dbStkTxn,dbVtx,dbVendor,bizDoc,trace)
      StkTxns.writeStkTxns(clientSession,colStkTxn,colPcm,txns,trace)
      offsetStore.saveOffset(clientSession,partition,offset)
      clientSession.commitTransaction()
      clientSession
    })

  }

注意：mongodb的事務處理必須在複製集replica-set上進行。這也很容易理解，在複製集上才方便交易回滾rollback。

完整的exactly-once實現程式碼如下：

  private def loadOffsets(partitions: Set[TopicPartition]): Future[Map[TopicPartition,Long]] = {
    offsetStore.getOffsets(partitions)
  }

  val mergedSource = Consumer
    .plainPartitionedManualOffsetSource(consumerSettings,subscription,
      loadOffsets)
    .flatMapMerge(maxReaders, _._2)

  def start = {
    (1 to numReaders).toList.map {_ =>
      RestartSource
        .onFailuresWithBackoff(restartSource) { () => mergedSource }
//        .viaMat(KillSwitches.single)(Keep.right)
        .async.mapAsync(1) { msg =>
          for {
            cmt <- stkTxns.stkTxnsWithRetry(msg.value(), msg.partition(), msg.offset()).toFuture().map(_ => "Completed")
            pmsg <- FastFuture.successful {
              log.step(s"ExactlyOnceReaderGroup-stkTxnsWithRetry: committed transaction-$cmt")(Messages.MachineId("", ""))
              msg
            }
          } yield pmsg
        }
        .async.mapAsync(1) { msg =>
          for {
            curstks <- curStk.updateStk(msg.value())
            pmsg <- FastFuture.successful {
              log.step(s"AtMostOnceReaderGroup-updateStk: curstks-$curstks")(Messages.MachineId("", ""))
              msg
            }
          } yield pmsg
        }
        .async.mapAsync(1) { msg =>
          for {
            pcm <- pcmTxns.writePcmTxn(msg.value())
            pmsg <- FastFuture.successful {
              log.step(s"AtMostOnceReaderGroup-updateStk: writePcmTxn-$pcm")(Messages.MachineId("", ""))
              msg
            }
          } yield pmsg
        }
        .async.mapAsync(1) { msg =>
          for {
            _ <- pcmTxns.updatePcm(msg.value())
          } yield "Completed"
        }
        .toMat(Sink.seq)(Keep.left)
        .run()
    }
  }

只有第一個非同步階段使用了事務處理。也就是說保證了writeStkTxns只執行一次。這個函式的功能主要是把前端產生的交易全部固化。為了避免消費過程中出現異常中斷造成了前端交易的遺失或者重複入賬，必須保證前端交易只固化一次。其它階段的資料處理都是基於已正確固化的交易記錄的。如果出現問題，可以通過重算交易記錄獲取正確的狀態。為了保證平臺執行效率，選擇了不使用事務處理的方式更新資料。

alpakka-kafka(7)-kafka應用案例，消費模式
2021-08-14
Kafka模式
alpakka-kafka(10)-用kafka實現分散式近實時交易
2022-02-16
Kafka分散式
kafka消費
2024-03-21
Kafka
Java如何實現消費資料隔離？
2022-02-17
Java
Kafka重複消費和丟失資料研究
2016-12-31
Kafka
alpakka-kafka(1)-producer
2021-02-20
Kafka
Kafka 消費組消費者分配策略
2021-01-01
Kafka
Kafka - 消費錯誤問題，多臺機器上面無法消費資料
2024-01-08
Kafka
使用Flume消費Kafka資料到HDFS
2018-11-19
Kafka
如何利用資料實現衝刺“她”消費
2022-03-17
「Kafka應用」PHP實現生產者與消費者
2021-01-20
KafkaPHP
Kafka 消費者解析
2022-05-28
Kafka
Kafka - 消費介面分析
2016-03-04
Kafka
kafka消費者消費訊息的流程
2020-11-28
Kafka
kafka 消費組功能驗證以及消費者資料重複資料丟失問題說明 3
2020-12-04
Kafka
用 gpss 從 kafka 消費資料載入到 greenplum
2020-02-26
Kafka
分享一些 Kafka 消費資料的小經驗
2018-11-20
Kafka
Kafka生產消費資料丟失和優化小結
2020-10-14
Kafka優化
MongoDB和資料流：使用MongoDB作為Kafka消費者
2018-03-26
MongoDBKafka
kafka 如何保證不重複消費又不丟失資料？
2024-06-20
Kafka
Spring整合Kafka實現批量消費和手動提交offset
2020-12-07
SpringKafka
「Kafka應用」消費者
2021-01-05
Kafka
Kafka 消費者組 Rebalance
2020-12-29
Kafka
Kafka之消費與心跳
2020-10-16
Kafka
alpakka-kafka(2)-consumer
2021-02-22
Kafka
實時數倉之Flink消費kafka訊息佇列資料入hbase
2024-03-26
Kafka佇列
聊聊如何實現一個帶冪等模板的Kafka消費者
2022-05-24
Kafka
使用BlockQueue實現生產者和消費者模式
2018-05-07
BloC模式
kafka消費者客戶端
2019-06-25
Kafka客戶端
Kafka 1.0.0 多消費者示例
2018-09-26
Kafka
flink連線消費kafka
2019-01-25
Kafka
java的kafka生產消費
2020-12-01
JavaKafka
Kafka消費與心跳機制
2020-09-28
Kafka
Kafka 入門（二）--資料日誌、副本機制和消費策略
2020-07-24
Kafka
kafka rebalance 機制與Consumer多種消費模式案例應用實戰-kafka 商業環境實戰
2018-11-11
Kafka模式
Kafka 簡單實驗二（Python實現簡單生產者消費者）
2017-10-12
KafkaPython
alpakka-kafka(6)-kafka應用案例，使用者介面
2021-08-10
Kafka
雲小課｜MRS資料分析-透過Spark Streaming作業消費Kafka資料
2023-02-23
SparkKafka

alpakka-kafka(8)-kafka資料消費模式實現

相關文章