隨著網上購物消費模式熱度的不斷提高，網上銷售平臺上各種促銷手段也層出不窮，其中“秒購”已經是各種網站普遍流行的促銷方式了。“秒購”對資料的實效性和精確性要求非常高，所以通過分散式運算實現高併發資料處理應該是正確的選擇。不過，高併發也意味著高頻率的資料操作衝突，而高頻使用“鎖”又會嚴重影響效率及容易造成不可控異常，所以又被迫選擇單執行緒執行模式。單執行緒、分散式雖然表面相悖，不過如上篇博文所述：可以利用akka-cluster-sharding分片可指定呼叫的特性將一種商品的所有操作放到同一個shard上運算（因為shard即是actor，mailbox裡的運算指令是按序執行的）可容許在一個分散式環境下有多個分片來同時操作。如此可在獲取分散式運算高效率的同時又保證了資料的安全性和完整性。
雖然通過分散式運算可以實現近實時的“秒購”交易，但每個“秒購”請求都直接被髮往一個actor信箱裡等待執行，如果在一個短時間內出現超大量請求的話就很可能使shard actor mailbox超載，造成系統崩潰，這時需要一種緩衝機制根據具體負載情況來推送任務。當然，這種機制必須具備資料持久化能力，所以kafka是這個緩衝機制的一個最佳選擇。
在這篇討論裡我想通過一個“近實時交易平臺nrtxn（near realtime transaction）”專案來示範“用kafka實現分散式近實時交易”具體的設計和實現。
nrtxn的應用方案是這樣的：提供一個平臺及相關api給平臺使用者。各使用者分別將自己的產品推送到平臺資料庫由平臺託管。使用者通過平臺提供的http api向nrtxn平臺提交交易請求（如庫存扣減請求），等待或查詢平臺返回操作狀態回應。
nrtxn的系統流程如下：
使用者呼叫http api提交請求 ->
http-server將請求派送給各使用者所屬的分片workManager ->
workManager將請求寫入kafka ->
kafka reader讀出請求並按請求中交易專案將請求傳送給專案所屬的分片txnProcessor->
txnProcessor完成操作後傳送回應至workManager ->
workManager在按請求所屬的回應地址將最終回應返回給http server -> 使用者獲取請求回應
值得注意的是交易請求在到達終點actor txnProcessor傳遞中途經過了kafka，所以在txnProcessor完成資料操作後需要通過一些actor地址管理才能正確地回應到http server上的請求執行緒。
我們大致可以從請求內容瞭解平臺提供的功能：
class TxnRequest( //唯一鍵：shopId+seller+itemCode+reqTime
shopId: String = "", //託售單位（平臺使用者）
seller: String = "", //售貨組別
buyer: String = "", //購貨單位
respond: Int = 1, //1=需要返回操作結果，0=不返回response (fire-and-go)
reqTime: String = "", //請求時間（yyyyMMddHHmmssSSS）
itemCode: String = "", //交易專案
reqValue: Double = 0.00, //操作價值
remarks: String = "", //操作備註
submitTm: String = "", //提交時間 由系統填寫（yyyyMMddHHmmssSSS）
//寫入kafka之前填寫，讀出時超出指定時段視為無效請求
)
操作請求支援兩種模式：request-response, fire-and-go， 由respond值表示。無論如何，平臺都會產生相應的response並記錄在平臺資料庫裡。使用者可在使用fire-and-go模式或者系統出現異常情況時通過原request查詢對應的response。
回應response內容如下：
case class TxnResponse(
shopId: String = "", //託售單位（平臺使用者）
seller: String = "", //售貨單位
buyer: String = "", //購貨單位
reqTime: String = "", //請求時間（yyyyMMddHHmmssSSS
failed: Int = 0, //操作失敗 <> 0
reason: String = "", //失敗原因說明
rspTime: String = "", //完成時間
itemCode: String = "", //操作目標
rsrvdValue: Double = 0.00, //系統預留價值
reqValue: Double = 0, //操作價值
preValue: Double = 0.00, //操作前價值
reqValue: Double = 0.00, //操作價值
)
1、http api 傳送request到workManager分片；
pathPrefix("submit") {
val entityRef = sharding.entityRefFor(WorkManager.EntityKey, s"$shopId:$posId")
entity(as[String]) { json =>
log.step(s"received submit($shopId,$json)")
//json validation
val reqInfo = TxnServices.parseRequest(shopId,json)
if (reqInfo._1) {
val req = reqInfo._3
//mark submit time
val jsonReq = toJson(req.copy(submitTm = mgoDateToString(mgoDateTimeNow,"yyyyMMddHHmmssSSS")))
if (req.respond == 1) {
val futResp = entityRef.ask[WorkManager.Response](WorkManager.TakeWork(jsonReq, _))
.map {
case WorkManager.ResultMsg(json) => HttpEntity(MediaTypes.`application/json`, json)
case WorkManager.ErrorMsg(msg) => HttpEntity(MediaTypes.`application/json`, msg)
}
onSuccess(futResp)(complete(_))
} else {
entityRef ! WorkManager.FireupWork(jsonReq)
complete("OK")
}
} else {
log.step(s"submit($shopId,$json): ${reqInfo._2}")
complete(toJson(Map[String,Any]("sts" -> -1, "msg" -> reqInfo._2)))
}
}
} ~
如果respond=1，entityRef.ask[WorkManager.Response](WorkManager.TakeWork(jsonReq, _)) 構建了一個future session 直至收到回覆或超時。
2、workManager是一種actor，負責管理請求回應地址及寫入Kafka：
case TakeWork(jsonReq,replyTo) =>
log.step(s"WorkManager: TakeWork($jsonReq)[${entityId}]")
val req = fromJson[TxnRequest](jsonReq)
val work = Work(
req.shopId + req.seller + req.itemCode + req.reqTime,
Instant.now(),
replyTo,
)
workStates = workStates.addWork(work)
log.step(s"WorkManager: adding work: $work, new workStates: ${workStates.toString()}")
for {
_ <- TxnServices.logRequest(jsonReq, trace)
_ <- SendProducer(ps).send(new ProducerRecord[String, String](kafkaTopic, jsonReq))
} yield "Done"
log.step(s"WorkManager: current workStates: ${workStates.toString()}")
Behaviors.same
workStates是一個請求管理類，如下：
case class Work(jobId: String, startTime: Instant, replyTo: ActorRef[Response]) {
def timeElapsed(): FiniteDuration = {
Duration.between(startTime, Instant.now()).toScala
}
}
case class WorkStates(pendingWorks: List[Work] = Nil) { self =>
def cleanWorkList(timeLap: FiniteDuration) = {
val pw = pendingWorks.foldRight(List[Work]()) { (w, accu) =>
if (w.timeElapsed() > timeLap) {
accu
} else w :: accu
}
copy(pendingWorks = pw)
}
def addWork(w: Work) = copy(pendingWorks = w :: pendingWorks)
def getWork(jobId: String) = {
pendingWorks.find { w =>
w.jobId == jobId
}
}
override def toString(): String = {
pendingWorks.foldRight("") {(w,accu) =>
accu + w.jobId + " " + w.startTime + "\n"
}
}
}
workManager在收到txnProcessor完成資料操作後的狀態回應後從workStates中找出對應的請求地址進行回應：
case WorkResponse(rsp) =>
log.step(s"WorkManager: WorkResponse($rsp)[${entityId}]")
log.step(s"WorkManager: current workStates: ${workStates.toString()}")
val jobid = rsp.shopId+rsp.seller+rsp.itemCode+rsp.reqTime
log.step(s"WorkManager: finding work with jobId = $jobid")
val someWork = workStates.getWork(jobid)
log.step(s"WorkManager: WorkResponse someWork = $someWork")
if(someWork.isDefined) {
someWork.get.replyTo ! ResultMsg(toJson(Map[String, Any] ("sts" -> 0, "msg" -> "", "data" -> rsp)))
Done(loc.shopid, loc.posid, s"got WorkResponse($rsp).")
}
Behaviors.same
3、Kafka reader 讀出請求後按請求交易專案編號指定txnProcessor分片派送請求：
def start =
(1 to numReaders).toList.map { _ =>
RestartSource
.onFailuresWithBackoff(restartSource) { () => commitableSource }
// .viaMat(KillSwitches.single)(Keep.right)
.mapAsyncUnordered(32) { msg =>
TxnServices.toTxnProcessor(msg.record.value(), responseTimeout, trace)
.onComplete {
case Success(res) =>
log.step(s"AtLeastOnceReaderGroup-toTxnProcessor returns ${res}: ${msg.record.value()}")
case Failure(err) =>
log.step(s"AtLeastOnceReaderGroup-toTxnProcessor Error ${err.getMessage}: ${msg.record.value()}")
}
FastFuture.successful(msg.committableOffset)
}
.toMat(Committer.sink(committerSettings))(Keep.left)
.run()
}
mapAsyncUnordered(32)可支援32個執行緒同時執行TxnServices.toTxnProcessor。通過back pressure，確保每個執行緒在完成後才進行下一個請求讀取。
TxnServices.toTxnProcessor中通過請求產生分片的entityId，然後向對應的txnProcessor分片傳送請求：
def toTxnProcessor(jsonReq: String, askTimeout: FiniteDuration, trace: Boolean)(
implicit sharding: ClusterSharding, streamFlowTimeout: Timeout, loc: MachineId) = {
log.stepOn = trace
log.step(s"TxnServices-toTxnProcessor($jsonReq)")
val txnReq = fromJson[TxnRequest](jsonReq)
val submt = mgoStringToDateTime("yyyyMMddHHmmssSSS",txnReq.submitTm)
if (Duration.between(submt.toInstant,Instant.now()).toScala < askTimeout) {
val entityId = hashItemCode(txnReq.itemCode)
log.step(s"TxnServices-toTxnProcessor: entityId = $entityId")
val workerEntityRef = sharding.entityRefFor(TxnProcessor.EntityKey, entityId)
if (txnReq.respond == 0) {
if (txnReq.reqValue > 0)
workerEntityRef.ask[TxnProcessor.Response](TxnProcessor.RunDec(txnReq, _))
else workerEntityRef.ask[TxnProcessor.Response](TxnProcessor.RunInc(txnReq, _))
} else {
if (txnReq.reqValue > 0)
workerEntityRef.ask[TxnProcessor.Response](TxnProcessor.DecValue(txnReq, _))
else workerEntityRef.ask[TxnProcessor.Response](TxnProcessor.IncValue(txnReq, _))
}
} else FastFuture.successful("Skip")
}
在TxnServices.toTxnProcessor中對請求進行了超時驗證。系統出現故障重啟後留在kafka佇列裡的請求視為無效，因為http 端請求早已經過時了。
4、txnProcessor收到請求完成操作後產生response併傳送給workManager：
case DecValue(txnreq,replyTo) =>
log.step(s"TxnProcessor: DecValue($txnreq)[${entityId}]")
ctx.pipeToSelf(
for {
decRes <- decValue(txnreq.shopId,txnreq.itemCode,txnreq.reqValue,trace)
_ <- FastFuture.successful {
log.step(s"TxnProcessor: DecValue($txnreq)[${entityId}] DecValue result = $decRes")
}
_ <- if (decRes._1._1 == 0) {
logResponse(txnreq, decRes._2._2.asInstanceOf[Double], decRes._2._1.asInstanceOf[Double],trace)
} else FastFuture.successful {
log.step(s"TxnProcessor: DecValue($txnreq)[${entityId}] skip logging response")
}
} yield decRes
) {
case Success(res) => {
val txnrsp = TxnResponse(
txnreq.shopId,
txnreq.seller,
txnreq.buyer,
txnreq.reqTime,
res._1._1,
res._1._2,
mgoDateToString(mgoDateTimeNow,"yyyyMMddHHmmssSSS"),
txnreq.itemCode,
res._2._2.asInstanceOf[Double],
res._2._1.asInstanceOf[Double],
txnreq.reqValue,
)
entityRef ! WorkManager.WorkResponse(txnrsp)
replyTo ! DoneOp
Done(loc.shopid, loc.posid, s"DecValue response: $txnrsp")
}
case Failure(err) =>
log.error(s"TxnProcessor: DecValue Error: ${err.getMessage}[${entityId}]")
val txnrsp = TxnResponse(
txnreq.shopId,
txnreq.seller,
txnreq.buyer,
txnreq.reqTime,
1,
err.getMessage,
mgoDateToString(mgoDateTimeNow,"yyyyMMddHHmmssSSS"),
txnreq.itemCode,
0,
0,
txnreq.reqValue,
)
entityRef ! WorkManager.WorkFailure(txnrsp)
replyTo ! DoneOp
Done(loc.shopid, loc.posid, s"DecValue with error: ${err.getMessage}")
}
Behaviors.same
workManager收到WorkResponse後從workStates找到對應work的actorRef並將response傳送至http route。
另外，nrtxn支援扣減DecValue和增加IncValue。在交易過程中，一般退還庫存應該優先處理，可以用actor的priorityMailbox來實現：
prio-dispatcher {
type = Dispatcher
mailbox-type = "com.datatech.nrtx.server.TxnProcessor$PriorityMailbox"
}
class PriorityMailbox (settings: Settings, cfg: Config)
extends UnboundedPriorityMailbox( PriorityGenerator {
case x: IncValue => 0
case x: RunInc => 1
case _ => 2
}
)
...
val priorityDispatcher = DispatcherSelector.fromConfig("prio-dispatcher")
log.step(s"initializing sharding for ${TxnProcessor.EntityKey} ...")
val txnProcessorType = Entity(TxnProcessor.EntityKey) { entityContext =>
TxnProcessor(entityContext.shard,mgoClient,indexer,entityContext.entityId,keepAlive,trace)
}.withStopMessage(TxnProcessor.StopWorker)
sharding.init(txnProcessorType.withEntityProps(priorityDispatcher))
這樣，txnProcessor會優先處理IncValue,RunInc訊息。