Twitter Storm中Bolt訊息傳遞路徑之原始碼解讀

徽滬一郎發表於2013-09-22

本文初次發表於storm-cn的google groups中,現以blog的方式再次發表,表明本人徽滬一郎確實讀過這些程式碼,:).

Bolt作為task被executor執行,而executor是一個個的執行緒,所以executor必須存在於具體的process之中,而這個process就是worker。至於worker是如何被supervisor建立,爾後worker又如何建立executor執行緒,這些暫且按下不表。

 
假設同屬於一個Topology的Spout與Bolt分別處於不同的JVM,即不同的worker中,不同的JVM可能處於同一臺物理機器,也可能處於不同的物理機器中。為了讓情景簡單,認為JVM處於不同的物理機器中。
 
Spout的輸出訊息到達Bolt,作為Bolt的輸入會經過這麼幾個階段。
 
1. spout的輸出通過該spout所處worker的訊息輸出執行緒,將tuple輸入到Bolt所屬的worker。它們之間的通路是socket連線,用ZeroMQ實現。
2. bolt所處的worker有一個專門處理socket訊息的receive thread 接收到spout傳送來的tuple
3. receive thread將接收到的訊息傳送給對應的bolt所在的executor。 在worker內部(即同一process內部),訊息傳遞使用的是Lmax Disruptor pattern.
4. executor接收到tuple之後,由event-handler進行處理
 
下面是具體的原始碼
1. worker建立訊息接收執行緒 
 
worker.clj
 
(defn launch-receive-thread [worker]
  (log-message "Launching receive-thread for " (:assignment-id worker) ":" (:port worker))
  (msg-loader/launch-receive-thread!
    (:mq-context worker)
    (:storm-id worker)
    (:port worker)
    (:transfer-local-fn worker)
    (-> worker :storm-conf (get TOPOLOGY-RECEIVER-BUFFER-SIZE))
    :kill-fn (fn [t] (halt-process! 11))))
 
注意加亮的行會將storm.yaml中配置使用ZMQ或其它
storm.messaging.transport:"backtype.storm.messaging.zmq"
 
2. worker從socket接收到新訊息
vthread (async-loop
                 (fn []
                   (let [socket (.bind ^IContext context storm-id port)]
                     (fn []
                       (let [batched (ArrayList.)
                             init (.recv ^IConnection socket 0)]
                         (loop [packet init]
                           (let [task (if packet (.task ^TaskMessage packet))
                                 message (if packet (.message ^TaskMessage packet))]
                             (if (= task -1)
                               (do (log-message "Receiving-thread:[" storm-id ", " port "] received shutdown notice")
                                 (.close socket)
                                 nil )
                               (do
                                 (when packet (.add batched [task message]))
                                 (if (and packet (< (.size batched) max-buffer-size))
                                   (recur (.recv ^IConnection socket 1))
                                   (do (transfer-local-fn batched)
                                     0 ))))))))))
 
加亮行使用的transfer-local-fn會將接收的TaskMessage傳遞給相應的executor
 
3. transfer-local-fn
 
(defn mk-transfer-local-fn [worker]
  (let [short-executor-receive-queue-map (:short-executor-receive-queue-map worker)
        task->short-executor (:task->short-executor worker)
        task-getter (comp #(get task->short-executor %) fast-first)]
    (fn [tuple-batch]
      (let [grouped (fast-group-by task-getter tuple-batch)]
        (fast-map-iter [[short-executor pairs] grouped]
          (let [q (short-executor-receive-queue-map short-executor)]
            (if q
              (disruptor/publish q pairs)
              (log-warn "Received invalid messages for unknown tasks. Dropping... ")
              )))))))
 
用disruptor線上程之間進行訊息傳遞。
 
多費一句話,mk-transfer-local-fn表示將外部世界的訊息傳遞給本程式內的執行緒。而mk-transfer-fn則剛好在方向上反過來。
 
4. 訊息被executor處理
 
executor.clj
==========================================================
(defn mk-task-receiver [executor-data tuple-action-fn]
  (let [^KryoTupleDeserializer deserializer (:deserializer executor-data)
        task-ids (:task-ids executor-data)
        debug? (= true (-> executor-data :storm-conf (get TOPOLOGY-DEBUG)))
        ]
    (disruptor/clojure-handler
      (fn [tuple-batch sequence-id end-of-batch?]
        (fast-list-iter [[task-id msg] tuple-batch]
          (let [^TupleImpl tuple (if (instance? Tuple msg) msg (.deserialize deserializer msg))]
            (when debug? (log-message "Processing received message " tuple))
            (if task-id
              (tuple-action-fn task-id tuple)
              ;; null task ids are broadcast tuples
              (fast-list-iter [task-id task-ids]
                (tuple-action-fn task-id tuple)
                ))
            ))))))
 
加亮行中tuple-action-fn定義於mk-threads(原始檔executor.clj)中。因為當前以Bolt為例,所以會呼叫的tuple-action-fn定義於defmethod mk-threads :bolt [executor-data task-datas]
 
那麼mk-task-receiver是如何與disruptor關聯起來的呢,可以見定義於mk-threads中的下述程式碼
(let [receive-queue (:receive-queue executor-data)
              event-handler (mk-task-receiver executor-data tuple-action-fn)]
          (disruptor/consumer-started! receive-queue)
          (fn []            
            (disruptor/consume-batch-when-available receive-queue event-handler)
            0)))
 
到了這裡,訊息的傳送與接收處理路徑打通。

相關文章