Consumer Client Re-Design (翻譯)

devos發表於2016-01-11
client

注：0.9版本Kafka的一個重大改變就是consumer和producer API的重新設計。

這篇Kafka的文件大致介紹了對於consumer API重新設計時想要實現的功能。0.9版本的確實現了這些功能，具體細節有幾篇文件講了，以後會翻譯。

Motivation

We've received quite a lot of feedback on the consumer side features over the past few months. Some of them are improvements to the current consumer design and some are simply new feature/API requests. I have attempted to write up the requirements that I've heard on this wiki - Kafka 0.9 Consumer Rewrite Design
This would involve some significant changes to the consumer APIs, so we would like to collect feedback on the proposal from our community. Since the list of changes is not small, we would like to understand if some features are preferred over others, and more importantly, if some features are not required at all.

 

Thin consumer client:

  1. We have a lot of users who have expressed interest in using and writing non-java clients. Currently, this is pretty straightfoward for the SimpleConsumer but not for the high level consumer. The high level consumer does some complex failure detection and rebalancing, which is non-trivial to re-implement correctly.
  2. The goal is to have a very thin consumer client, with minimum dependencies to make this easy for the users.

Central co-ordination :

  1. The current version of the high level consumer suffers from herd and split brain problems, where multiple consumers in a group run a distributed algorithm to agree on the same partition ownership decision. Due to different view of the zookeeper data, they run into conflicts that makes the rebalancing attempt fail. But there is no way for a consumer to verify if a rebalancing operation completed successfully on the entire group. This also leads to some potential bugs in the rebalancing logic, for example,https://issues.apache.org/jira/browse/KAFKA-242
  2. This can be mediated by moving the failure detection and rebalancing logic to a centralized highly-available co-ordinator - Kafka 0.9 Consumer Rewrite Design

We think the first two requirements are the prerequisite of the rest. So we have proceeded by trying to design a centralized coordinator for consumer rebalancing without Zookeeper, for details please read here.

Allow manual partition assignment

  1. There are a number of stateful data systems would like to manually assign partitions to consumers. The main motive is to enable them to keep some local per-partition state since the mapping from their consumer to partition never changes; also there are some use cases where it makes sense to co-locate brokers and consumer processes, hence would be nice to optimize the automatic partition assignment algorithm to consider co-location. Examples of such systems are databases, search indexers etc
  2. A side effect of this requirement is wanting to turn off automatic rebalancing in the high level consumer.
  3. This feature depends on the central co-ordination feature since it is cannot be correctly and easily implemented with the current distributed co-ordination model.

Allow manual offset management

  1. Some systems require offset management in a custom database, at specific intervals. Overall, the requirement is to have access to the message metadata like topic, partition, offset of the message, and to be able to provide per-partition offsets on consumer startup.
  2. This would require designing new consumer APIs that allow providing offsets on startup and return message metadata with the consumer iterator.
  3. One thing that needs to be thought through is if the consumer client can be allowed to pick manual offset management for some, but not all topics. One option is to allow the consumer to pick one offset management only. This could potentially make the API a bit simpler
  4. This feature depends on the central co-ordination feature since it is cannot be correctly and easily implemented with the current distributed co-ordination model.

Invocation of user specified callback on rebalance

  1. Some applications maintain transient per-partition state in-memory. On rebalance operation, they would need to “flush” the transient state to some persistent storage.
  2. The requirement is to let the user plugin some sort of callback that the high level consumer invokes when a rebalance operation is triggered.
  3. This requirement has some overlap with the manual partition assignment requirement. Probably, if we allow manual partition assignment, such applications might be able to leverage that to flush transient state. But, the issue is that these applications do want automatic rebalancing and might not want to use the manual partition assignment feature.

Non blocking consumer APIs

    1. This requirement is coming from stream processing applications that implement high-level stream processing primitives like filter by, group by, join operations on kafka streams.
    2. To facilitate stream join operations, it is desirable that Kafka provides non-blocking consumer APIs. Today, since the consumer streams are essentially blocking, these sort of stream join operations are not possible.
    3. This requirement seems to involve some significant redesign of the consumer APIs and the consumer stream logic. So it will be good to give this some more thought.

 

動機

在過去的幾個月內，我們收到了很多關於消費者端的特性的反饋。其中有一些是對於當前的consumer設計的改進，有一些是對新的特性/API的需求。我嘗試把獲取的需求寫到一起到這個wiki上- Kafka 0.9 Consumer Rewrite Design

這將會關係到一些對於consumer API的顯著的改變，所以我們需要從社群中收集對這些提議的反饋。因為改變列表並不上，我們需要了解是否有些特性比其它的更受歡迎，更重要的，有哪些特性是根本不需要的。

 

Central co-ordination:

（叫contral co-ordiantion是為與當前的使用zk做的distributed co-cordination相比較）

 

１. high level consumer容易受到herd以及split brain problem的影響（在consumer執行分散式演算法來決定partition的歸屬時）．並且，high level consumer在rebalance的過程中容易產生衝突，從而使得rebalance失敗．但是consumer group裡的單個consumer無法知道一個rebalance操作是否在整個group級別上成功執行．這也造成了在rebalance邏輯上存在一些潛在的bug，比如https://issues.apache.org/jira/browse/KAFKA-242

2. 所以需要把failure detection和rebalance的功能移到一個centralized high-available co-coordinator上 Kafka 0.9 Consumer Rewrite Design

 

這兩個問題的解決是處理下面幾個問題的前提，所以需要設計一個不再用Zookeeper來做consumer rebalance的中央協調系統．

允許手動地分配partition給consumer

１．在有些情況下，使用者想要手動地分配partition給consumer．主要原因是，在有些情況下，consumer和partition的對應關係是不會變的，這樣就能允許維護一個local per-partition state；有些情況下，使用者也需要把broker和conumser　process放在一起，這樣就需要使得automatic partition assignment演算法考慮co-location的事．這樣的系統包括資料庫，search indexeers等　．

２．要想實現這個功能，附加著就需要可以關閉high level consumer的 automatic rebalance功能

３. 這個功能依賴於contral co-cordination功能，因為在當前的分散式協調模型中，實現這個功能不容易．

 

允許手動的offset管理

  1. 有些系統需要使用自己的資料庫管理offset，按照指定的間隔．總的來說，這就需要使用者能獲取到訊息的metadata，比如topic, partition ,offset, 並且可以在consumer啟動的時候為每個分割槽指定offset．

  2. 這就需要設計新的consumer API，使它能夠在啟動時設定offset，在consumer iterator中返回訊息的metadata.

  3. 需要仔細考慮的情況一個情況是是否允許一個conumser對一些topic使用手動的offset管理，對另外一些topic使用自動的offset管理．可以考慮的一個選擇時，只允許consumer選擇一種offset管理方式，這樣會使得它的API簡單一些．

  4. 這個特性依賴於contral co-cordiante特性，因為當前的分散式協調模型不能正確地以及容易地實現這個特性．

 

允許在rebalance時指定回撥函式

 

  1. 有些程式在記憶體中儲存於臨時的per-partition狀態．在rebalance操作發生時，這些程式需要把transient state刷到永續性儲存中．

  2. 這就需要可以讓使用者指定一個回撥，當rebalance操作觸發時，就會執行這個回撥．

  3. 這個需求和手動partition分配的需求有所重複．可能會，如果我們允行手動分配partition，程式或許可以藉助那個功能來flush traisient state．但是，問題在於，有些程式的確想使用自動的rebalancing,　而不想用手動分配partition的特性．

提供非阻塞的consumer API

  1. 這個需求來自於流處理程式，這些程式想要在Kafka流上實現高層流處理的基本操作，比如：filter by, group by, join．

  2. 為了使流的join操作更容易，需要Kafka提供非阻塞的consumer API．現在，因為consumer stream本質是阻塞的，所以實現這種stream join操作是不可能的．

  3. 這個需要看起來需要對consumer API以及consumer stream logic進行可觀的重新設計．所以多花些時間考慮是有益的．

 

總結：

新的consumer API總的來說

1. 在分散式協調方面更健壯

2. 給使用者提供了更多關於訊息的後設資料

3. 使用者可以自己管理offset和partition分配

4. 提供了非阻塞的API

5. 為了做到這些，它使用了contral co-ordinate系統替代了之前的分散式協調系統。 

這樣，這個新的consumer API就既可以完全替代之前的high level consumer， 又提供了以前只有simple API才能提供的一些功能(但是卻隱藏了一些直接使用simple API的複雜性)，所以會是一個更通用的API。

相關文章