歡迎訪問我的GitHub
這裡分類和彙總了欣宸的全部原創(含配套原始碼):https://github.com/zq2599/blog_demos
本篇概覽
-
本文是《Strimzi Kafka Bridge(橋接)實戰之》系列的第二篇,我們們直奔bridge的重點:常用介面,用實際操作體驗如何用bridge完成常用的訊息收發業務
-
官方的openapi介面文件地址 : https://strimzi.io/docs/bridge/in-development/#_openapi
-
整篇文章由以下內容構成:
- 準備工作:建立topic
- 生產訊息
- 消費訊息,strimzi bridge消費訊息的邏輯略有些特殊,就是要提前建立strimzi bridge consumer,再透過consumer來呼叫拉取訊息的介面
- 完成本篇實戰後,相信您已經可以數量的透過http來使用kafka的服務了
準備工作:建立topic
- 遺憾的是,bridge未提供建立topic的API,所以我們們還是用命令來建立吧
- ssh登入kubernetes的宿主機
- 執行建立名為bridge-quickstart-topic的topic,共四個分割槽
kubectl -n aabbcc \
run kafka-producer \
-ti \
--image=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1 \
--rm=true \
--restart=Never \
-- bin/kafka-topics.sh \
--bootstrap-server my-cluster-kafka-bootstrap:9092 \
--create \
--topic bridge-quickstart-topic \
--partitions 4 \
--replication-factor 1
- 檢查topic建立是否成功
kubectl -n aabbcc \
run kafka-producer \
-ti \
--image=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1 \
--rm=true \
--restart=Never \
-- bin/kafka-topics.sh \
--bootstrap-server my-cluster-kafka-bootstrap:9092 \
--describe \
--topic bridge-quickstart-topic
- 如下圖,可見topic的建立符合預期
- 接下來的操作都是向bridge傳送http請求完成的,我這邊宿主機的IP地址是192.168.0.1,bridge的NodePort埠號31331
檢視指定topic的詳情
- 如下請求,可以取得topicbridge-quickstart-topic的詳情
curl -X GET \
http://192.168.0.1:31331/topics/bridge-quickstart-topic
- 收到響應如下,是這個topic的詳細資訊
{
"name": "bridge-quickstart-topic",
"configs": {
"compression.type": "producer",
"leader.replication.throttled.replicas": "",
"message.downconversion.enable": "true",
"min.insync.replicas": "1",
"segment.jitter.ms": "0",
"cleanup.policy": "delete",
"flush.ms": "9223372036854775807",
"follower.replication.throttled.replicas": "",
"segment.bytes": "1073741824",
"retention.ms": "604800000",
"flush.messages": "9223372036854775807",
"message.format.version": "3.0-IV1",
"max.compaction.lag.ms": "9223372036854775807",
"file.delete.delay.ms": "60000",
"max.message.bytes": "1048588",
"min.compaction.lag.ms": "0",
"message.timestamp.type": "CreateTime",
"preallocate": "false",
"min.cleanable.dirty.ratio": "0.5",
"index.interval.bytes": "4096",
"unclean.leader.election.enable": "false",
"retention.bytes": "-1",
"delete.retention.ms": "86400000",
"segment.ms": "604800000",
"message.timestamp.difference.max.ms": "9223372036854775807",
"segment.index.bytes": "10485760"
},
"partitions": [
{
"partition": 0,
"leader": 0,
"replicas": [
{
"broker": 0,
"leader": true,
"in_sync": true
}
]
},
{
"partition": 1,
"leader": 0,
"replicas": [
{
"broker": 0,
"leader": true,
"in_sync": true
}
]
},
{
"partition": 2,
"leader": 0,
"replicas": [
{
"broker": 0,
"leader": true,
"in_sync": true
}
]
},
{
"partition": 3,
"leader": 0,
"replicas": [
{
"broker": 0,
"leader": true,
"in_sync": true
}
]
}
]
}
批次生產訊息(同步)
- 試試bridge提供的批次生產訊息的API,以下命令會生產了三條訊息,第一條透過key的hash值確定分割槽,第二條用partition引數明確指定了分割槽是2,第三條的分割槽是按照輪詢策略更新的
curl -X POST \
http://42.193.162.141:31331/topics/bridge-quickstart-topic \
-H 'content-type: application/vnd.kafka.json.v2+json' \
-d '{
"records": [
{
"key": "my-key",
"value": "sales-lead-0001"
},
{
"value": "sales-lead-0002",
"partition": 2
},
{
"value": "sales-lead-0003"
}
]
}'
- bridge響應如下,會返回每一條訊息的partition和offset,這就是同步訊息的特點,等到meta資訊更新完畢後才會返回
{
"offsets": [{
"partition": 0,
"offset": 0
}, {
"partition": 2,
"offset": 0
}, {
"partition": 3,
"offset": 0
}]
}
批次生產訊息(非同步)
- 有的場景下,例如追求高QPS並且對返回的meta資訊不關注,可以考慮非同步的方式傳送訊息,也就是說bridge收到響應後立即返回200,這種非同步模式和前面的同步模式只有一個引數的差別:在請求url中增加async=true即可
curl -X POST \
http://42.193.162.141:31331/topics/bridge-quickstart-topic?async=true \
-H 'content-type: application/vnd.kafka.json.v2+json' \
-d '{
"records": [
{
"key": "my-key",
"value": "sales-lead-0001"
},
{
"value": "sales-lead-0002",
"partition": 2
},
{
"value": "sales-lead-0003"
}
]
}'
- 沒有響應body,請您自行請求感受一下,響應明顯比同步模式快
檢視partition
- 檢視tipic的parition情況
curl -X GET \
http://42.193.162.141:31331/topics/bridge-quickstart-topic/partitions
- 響應
[{
"partition": 0,
"leader": 0,
"replicas": [{
"broker": 0,
"leader": true,
"in_sync": true
}]
}, {
"partition": 1,
"leader": 0,
"replicas": [{
"broker": 0,
"leader": true,
"in_sync": true
}]
}, {
"partition": 2,
"leader": 0,
"replicas": [{
"broker": 0,
"leader": true,
"in_sync": true
}]
}, {
"partition": 3,
"leader": 0,
"replicas": [{
"broker": 0,
"leader": true,
"in_sync": true
}]
}]
- 檢視指定partition
curl -X GET \
http://42.193.162.141:31331/topics/bridge-quickstart-topic/partitions/0
- 響應
{
"partition": 0,
"leader": 0,
"replicas": [{
"broker": 0,
"leader": true,
"in_sync": true
}]
}
- 檢視指定partition的offset情況
curl -X GET \
http://42.193.162.141:31331/topics/bridge-quickstart-topic/partitions/0/offsets
- 響應
{
"beginning_offset": 0,
"end_offset": 5
}
建立bridge consumer
- 透過bridge消費訊息,有個特別且重要的前提:建立bridge consumer,只有先建立了bridge consumer,才能順利從kafka的broker取到訊息
- 以下命令建立了一個bridge consumer,各引數的含義稍後會說明
curl -X POST http://42.193.162.141:31331/consumers/bridge-quickstart-consumer-group \
-H 'content-type: application/vnd.kafka.v2+json' \
-d '{
"name": "bridge-quickstart-consumer",
"auto.offset.reset": "earliest",
"format": "json",
"enable.auto.commit": false,
"fetch.min.bytes": 16,
"consumer.request.timeout.ms": 300000
}'
- 上述請求的引數解釋:
- 對應kafka的group為bridge-quickstart-consumer-group
- 此bridge consumer的name等於bridge-quickstart-consumer
- 引數enable.auto.commit表示是否自動提交offset,這裡設定成false,表示無需自動提交,後面的操作中會呼叫API請求來更新offset
- 引數fetch.min.bytes要特別注意,其值等於16,表示唯有訊息內容攢夠了16位元組,拉取訊息的請求才能獲取到訊息,如果訊息內容長度不到16位元組,收到的響應body就是空
- 引數consumer.request.timeout.ms也要注意,這裡我設定了300秒,如果超過300秒沒有去拉取訊息,這個消費者就會被kafka移除(被移除後如果再去拉取訊息,kafka會報錯:Offset commit cannot be completed since the consumer is not part of an active group for auto partition assignment; it is likely that the consumer was kicked out of the grou)
- 收到響應如下,instance_id表示這個bridge consumer的身份id,base_uri則是訂閱訊息時必須使用的請求地址
{
"instance_id": "bridge-quickstart-consumer",
"base_uri": "http://42.193.162.141:31331/consumers/bridge-quickstart-consumer-group/instances/bridge-quickstart-consumer"
}
如何刪除bridge consumer
- 以下命令可以刪除consumer,重點是將身份id放入path中
curl -X DELETE http://42.193.162.141:31331/consumers/bridge-quickstart-consumer-group/instances/bridge-quickstart-consumer
訂閱指定topic的訊息
- 建立bridge consumer成功後,接下來就能以這個consumer的身份去訂閱kafka訊息了
- 執行以下命令可以訂閱topic為bridge-quickstart-topic的kafka訊息,注意請求地址就是前面建立bridge consumer時返回的base_uri欄位
curl -X POST http://42.193.162.141:31331/consumers/bridge-quickstart-consumer-group/instances/bridge-quickstart-consumer/subscription \
-H 'content-type: application/vnd.kafka.v2+json' \
-d '{
"topics": [
"bridge-quickstart-topic"
]
}'
- 從上述請求body可以看出,此請求可以一次訂閱多個topic,而且還可以使用topic_pattern(正規表示式)的形式來一次訂閱多個topic
- 訂閱完成後,接下來就能主動拉取訊息了
拉取訊息
- 在拉取訊息之前,請確保已經提前生產了訊息
- 執行以下命令拉取一條訊息
curl -X GET http://42.193.162.141:31331/consumers/bridge-quickstart-consumer-group/instances/bridge-quickstart-consumer/records \
-H 'accept: application/vnd.kafka.json.v2+json'
- 然而,當您執行了上述命令後,會發現返回body為空,別擔心,這是正常的現象,按照官方的說法,拉取到的第一條訊息就是空的,這是因為拉取操作出觸發了rebalancing邏輯(rebalancing是kafka的概覽,是處理多個partition消費的操作),再次執行上述命令去拉取訊息,這下正常了,body如下
[
{
"topic": "bridge-quickstart-topic",
"key": "my-key",
"value": "sales-lead-0001",
"partition": 0,
"offset": 0
}, {
"topic": "bridge-quickstart-topic",
"key": "my-key",
"value": "sales-lead-0001",
"partition": 0,
"offset": 1
}
]
提交offset
- 前面在建立bridge consumer的時候,引數enable.auto.commit的值等於fasle,表示由呼叫方主動提交offset到kafka,因此在拉取到訊息之後,需要手動更新kafka consumer的offset
curl -X POST http://42.193.162.141:31331/consumers/bridge-quickstart-consumer-group/instances/bridge-quickstart-consumer/offsets
- 該請求無返回body,只要返回碼是204就表示成功
設定offset
- 試想這樣的場景:共生產了100條訊息,消費者也已經將這100條全部消費完畢,現在由於某種原因,需要從91條開始,重新消費91-100這10條訊息(例如需要重新計算),此時可以主動設定offset
- 先執行以下命令,生產一條訊息
curl -X POST \
http://42.193.162.141:31331/topics/bridge-quickstart-topic \
-H 'content-type: application/vnd.kafka.json.v2+json' \
-d '{
"records": [
{
"value": "sales-lead-a002-01234567890123456789",
"partition": 2
}
]
}'
- 如下圖紅色箭頭,可見當前partition已經生產了75條訊息了
- 我們們先拉取訊息,將訊息都消費掉
- 由於沒有新生產訊息,此時再拉去應該拉取不到了
- 現在執行以下請求,就可以將offset設定到74
curl -X POST http://42.193.162.141:31331/consumers/bridge-quickstart-consumer-group/instances/bridge-quickstart-consumer/positions \
-H 'content-type: application/vnd.kafka.v2+json' \
-d '{
"offsets": [
{
"topic": "bridge-quickstart-topic",
"partition": 2,
"offset": 74
}
]
}'
- 再次拉取訊息,發現74和之後的所有訊息都可以拉去到了(注意,包含了74)
- 至此,我們們對生產和傳送訊息的常用介面都已經操作了一遍,對於常規的業務場景已經夠用,接下來的文章,我們們以此為基礎,玩出更多花樣來