### 歡迎訪問我的GitHub
> 這裡分類和彙總了欣宸的全部原創(含配套原始碼)：[https://github.com/zq2599/blog_demos](https://github.com/zq2599/blog_demos)

- 基於Docker可以很輕鬆的搭建一個kafka叢集，其他機器上的應用如何使用這個kafka叢集服務呢？本次實戰就來解決這個問題。

### 基本情況
- 整個實戰環境一共有三臺機器，各自的職責如下圖所示：

| IP地址 | 身份 | 備註 |
|--|--|--|
| 192.168.1.102 | 訊息生產者 | 這是個spring boot應用，<br>應用名稱是kafka01103producer，<br>01103代表kafka版本0.11.0.3 |
| 192.168.1.101 | Docker server | 此機器上安裝了Docker，並且執行了兩個容器：zookeeper和kafka |
| 192.168.1.104 | 訊息消費者 | 這是個spring boot應用，<br>應用名稱是kafka01103consumer，<br>01103代表kafka版本0.11.0.3 |

- 整個環境的部署情況如下圖：
![在這裡插入圖片描述](https://img-blog.csdnimg.cn/20190101141259335.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70)

### 版本資訊
1. 作業系統：Centos7
2. docker：17.03.2-ce
3. docker-compose：1.23.2
4. kafka：0.11.0.3
5. zookeeper：3.4.9
6. JDK：1.8.0_191
7. spring boot：1.5.9.RELEASE
8. spring-kafka：1.3.8.RELEASE

### 重點介紹
- 本次實戰有幾處重點需要注意：
1. spring-kafka和kafka的版本匹配問題，請關注官方文件：https://spring.io/projects/spring-kafka
2. kafka的kafka的advertised.listeners配置，應用透過此配置來連線broker；
3. 應用所在伺服器要配置host，才能連線到broker；

- 接下來開始實戰吧；

### 配置host
- 為了讓生產和消費訊息的應用能夠連線kafka成功，需要配置應用所在伺服器的/etc/hosts檔案，增加以下一行內容：
```shell
192.168.1.101 kafka1
```
- 192.168.1.101是docker所在機器的IP地址；

- 請注意，生產和消費訊息的應用所在伺服器都要做上述配置；

- 可能有的讀者在此會有疑問：為什麼要配置host呢？我把kafka配置的advertised.listeners配置成kafka的IP地址不就行了麼？這樣的配置我試過，但是用kafka-console-producer.sh和kafka-console-consumer.sh連線kafka的時候會報錯"LEADER_NOT_AVAILABLE"。

### 在docker上部署kafka
- 在docker機器上編寫docker-compose.yml檔案，內容如下：
```yml
version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka1:
    image: wurstmeister/kafka:2.11-0.11.0.3
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "topic001:2:1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
```
- 上述配置中有兩處需要注意：
- 第一，KAFKA_ADVERTISED_LISTENERS的配置，這個引數會寫到kafka配置的advertised.listeners這一項中，應用會用來連線broker；
- 第二，KAFKA_CREATE_TOPICS的配置，表示容器啟動時會建立名為"topic001"的主題，並且partition等於2，副本為1；

- 在docker-compose.yml所在目錄執行命令**docker-compose up -d**，啟動容器；
- 執行命令**docker ps**，可見容器情況，kafka的容器名為temp_kafka1_1：

```shell
[root@hedy temp]# docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                                                NAMES
ba5374d6245c        wurstmeister/zookeeper             "/bin/sh -c '/usr/..."   About an hour ago   Up About an hour    22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   temp_zookeeper_1
2c58f46bb772        wurstmeister/kafka:2.11-0.11.0.3   "start-kafka.sh"         About an hour ago   Up About an hour    0.0.0.0:9092->9092/tcp                               temp_kafka1_1
```

- 執行以下命令可以檢視topic001的基本情況：

```shell
docker exec temp_kafka1_1 \
kafka-topics.sh \
--describe \
--topic topic001 \
--zookeeper zookeeper:2181
```

- 看到的資訊如下：

```shell
Topic:topic001	PartitionCount:2	ReplicationFactor:1	Configs:
	Topic: topic001	Partition: 0	Leader: 1001	Replicas: 1001	Isr: 1001
	Topic: topic001	Partition: 1	Leader: 1001	Replicas: 1001	Isr: 1001
```

### 原始碼下載
- 接下來的實戰是編寫生產訊息和消費訊息的兩個應用的原始碼，您可以選擇直接從GitHub下載這兩個工程的原始碼，地址和連結資訊如下表所示：

| 名稱      |   連結 |   備註|
| :-------- | :----| :----|
| 專案主頁| https://github.com/zq2599/blog_demos | 該專案在GitHub上的主頁 |
| git倉庫地址(https)| https://github.com/zq2599/blog_demos.git | 該專案原始碼的倉庫地址，https協議 |
| git倉庫地址(ssh)| git@github.com:zq2599/blog_demos.git | 該專案原始碼的倉庫地址，ssh協議 |

- 這個git專案中有多個資料夾，本章原始碼在kafka01103consumer和kafka01103producer這兩個資料夾下，如下圖紅框所示：
![在這裡插入圖片描述](https://img-blog.csdnimg.cn/20190101125129128.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70)

- 接下來開始編碼：

### 開發生產訊息的應用
- 建立一個maven工程，pom.xml內容如下：
```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelversion>4.0.0</modelversion>
    <parent>
        <groupid>org.springframework.boot</groupid>
        <artifactid>spring-boot-starter-parent</artifactid>
        <version>1.5.9.RELEASE</version>
        <relativepath></relativepath> 
    </parent>
    <groupid>com.bolingcavalry</groupid>
    <artifactid>kafka01103producer</artifactid>
    <version>0.0.1-SNAPSHOT</version>
    <name>kafka01103producer</name>
    <description>Demo project for Spring Boot</description>

<dependencies>
        <dependency>
            <groupid>org.springframework.boot</groupid>
            <artifactid>spring-boot-starter-web</artifactid>
        </dependency>
        <dependency>
            <groupid>org.springframework.boot</groupid>
            <artifactid>spring-boot-starter</artifactid>
        </dependency>
        <dependency>
            <groupid>org.springframework.kafka</groupid>
            <artifactid>spring-kafka</artifactid>
            <version>1.3.8.RELEASE</version>
        </dependency>
		<dependency>
            <groupid>com.alibaba</groupid>
            <artifactid>fastjson</artifactid>
            <version>1.2.28</version>
        </dependency>

<dependency>
            <groupid>org.springframework.boot</groupid>
            <artifactid>spring-boot-starter-test</artifactid>
            <scope>test</scope>
        </dependency>
    </dependencies>

<build>
        <plugins>
            <plugin>
                <groupid>org.springframework.boot</groupid>
                <artifactid>spring-boot-maven-plugin</artifactid>
            </plugin>
        </plugins>
    </build>
</project>
```
- 再次強調spring-kafka版本和kafka版本的匹配很重要；
- 配置檔案application.properties內容：

```properties
#kafka相關配置
spring.kafka.bootstrap-servers=kafka1:9092
#設定一個預設組
spring.kafka.consumer.group-id=0
#key-value序列化反序列化
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
#每次批次傳送訊息的數量
spring.kafka.producer.batch-size=65536
spring.kafka.producer.buffer-memory=524288
```

- 傳送訊息的業務程式碼只有一個MessageController類：

```java
package com.bolingcavalry.kafka01103producer.controller;

import com.alibaba.fastjson.JSONObject;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.util.concurrent.ListenableFuture;
import org.springframework.web.bind.annotation.*;

import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.UUID;

/**
 * @Description: 接收web請求，傳送訊息到kafka
 * @author: willzhao E-mail: zq2599@gmail.com
 * @date: 2019/1/1 11:44
 */
@RestController
public class MessageController {

@Autowired
    private KafkaTemplate kafkaTemplate;

@RequestMapping(value = "/send/{name}/{message}", method = RequestMethod.GET)
    public @ResponseBody
    String send(@PathVariable("name") final String name, @PathVariable("message") final String message) {
        SimpleDateFormat simpleDateFormat=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        String timeStr = simpleDateFormat.format(new Date());

JSONObject jsonObject = new JSONObject();
        jsonObject.put("name", name);
        jsonObject.put("message", message);
        jsonObject.put("time", timeStr);
        jsonObject.put("timeLong", System.currentTimeMillis());
        jsonObject.put("bizID", UUID.randomUUID());

String sendMessage = jsonObject.toJSONString();

ListenableFuture future = kafkaTemplate.send("topic001", sendMessage);
        future.addCallback(o -> System.out.println("send message success : " + sendMessage),
                throwable -> System.out.println("send message fail : " + sendMessage));

return "send message to [" +  name + "] success (" + timeStr + ")";
    }
}

```

- 編碼完成後，在pom.xml所在目錄執行命令**mvn clean package -U -DskipTests**，即可在target目錄下發現檔案kafka01103producer-0.0.1-SNAPSHOT.jar，將此檔案複製到192.168.1.102機器上；

- 登入192.168.1.102，在檔案kafka01103producer-0.0.1-SNAPSHOT.jar所在目錄執行命令**java -jar kafka01103producer-0.0.1-SNAPSHOT.jar**，即可啟動生產訊息的應用；

### 開發消費訊息的應用
- 建立一個maven工程，pom.xml內容如下：
```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelversion>4.0.0</modelversion>
    <parent>
        <groupid>org.springframework.boot</groupid>
        <artifactid>spring-boot-starter-parent</artifactid>
        <version>1.5.9.RELEASE</version>
        <relativepath></relativepath> 
    </parent>
    <groupid>com.bolingcavalry</groupid>
    <artifactid>kafka01103consumer</artifactid>
    <version>0.0.1-SNAPSHOT</version>
    <name>kafka01103consumer</name>
    <description>Demo project for Spring Boot</description>

<dependencies>
        <dependency>
            <groupid>org.springframework.boot</groupid>
            <artifactid>spring-boot-starter</artifactid>
        </dependency>
        <dependency>
            <groupid>org.springframework.kafka</groupid>
            <artifactid>spring-kafka</artifactid>
            <version>1.3.8.RELEASE</version>
        </dependency>

<dependency>
            <groupid>org.springframework.boot</groupid>
            <artifactid>spring-boot-starter-test</artifactid>
            <scope>test</scope>
        </dependency>
    </dependencies>

<build>
        <plugins>
            <plugin>
                <groupid>org.springframework.boot</groupid>
                <artifactid>spring-boot-maven-plugin</artifactid>
            </plugin>
        </plugins>
    </build>
</project>
```
- 再次強調spring-kafka版本和kafka版本的匹配很重要；
- 配置檔案application.properties內容：

```properties
#kafka相關配置
spring.kafka.bootstrap-servers=192.168.1.101:9092
#設定一個預設組
spring.kafka.consumer.group-id=0
#key-value序列化反序列化
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
#每次批次傳送訊息的數量
spring.kafka.producer.batch-size=65536
spring.kafka.producer.buffer-memory=524288
```

- 消費訊息的業務程式碼只有一個Consumer類，收到訊息後，會將內容內容和訊息的詳情列印出來：

```java
@Component
public class Consumer {
    @KafkaListener(topics = {"topic001"})
    public void listen(ConsumerRecord, ?> record) {
        Optional> kafkaMessage = Optional.ofNullable(record.value());

if (kafkaMessage.isPresent()) {

Object message = kafkaMessage.get();

System.out.println("----------------- record =" + record);
            System.out.println("------------------ message =" + message);
        }
    }
}
```

- 編碼完成後，在pom.xml所在目錄執行命令**mvn clean package -U -DskipTests**，即可在target目錄下發現檔案kafka01103consumer-0.0.1-SNAPSHOT.jar，將此檔案複製到192.168.1.104機器上；

- 登入192.168.1.104，在檔案kafka01103consumer-0.0.1-SNAPSHOT.jar所在目錄執行命令**java -jar kafka01103consumer-0.0.1-SNAPSHOT.jar**，即可啟動消費訊息的應用，控制檯輸出如下：

```shell
2019-01-01 13:41:41.747  INFO 1422 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version : 0.11.0.2
2019-01-01 13:41:41.748  INFO 1422 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId : 73be1e1168f91ee2
2019-01-01 13:41:41.787  INFO 1422 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 
2019-01-01 13:41:41.912  INFO 1422 --- [           main] c.b.k.Kafka01103consumerApplication      : Started Kafka01103consumerApplication in 11.876 seconds (JVM running for 16.06)
2019-01-01 13:41:42.699  INFO 1422 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : Discovered coordinator kafka1:9092 (id: 2147482646 rack: null) for group 0.
2019-01-01 13:41:42.721  INFO 1422 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : Revoking previously assigned partitions [] for group 0
2019-01-01 13:41:42.723  INFO 1422 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked:[]
2019-01-01 13:41:42.724  INFO 1422 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : (Re-)joining group 0
2019-01-01 13:41:42.782  INFO 1422 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : Successfully joined group 0 with generation 5
2019-01-01 13:41:42.788  INFO 1422 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : Setting newly assigned partitions [topic001-1, topic001-0] for group 0
2019-01-01 13:41:42.805  INFO 1422 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned:[topic001-1, topic001-0]
2019-01-01 13:48:00.938  INFO 1422 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : Revoking previously assigned partitions [topic001-1, topic001-0] for group 0
2019-01-01 13:48:00.939  INFO 1422 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked:[topic001-1, topic001-0]
```

- 上述內容顯示了當前應用消費了兩個partition；

- 再啟動一個同樣的應用，這樣每個應用負責一個parititon的消費，做法是在檔案kafka01103consumer-0.0.1-SNAPSHOT.jar所在目錄執行命令**java -jar kafka01103consumer-0.0.1-SNAPSHOT.jar --server.port=8081**，看看控制檯的輸出：

```shell
2019-01-01 13:47:58.068  INFO 1460 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version : 0.11.0.2
2019-01-01 13:47:58.069  INFO 1460 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId : 73be1e1168f91ee2
2019-01-01 13:47:58.103  INFO 1460 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 
2019-01-01 13:47:58.226  INFO 1460 --- [           main] c.b.k.Kafka01103consumerApplication      : Started Kafka01103consumerApplication in 11.513 seconds (JVM running for 14.442)
2019-01-01 13:47:59.007  INFO 1460 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : Discovered coordinator kafka1:9092 (id: 2147482646 rack: null) for group 0.
2019-01-01 13:47:59.030  INFO 1460 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : Revoking previously assigned partitions [] for group 0
2019-01-01 13:47:59.031  INFO 1460 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked:[]
2019-01-01 13:47:59.032  INFO 1460 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : (Re-)joining group 0
2019-01-01 13:48:00.967  INFO 1460 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : Successfully joined group 0 with generation 6
2019-01-01 13:48:00.985  INFO 1460 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : Setting newly assigned partitions [topic001-0] for group 0
2019-01-01 13:48:01.015  INFO 1460 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned:[topic001-0]
```

- 可見新的程式消費的是0號partition，此時再去看看先啟動的程式的控制檯，見到了新的日誌，顯示該程式只消費1號pairtition了：

```shell
2019-01-01 13:48:00.955  INFO 1422 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : Successfully joined group 0 with generation 6
2019-01-01 13:48:00.960  INFO 1422 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : Setting newly assigned partitions [topic001-1] for group 0
2019-01-01 13:48:00.967  INFO 1422 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned:[topic001-1]
```

### 驗證訊息的生產和消費
- 在瀏覽器輸入以下地址：192.168.1.102:8080/send/Tom/hello
- 瀏覽器顯示返回的結果是：send message to [Tom] success (2019-01-01 13:58:08)，表示操作成功；
- 去檢查兩個消費者程式的控制檯，發現其中一個成功的消費了訊息，如下：

```shell
----------------- record =ConsumerRecord(topic = topic001, partition = 0, offset = 0, CreateTime = 1546351226016, serialized key size = -1, serialized value size = 133, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = {"timeLong":1546351225804,"name":"Tom","bizID":"4f1b6cf6-78d4-455d-b530-3956723a074f","time":"2019-01-01 22:00:25","message":"hello"})
------------------ message ={"timeLong":1546351225804,"name":"Tom","bizID":"4f1b6cf6-78d4-455d-b530-3956723a074f","time":"2019-01-01 22:00:25","message":"hello"}

```

- 至此，外部應用使用基於Docker的kafa服務實戰就完成了，如果您也在用Docker部署kafka服務，給外部應用使用，希望本文能給您提供一些參考；

### 歡迎關注ITpub：程式設計師欣宸
> [學習路上，你不孤單，欣宸原創一路相伴...](http://blog.itpub.net/70017844/)

如何使用Docker內的kafka服務

相關文章