【記錄】CDH5.12.1整合Flink1.9.0，並實現cm5.12.1監控flink

spark9527發表於2020-09-30

寫在前頭：最近在CDH上整合flink，現在新版本的CDH6.3以上的，官網有提供parcel跟csd檔案，但是我們使用的是CDH5.12.1，還是比較老的版本，所以要自己編譯。在這期間蒐集了不少資料，但是沒有一篇詳細的文章是從flink編譯=>生成csd、parcel檔案=>整合cm這一整套流程完整走下來的。因此我整理了之前收集的資料，並把自己在安裝過程出現的問題也整理了出來。希望對大家有所幫助

一、安裝準備

我使用的軟體版本：

Hadoop 2.6.0-cdh5.12.1
Apache Maven 3.6.3 
Java version: 1.8.0_161
Cloudera Manager 5.12.1

我們需要做的事情：

1、從官網上下載flink的原始碼及依賴包
2、對原始碼及依賴包進行相應的編譯
3、將編譯完成的flink包製作成cloudera manager需要的csd檔案、parcel檔案
4、cm一鍵安裝部署flink，並將flink服務新增到管理列表中

明確了我們需要做的事情之後，一步步操作就好了，這個過程我會盡可能詳細的描述操作過程。

二、軟體包下載

這裡可以根據自己需要的版本下載自己需要包即可。這裡我選擇的1.9.0，對應的依賴包是7.0

flink的安裝包下載

wget https://github.com/apache/flink/archive/release-1.9.0.zip

flink依賴包

wget https://archive.apache.org/dist/flink/flink-shaded-7.0/flink-shaded-7.0-src.tgz

三、編譯

flink-1.9.0編譯

1.解壓

unzip release-1.9.0.zip -d ./

2.修改根目錄下的pom.xml

vim flink-release-1.9.0/pom.xml

刪除test相關modules，減少編譯時間，大約在80-90行
在這裡插入圖片描述
增加下載源

<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
<repository>
<id>mvnrepository</id>
<url>https://mvnrepository.com</url>
</repository>
</repositories>

3.開始編譯

mvn clean install -DskipTests  -Pvendor-repos -Dhadoop.version=2.6.0-cdh5.12.1 -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Drat.skip=true

編譯成功

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for flink 1.9.0:
[INFO] 
[INFO] force-shading ...................................... SUCCESS [  1.579 s]
[INFO] flink .............................................. SUCCESS [  1.484 s]
[INFO] flink-annotations .................................. SUCCESS [ 40.863 s]
[INFO] flink-shaded-curator ............................... SUCCESS [01:34 min]
[INFO] flink-metrics ...................................... SUCCESS [  0.059 s]
[INFO] flink-metrics-core ................................. SUCCESS [01:58 min]
[INFO] flink-test-utils-parent ............................ SUCCESS [  0.066 s]
[INFO] flink-test-utils-junit ............................. SUCCESS [  0.485 s]
[INFO] flink-core ......................................... SUCCESS [02:14 min]
[INFO] flink-java ......................................... SUCCESS [ 19.734 s]
[INFO] flink-queryable-state .............................. SUCCESS [  0.047 s]
[INFO] flink-queryable-state-client-java .................. SUCCESS [  0.469 s]
[INFO] flink-filesystems .................................. SUCCESS [  0.050 s]
[INFO] flink-hadoop-fs .................................... SUCCESS [04:23 min]
[INFO] flink-runtime ...................................... SUCCESS [32:45 min]
[INFO] flink-scala ........................................ SUCCESS [01:01 min]
[INFO] flink-mapr-fs ...................................... SUCCESS [  0.397 s]
[INFO] flink-filesystems :: flink-fs-hadoop-shaded ........ SUCCESS [02:05 min]
[INFO] flink-s3-fs-base ................................... SUCCESS [01:35 min]
[INFO] flink-s3-fs-hadoop ................................. SUCCESS [ 18.102 s]
[INFO] flink-s3-fs-presto ................................. SUCCESS [08:16 min]
[INFO] flink-swift-fs-hadoop .............................. SUCCESS [03:06 min]
[INFO] flink-oss-fs-hadoop ................................ SUCCESS [02:38 min]
[INFO] flink-azure-fs-hadoop .............................. SUCCESS [12:05 min]
[INFO] flink-optimizer .................................... SUCCESS [  1.839 s]
[INFO] flink-clients ...................................... SUCCESS [  9.584 s]
[INFO] flink-streaming-java ............................... SUCCESS [ 12.332 s]
[INFO] flink-test-utils ................................... SUCCESS [03:42 min]
[INFO] flink-runtime-web .................................. SUCCESS [10:55 min]
[INFO] flink-examples ..................................... SUCCESS [  0.084 s]
[INFO] flink-examples-batch ............................... SUCCESS [ 14.505 s]
[INFO] flink-connectors ................................... SUCCESS [  0.039 s]
[INFO] flink-hadoop-compatibility ......................... SUCCESS [  7.456 s]
[INFO] flink-state-backends ............................... SUCCESS [  0.038 s]
[INFO] flink-statebackend-rocksdb ......................... SUCCESS [08:05 min]
[INFO] flink-tests ........................................ SUCCESS [ 41.102 s]
[INFO] flink-streaming-scala .............................. SUCCESS [ 51.065 s]
[INFO] flink-table ........................................ SUCCESS [  0.039 s]
[INFO] flink-table-common ................................. SUCCESS [  1.371 s]
[INFO] flink-table-api-java ............................... SUCCESS [  0.940 s]
[INFO] flink-table-api-java-bridge ........................ SUCCESS [  0.460 s]
[INFO] flink-table-api-scala .............................. SUCCESS [  7.612 s]
[INFO] flink-table-api-scala-bridge ....................... SUCCESS [ 11.582 s]
[INFO] flink-sql-parser ................................... SUCCESS [07:27 min]
[INFO] flink-libraries .................................... SUCCESS [  0.035 s]
[INFO] flink-cep .......................................... SUCCESS [  2.308 s]
[INFO] flink-table-planner ................................ SUCCESS [04:18 min]
[INFO] flink-orc .......................................... SUCCESS [ 37.190 s]
[INFO] flink-jdbc ......................................... SUCCESS [02:48 min]
[INFO] flink-table-runtime-blink .......................... SUCCESS [ 34.623 s]
[INFO] flink-table-planner-blink .......................... SUCCESS [03:13 min]
[INFO] flink-hbase ........................................ SUCCESS [21:52 min]
[INFO] flink-hcatalog ..................................... SUCCESS [13:21 min]
[INFO] flink-metrics-jmx .................................. SUCCESS [  0.321 s]
[INFO] flink-connector-kafka-base ......................... SUCCESS [05:50 min]
[INFO] flink-connector-kafka-0.9 .......................... SUCCESS [05:34 min]
[INFO] flink-connector-kafka-0.10 ......................... SUCCESS [ 21.502 s]
[INFO] flink-connector-kafka-0.11 ......................... SUCCESS [08:17 min]
[INFO] flink-formats ...................................... SUCCESS [  0.036 s]
[INFO] flink-json ......................................... SUCCESS [  0.361 s]
[INFO] flink-connector-elasticsearch-base ................. SUCCESS [15:19 min]
[INFO] flink-connector-elasticsearch2 ..................... SUCCESS [ 17.888 s]
[INFO] flink-connector-elasticsearch5 ..................... SUCCESS [17:39 min]
[INFO] flink-connector-elasticsearch6 ..................... SUCCESS [22:05 min]
[INFO] flink-csv .......................................... SUCCESS [  0.250 s]
[INFO] flink-connector-hive ............................... SUCCESS [  01:38 h]
[INFO] flink-connector-rabbitmq ........................... SUCCESS [01:41 min]
[INFO] flink-connector-twitter ............................ SUCCESS [06:29 min]
[INFO] flink-connector-nifi ............................... SUCCESS [14:28 min]
[INFO] flink-connector-cassandra .......................... SUCCESS [46:47 min]
[INFO] flink-avro ......................................... SUCCESS [05:38 min]
[INFO] flink-connector-filesystem ......................... SUCCESS [  0.709 s]
[INFO] flink-connector-kafka .............................. SUCCESS [26:12 min]
[INFO] flink-connector-gcp-pubsub ......................... SUCCESS [21:06 min]
[INFO] flink-sql-connector-elasticsearch6 ................. SUCCESS [  9.831 s]
[INFO] flink-sql-connector-kafka-0.9 ...................... SUCCESS [  0.403 s]
[INFO] flink-sql-connector-kafka-0.10 ..................... SUCCESS [  0.545 s]
[INFO] flink-sql-connector-kafka-0.11 ..................... SUCCESS [  0.770 s]
[INFO] flink-sql-connector-kafka .......................... SUCCESS [  1.288 s]
[INFO] flink-connector-kafka-0.8 .......................... SUCCESS [07:11 min]
[INFO] flink-avro-confluent-registry ...................... SUCCESS [02:30 min]
[INFO] flink-parquet ...................................... SUCCESS [28:53 min]
[INFO] flink-sequence-file ................................ SUCCESS [  0.283 s]
[INFO] flink-examples-streaming ........................... SUCCESS [01:48 min]
[INFO] flink-examples-table ............................... SUCCESS [  8.582 s]
[INFO] flink-examples-build-helper ........................ SUCCESS [  0.078 s]
[INFO] flink-examples-streaming-twitter ................... SUCCESS [  0.875 s]
[INFO] flink-examples-streaming-state-machine ............. SUCCESS [  0.448 s]
[INFO] flink-examples-streaming-gcp-pubsub ................ SUCCESS [ 19.498 s]
[INFO] flink-container .................................... SUCCESS [  0.341 s]
[INFO] flink-queryable-state-runtime ...................... SUCCESS [  0.544 s]
[INFO] flink-end-to-end-tests ............................. SUCCESS [  0.034 s]
[INFO] flink-cli-test ..................................... SUCCESS [  0.158 s]
[INFO] flink-parent-child-classloading-test-program ....... SUCCESS [  0.165 s]
[INFO] flink-parent-child-classloading-test-lib-package ... SUCCESS [  0.098 s]
[INFO] flink-dataset-allround-test ........................ SUCCESS [  0.223 s]
[INFO] flink-dataset-fine-grained-recovery-test ........... SUCCESS [  0.157 s]
[INFO] flink-datastream-allround-test ..................... SUCCESS [  1.400 s]
[INFO] flink-batch-sql-test ............................... SUCCESS [  0.162 s]
[INFO] flink-stream-sql-test .............................. SUCCESS [  0.193 s]
[INFO] flink-bucketing-sink-test .......................... SUCCESS [  0.327 s]
[INFO] flink-distributed-cache-via-blob ................... SUCCESS [  0.167 s]
[INFO] flink-high-parallelism-iterations-test ............. SUCCESS [ 11.756 s]
[INFO] flink-stream-stateful-job-upgrade-test ............. SUCCESS [  1.165 s]
[INFO] flink-queryable-state-test ......................... SUCCESS [  2.381 s]
[INFO] flink-local-recovery-and-allocation-test ........... SUCCESS [  0.211 s]
[INFO] flink-elasticsearch2-test .......................... SUCCESS [  6.770 s]
[INFO] flink-elasticsearch5-test .......................... SUCCESS [  8.624 s]
[INFO] flink-elasticsearch6-test .......................... SUCCESS [  4.310 s]
[INFO] flink-quickstart ................................... SUCCESS [  0.927 s]
[INFO] flink-quickstart-java .............................. SUCCESS [06:36 min]
[INFO] flink-quickstart-scala ............................. SUCCESS [  0.107 s]
[INFO] flink-quickstart-test .............................. SUCCESS [  0.516 s]
[INFO] flink-confluent-schema-registry .................... SUCCESS [  2.396 s]
[INFO] flink-stream-state-ttl-test ........................ SUCCESS [  5.924 s]
[INFO] flink-sql-client-test .............................. SUCCESS [  0.722 s]
[INFO] flink-streaming-file-sink-test ..................... SUCCESS [  0.168 s]
[INFO] flink-state-evolution-test ......................... SUCCESS [  1.181 s]
[INFO] flink-e2e-test-utils ............................... SUCCESS [01:34 min]
[INFO] flink-mesos ........................................ SUCCESS [03:29 min]
[INFO] flink-yarn ......................................... SUCCESS [  0.966 s]
[INFO] flink-gelly ........................................ SUCCESS [  2.158 s]
[INFO] flink-gelly-scala .................................. SUCCESS [ 19.871 s]
[INFO] flink-gelly-examples ............................... SUCCESS [ 19.212 s]
[INFO] flink-metrics-dropwizard ........................... SUCCESS [ 13.693 s]
[INFO] flink-metrics-graphite ............................. SUCCESS [  5.615 s]
[INFO] flink-metrics-influxdb ............................. SUCCESS [03:16 min]
[INFO] flink-metrics-prometheus ........................... SUCCESS [01:00 min]
[INFO] flink-metrics-statsd ............................... SUCCESS [  0.190 s]
[INFO] flink-metrics-datadog .............................. SUCCESS [  0.327 s]
[INFO] flink-metrics-slf4j ................................ SUCCESS [  0.184 s]
[INFO] flink-cep-scala .................................... SUCCESS [ 13.547 s]
[INFO] flink-table-uber ................................... SUCCESS [  3.382 s]
[INFO] flink-table-uber-blink ............................. SUCCESS [  3.937 s]
[INFO] flink-sql-client ................................... SUCCESS [03:47 min]
[INFO] flink-state-processor-api .......................... SUCCESS [  0.659 s]
[INFO] flink-python ....................................... SUCCESS [02:14 min]
[INFO] flink-scala-shell .................................. SUCCESS [ 13.666 s]
[INFO] flink-dist ......................................... SUCCESS [01:21 min]
[INFO] flink-end-to-end-tests-common ...................... SUCCESS [  0.334 s]
[INFO] flink-metrics-availability-test .................... SUCCESS [  0.167 s]
[INFO] flink-metrics-reporter-prometheus-test ............. SUCCESS [  0.140 s]
[INFO] flink-heavy-deployment-stress-test ................. SUCCESS [ 11.352 s]
[INFO] flink-connector-gcp-pubsub-emulator-tests .......... SUCCESS [04:15 min]
[INFO] flink-streaming-kafka-test-base .................... SUCCESS [  0.285 s]
[INFO] flink-streaming-kafka-test ......................... SUCCESS [ 11.284 s]
[INFO] flink-streaming-kafka011-test ...................... SUCCESS [ 10.451 s]
[INFO] flink-streaming-kafka010-test ...................... SUCCESS [  9.843 s]
[INFO] flink-plugins-test ................................. SUCCESS [  0.079 s]
[INFO] flink-tpch-test .................................... SUCCESS [ 23.368 s]
[INFO] flink-contrib ...................................... SUCCESS [  0.034 s]
[INFO] flink-connector-wikiedits .......................... SUCCESS [  9.878 s]
[INFO] flink-yarn-tests ................................... SUCCESS [ 13.034 s]
[INFO] flink-fs-tests ..................................... SUCCESS [  0.403 s]
[INFO] flink-docs ......................................... SUCCESS [ 30.525 s]
[INFO] flink-ml-parent .................................... SUCCESS [  0.033 s]
[INFO] flink-ml-api ....................................... SUCCESS [  0.243 s]
[INFO] flink-ml-lib ....................................... SUCCESS [  0.176 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

PS：編譯時間有點長，在這個過程中可能會報錯，我將自己遇到的錯誤及整理資料發現的錯誤進行一下彙總
問題一：

[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project flink-parent: Too many files with unapproved license: 1 See RAT report in: /home/dwhuangzhou/flink-release-1.9.0/target/rat.txt -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :flink-parent

解決方案：編譯命令增加 -Drat.skip=true ，跳過檢查

flink-shaded-7.0編譯

1.解壓

tar -zxvf  flink-shaded-7.0-src.tgz -C ./

2.修改根目錄下的pom.xml

vim flink-shaded-7.0/pom.xml

增加下載源

<repositories>
<repository>
<id>vdc</id>
<url>http://nexus.saas.hand-china.com/content/repositories</url>
</repository>
<repository>
<id>horton-works-releases</id>
<url>http://repo.hortonworks.com/content/groups/public/</url>
</repository>
<repository>
<id>mvn repository</id>
<url>https://mvnrepository.com/artifact/</url>
</repository>
<repository>
<id>CDH</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>

3.修改根目錄下flink-shaded-hadoop-2-uber的pom.xml
這步很重要，不做在執行FLINK的時候會報錯的！

vim flink-shaded-7.0/flink-shaded-hadoop-2-uber/pom.xml

增加如下依賴：

<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
<version>1.3.1</version>
</dependency>

4.回到根目錄下開始編譯

mvn clean install -DskipTests  -Pvendor-repos -Dhadoop.version=2.6.0-cdh5.12.1 -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Drat.skip=true

四、製作parcel

1.建立一個資料夾用來製作parcel，並將編譯好的flink包拷貝出來

mkdir /home/makeflink
cp flink-release-1.9.0/flink-dist/target/flink-1.9.0-bin/flink-1.9.0/flink-1.9.0.tgz  /home/makeflink/
cp flink-shaded-7.0/flink-shaded-hadoop-2-uber/target/flink-shaded-hadoop-2-uber-2.6.0-cdh5.12.1-7.0.jar   /home/makeflink/

2.下載製作parcel的模板

git clone https://github.com/gaozhangmin/flink-parcel

3.將拷貝過來的flink包放到flink-parcel裡邊，並改名

mv flink-1.9.0.tgz flink-parcel/flink-1.9.0-bin-scala_2.11.tgz

flink-parcel資料夾
4.修改parcel的製作規則

vim flink-parcel.properties

#FLINk 下載地址
FLINK_URL=./flink-1.9.0-bin-scala_2.11.tgz

#flink版本號
FLINK_VERSION=1.9.0

#擴充套件版本號
EXTENS_VERSION=BIN-SCALA_2.11

#作業系統版本，以centos為例
OS_VERSION=6

#CDH 小版本
CDH_MIN_FULL=5.2
CDH_MAX_FULL=6.2

#CDH大版本
CDH_MIN=5
CDH_MAX=6

5.開始製作

bash build.sh parcel

6.parcel製作完成
parcel完成
7.製作flink的csd（描述檔案）
因為我只需要standalone版本所以執行

bash build.sh csd_standalone

執行完畢
在這裡插入圖片描述

ps:如果你是需要的flink on yarn 的話則執行

bash build.sh csd_on_yarn

8.將生成好的檔案放在cm能識別到的地方

把生成的csd檔案放在cloudera的csd目錄下

cp FLINK-1.9.0.jar /opt/cloudera/csd/

需要注意的是，csd目錄的許可權應為cloudera-scm
在這裡插入圖片描述

把生成好的parcel放在cloudera的pracel-repo中

cp FLINK-1.9.0-BIN-SCALA_2.11_build/*   /opt/cloudera/parcel-repo/

步驟進行到這裡，flink的parcel檔案就製作好了

三、CM監控及一鍵安裝部署

1、開啟cm管理介面

http://namenode01:7180/cmf/home

2、找到parcel

在這裡插入圖片描述
3、安裝部署flink

此時，在/opt/cloudera/parcels目錄下應該已經有了FLINK

然後把我們製作的flink依賴包拷貝到FLINK/lib/flink/lib/下
注意：這兩步一定要做，不然flink起不起來的！

cp  /home/makeflink/flink-shaded-hadoop-2-uber-2.6.0-cdh5.12.1-7.0.jar  /opt/cloudera/parcels/FLINK/lib/flink/lib/

然後給其他機器也發一下

cd /opt/cloudera/parcels/FLINK/lib/flink/lib/
scp flink-shaded-hadoop-2-uber-2.6.0-cdh5.12.1-7.0.jar   node01:$PWD
scp flink-shaded-hadoop-2-uber-2.6.0-cdh5.12.1-7.0.jar   node02:$PWD

4、新增服務到cm頁面上

新增服務
在這裡插入圖片描述
根據自己需求新增依賴關係

角色分配