實時數倉-持續更新

M_Fight๑҉發表於2024-12-09

映象伺服器清理

對於整個機器

rm -rf /tmp/*
rm -rf /usr/tmp/*
rm -rf /var/log/*
rm -rf /var/run/log/*
rm -rf /root/*
rm -rf /paimon

對於Dinky

rm -rf /opt/service/dinky-release-1.17-1.0.3/logs/*
rm -rf /opt/service/dinky-release-1.17-1.0.3/tmp/*

對於Hadoop

rm -rf /opt/service/hadoop-3.2.4/data/*
rm -rf /opt/service/hadoop-3.2.4/logs/*
rm -rf /opt/service/hadoop-3.2.4/tmp/*

對於Kafka

rm -rf /opt/service/kafka_2.12-3.0.0/data/*
rm -rf /opt/service/kafka_2.12-3.0.0/logs/*

對於zookeeper

rm -rf /opt/service/zookeeper-3.5.10/data/zkdata/*
rm -rf /opt/service/zookeeper-3.5.10/data/zkdatalog/*
rm -rf /opt/service/zookeeper-3.5.10/logs/*

JDK(基礎元件)

https://www.oracle.com/java/technologies/downloads/archive/

解壓後配置`/etc/profile`

#JAVA
export JAVA_HOME=/opt/service/jdk1.8.0_401
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:$CLASSPATH
export JAVA_PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin
export PATH=$PATH:${JAVA_PATH}

使命令生效

source /etc/profile

檢驗Java版本

java -version

MYSQL

安裝指令碼

#!/bin/bash
set -x
function Install_Mysql(){
    while : 
        do
        read -p "Do you need to install MySQL(yes/no): " my_result
        if [ "$my_result" == "no" ];then
            which mysql >/dev/null 2>&1 
            if [ "$?" != "0" ];then
                echo "MySQL client is not installed on this machine. Start to install now"
                cd $dir
                wget -O "mysql5.7.tar.gz" https://s3-gzpu.didistatic.com/pub/mysql5.7.tar.gz
                mkdir -p $dir/mysql/ && cd $dir/mysql/
                tar -zxf $dir/mysql5.7.tar.gz -C $dir/mysql/
                rpm -ivh $dir/mysql/mysql-community-common-5.7.36-1.el7.x86_64.rpm
                rpm -ivh $dir/mysql/mysql-community-libs-5.7.36-1.el7.x86_64.rpm
                rpm -ivh $dir/mysql/mysql-community-client-5.7.36-1.el7.x86_64.rpm
            fi
            read -p "Please enter the MySQL service address: " mysql_ip
            read -p "Please enter MySQL service port(default is 3306): " mysql_port
            read -p "Please enter the root password of MySQL service: " mysql_pass
            if [ "$mysql_port" == "" ];then
                mysql_port=3306
            fi
            break
        elif [ "$my_result" == "yes" ];then
            read -p "Installing MySQL service will uninstall the installed(if any), Do you want to continue(yes/no): " option
            if [ "$option" == "yes" ];then  
                cd $dir
                wget -O "mysql5.7.tar.gz" https://s3-gzpu.didistatic.com/pub/mysql5.7.tar.gz
                rpm -qa | grep -w -E "mariadb|mysql" | xargs yum -y remove >/dev/null 2>&1 
                mv -f /var/lib/mysql/ /var/lib/mysqlbak$(date "+%s") >/dev/null 2>&1 
                mkdir -p $dir/mysql/ && cd $dir/mysql/
                tar -zxf $dir/mysql5.7.tar.gz -C $dir/mysql/
                yum -y localinstall mysql* libaio*
                systemctl start mysqld
                systemctl enable mysqld >/dev/null 2>&1 
                old_pass=`grep 'temporary password' /var/log/mysqld.log | awk '{print $NF}' | tail -n 1`
                mysql -NBe "alter user USER() identified by '$mysql_pass';" --connect-expired-password -uroot -p$old_pass
                if [ $? -eq 0 ];then
                    mysql_ip="127.0.0.1"
                    mysql_port="3306"
                    echo  "Mysql database installation completed"
                    echo  -------- Mysql Password $old_pass ---------
                else
                    echo -e "${RED} Mysql database configuration failed. The script exits ${RES}"
                    exit
                fi
                break
            else 
                exit 1
            fi
        else
            Printlog "Input error, please re-enter（yes/no）"
            continue
        fi
    done
}
#引數宣告
dir=`pwd`
RED='\E[1;31m'
RES='\E[0m'
#呼叫
Install_Mysql

SSH(叢集免密)

--參考地址

https://www.jianshu.com/p/b71c58a598b5

埠：22

修改配置檔案（三臺） `/etc/hosts`

echo "::1     localhost       localhost.localdomain   localhost6      localhost6.localdomain6
127.0.0.1       localhost       localhost.localdomain   localhost4      localhost4.localdomain4
192.168.99.215  bigdata01 iZ2ze3nalp8guto80cb08tZ
192.168.99.216  bigdata02 iZ2ze3nalp8guto80cb08sZ
192.168.99.214  bigdata03 iZ2ze3nalp8guto80cb08rZ" > /etc/hosts

互相免密設定（三臺）

ssh-keygen -t rsa
.ae[%oTuPP~G%}3,hy{UPB&&}8M}18
#連續三次回車
ssh-copy-id root@bigdata01
#輸入密碼
#Odds!@#123
ssh-copy-id root@bigdata02
#輸入密碼
ssh-copy-id root@bigdata03
#輸入密碼

Zookeeper(叢集)

分散式協調服務，可以幫助分散式應用程式****實現資料同步

-- 參考地址

https://www.cnblogs.com/maoxianwan/articles/17486380.html

埠：2181/2888/3888

解壓後配置`/etc/profile`

#ZOOKEEPER
export ZOOKEEPER_HOME=/opt/hadoop/apache-zookeeper-3.8.0-bin/
export PATH=$PATH:$ZOOKEEPER_HOME/bin

使命令生效

source /etc/profile

修改配置檔案` ```$ZOOKEEPER_HOME/conf/zoo.cfg``` `

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/opt/service/zookeeper-3.5.10/data/zkdata
dataLogDir=/opt/service/zookeeper-3.5.10/data/zkdatalog
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=bigdata01:2888:3888
server.2=bigdata02:2888:3888
server.3=bigdata03:2888:3888

在目錄dataDir建立服務節點編號檔案myid，與server.後的數字對應

自定義叢集啟停指令碼

#!/bin/bash
for host in bigdata01 bigdata02 bigdata03
do
        case $1 in
        "start"){
                echo "                                               "
                echo "--------------- 啟 動 zookeeper ---------------"
                echo "------------ $host zookeeper -----------"
                ssh $host "source /etc/profile; zkServer.sh start"
        };;
        "stop"){
                echo "                                               "
                echo "--------------- 關 閉 zookeeper ---------------"
                echo "------------ $host zookeeper -----------"
                ssh $host "source /etc/profile; zkServer.sh stop"
        };;
        "status"){
                echo "                                               "
                echo "-------------- 檢視zookeeper狀態 --------------"
                echo "------------ $host zookeeper -----------"
                ssh $host "source /etc/profile; zkServer.sh status"
        };;
        esac
done

`chmod +x /bash/zkCluster.sh`

`cp /bash/zkCluster.sh /bin/`

測試

`zkCluster.sh status`

Kafka(叢集)

--參考地址

https://blog.csdn.net/snipercai/article/details/131812772

埠：9092/9999

解壓後配置`/etc/profile`

#HADOOP
export HADOOP_HOME=/opt/service/hadoop-3.2.4
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Hadoop(叢集)

--參考地址

https://blog.csdn.net/snipercai/article/details/131812772

埠：9000/9870/8485/10020/19888/8088

解壓後配置`/etc/profile`

#HADOOP
export HADOOP_HOME=/opt/service/hadoop-3.2.4
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

各配置檔案

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <!-- 設定預設使用的檔案系統 Hadoop支援file、HDFS、GFS等檔案系統 -->
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://ccns</value>
  </property>
  
  <!-- 設定Hadoop臨時目錄路徑 -->
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/opt/service/hadoop-3.2.4/tmp</value>
  </property>
  
  <!-- 指定zookeeper地址 -->
  <property>
    <name>ha.zookeeper.quorum</name>
    <value>bigdata01:2181,bigdata02:2181,bigdata03:2181</value>
  </property>
  
  <!-- 設定Hadoop本地儲存資料路徑 -->
  <property>
    <name>hadoop.data.dir</name>
    <value>/opt/service/hadoop-3.2.4/data</value>
  </property>
  
  <!-- 設定HDFS web UI使用者身份 -->  
  <property>
    <name>hadoop.http.staticuser.user</name>
    <value>root</value>
  </property>
  

  <!-- 檔案系統垃圾桶儲存時間 -->
  <property>
    <name>fs.trash.interval</name>
    <value>1440</value>
  </property>

  <property>
    <name>fs.alluxio.impl</name>
    <value>alluxio.hadoop.FileSystem</value>
  </property>
  
</configuration>

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <!--執行hdfs的nameservice為ns，注意要和core-site.xml中的名稱保持一致 -->
  <property>
    <name>dfs.nameservices</name>
    <value>ccns</value>
  </property>
  
  <!-- nameservice包含的namenode，ns叢集下有兩個namenode，分別為nn1, nn2 -->
  <property>
    <name>dfs.ha.namenodes.ccns</name>
    <value>nn1,nn2</value>
  </property>
  
  <!-- nn1的rpc地址和埠號，rpc用來和datanode通訊，預設值：9000-->
  <property>
    <name>dfs.namenode.rpc-address.ccns.nn1</name>
    <value>bigdata01:9000</value>
  </property>

  <!-- nn2的rpc地址和埠號，rpc用來和datanode通訊，預設值：9000-->
  <property>
    <name>dfs.namenode.rpc-address.ccns.nn2</name>
    <value>bigdata02:9000</value>
  </property>

  <!-- nn1的http地址和埠號，web客戶端 -->
  <property>
    <name>dfs.namenode.http-address.ccns.nn1</name>
    <value>bigdata01:9870</value>
  </property>
  
  <!-- nn2的http地址和埠號，web客戶端 -->
  <property>
    <name>dfs.namenode.http-address.ccns.nn2</name>
    <value>bigdata02:9870</value>
  </property>
  
  <!-- 指定namenode的後設資料在JournalNode上存放的位置，namenode2可以從journalnode叢集裡的指定位置上獲取資訊，達到熱備效果 -->
  <property>
    <name>dfs.namenode.shared.edits.dir</name>
    <value>qjournal://bigdata01:8485;bigdata02:8485;bigdata03:8485/ccns</value>
  </property>

  <!-- 配置失敗自動切換實現方式,客戶端連線可用狀態的NameNode所用的代理類,預設值：org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider -->
  <property>
    <name>dfs.client.failover.proxy.provider.ccns</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>

  <!-- 配置隔離機制,HDFS的HA功能的防腦裂方法。建議使用sshfence(hadoop:9922)，括號內的是使用者名稱和埠，注意，2臺NN之間可免密碼登陸.sshfences是防止腦裂的方法，保證NN中僅一個是Active的，如果2者都是Active的，新的會把舊的強制Kill -->
  <property>
    <name>dfs.ha.fencing.methods</name>
    <value>sshfence</value>
  </property>
  
  <!-- 開啟NameNode失敗自動切換 -->
  <property>
    <name>dfs.ha.automatic-failover.enabled</name>
    <value>true</value>
  </property>
  
  <!-- 指定上述選項ssh通訊使用的金鑰檔案在系統中的位置 -->
  <property>
    <name>dfs.ha.fencing.ssh.private-key-files</name>
    <value>/root/.ssh/id_rsa</value>
  </property>

  <!-- 指定JournalNode在本地磁碟存放資料的位置。 -->
  <property>
    <name>dfs.journalnode.edits.dir</name>
    <value>/opt/service/hadoop-3.2.4/data/journalnode</value>
  </property>
  
  <!--配置namenode存放後設資料的目錄，預設放到hadoop.tmp.dir下-->
  <property>   
    <name>dfs.namenode.name.dir</name>   
    <value>file:///opt/service/hadoop-3.2.4/data/namenode</value>   
  </property>   

  <!--配置datanode存放後設資料的目錄，預設放到hadoop.tmp.dir下-->
  <property>   
    <name>dfs.datanode.data.dir</name>   
    <value>file:///opt/service/hadoop-3.2.4/data/datanode</value>    
  </property>

  <!-- 副本數量配置 -->
  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
  
  <!--設定使用者的操作許可權，false表示關閉許可權驗證，任何使用者都可以操作-->                         
  <property>   
    <name>dfs.permissions</name>   
    <value>false</value>   
  </property>
</configuration>

yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>
    <!--開啟ResourceManager
  HA功能-->
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>

    <!--標誌ResourceManager-->
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>ccyarn</value>
    </property>

    <!--叢集中ResourceManager的ID列表,後面的配置將引用該ID-->
    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
    </property>

    <!-- 設定YARN叢集主角色執行節點rm1-->
    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>bigdata01</value>
    </property>

    <!-- 設定YARN叢集主角色執行節點rm2-->
    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>bigdata02</value>
    </property>

    <!--ResourceManager1的Web頁面訪問地址-->
    <property>
        <name>yarn.resourcemanager.webapp.address.rm1</name>
        <value>bigdata01:8088</value>
    </property>

    <!--ResourceManager2的Web頁面訪問地址-->
    <property>
        <name>yarn.resourcemanager.webapp.address.rm2</name>
        <value>bigdata02:8088</value>
    </property>

    <!--ZooKeeper叢集列表-->
    <property>
        <name>hadoop.zk.address</name>
        <value>bigdata01:2181,bigdata02:2181,bigdata03:2181</value>
    </property>

    <!--啟用ResouerceManager重啟的功能,預設為false-->
    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>

    <!--用於ResouerceManager狀態儲存的類-->
    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>

    <!-- Nodemanager重啟恢復機制-->
    <!-- ShuffleHandler服務也已經支援在NM重啟後恢復以前的狀態-->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <!-- 是否將對容器實施實體記憶體限制 -->
    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>

    <!-- 是否將對容器實施虛擬記憶體限制 -->
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>

    <!-- 開啟日誌聚集功能 -->
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>

    <!-- 設定日誌聚集伺服器地址 -->
    <property>
        <name>yarn.log.server.url</name>
        <value>http://bigdata01:19888/jobhistory/logs</value>
    </property>

    <!-- 設定日誌保留時間 -->
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>172800</value>
    </property>
    <!--Flink相關-->
    <property>
        <name>yarn.resourcemanager.am.max-attempts</name>
        <value>4</value>
        <description>
            The maximum number of application master execution attempts.
        </description>
    </property>
    <property>
        <name>yarn.app.attempts</name>
        <value>3</value>
    </property>
    <property>
        <name>yarn.application.attempt.failures-validity-interval</name>
        <value>5 minute</value>
    </property>
    <!-- 選擇排程器。預設容量,此處用公平排程-->
    <property>
        <name>yarn.resourcemanager.scheduler.class</name>
        <!--
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value> -->
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    </property>
    <property>
        <name>yarn.scheduler.fair.allocation.file</name>
        <value>/opt/service/hadoop-3.2.4/etc/hadoop/fair-scheduler.xml</value>
        <description>指明公平排程器佇列分配配置檔案</description>
    </property>
    <!--
  在未指定佇列的情況下，是否使用【使用者名稱】作為佇列名
  當設定為true時，當`yellow`使用者提交作業時，會自動建立並使用`root.yellow`佇列
  當設定為false時，所有使用者預設使用`root.default`佇列
  當配置了`yarn.scheduler.fair.allocation.file`時，本配置將被忽略
  -->
    <property>
        <name>yarn.scheduler.fair.user-as-default-queue</name>
        <value>false</value>
    </property>
    <!-- 是否啟用佇列間搶佔-->
    <!-- <property>
        <name>yarn.scheduler.fair.preemption</name>
        <value>true</value>
    </property> -->
    <!-- 觸發搶佔的閾值：資源使用量與總容量的佔比 -->
    <!-- <property>
        <name>yarn.scheduler.fair.preemption.cluster-utilization-threshold</name>
        <value>0.7f</value>
    </property> -->
    <!-- 應用最大優先順序 -->
    <!-- <property>
        <name>yarn.cluster.max-application-priority</name>
        <value>100</value>
    </property> -->

    <!--ResourceManager處理排程器請求的現場數量，一共3*4=12，留出幾個供其他使用-->
    <property>
        <name>yarn.resourcemanager.scheduler.client.thread-count</name>
        <value>5</value>
    </property>
    <!--NodeManager相關-->
    <!--是否讓yarn自動檢測硬體進行配置-->
    <property>
        <name>yarn.nodemanager.resource.detect-hardware-capabilities</name>
        <value>false</value>
    </property>
    <!--是否將虛擬核數當作CPU核數
    預設false-->
    <property>
        <name>yarn.nodemanager.resource.count-logical-processors-as-cores</name>
        <value>true</value>
    </property>
    <!--虛擬核數和物理核數乘數-->
    <property>
        <name>yarn.nodemanager.resource.pcores-vcores-multiplier</name>
        <value>16</value>
    </property>
    <!--nodemanager使用記憶體數，預設8g，此處設定為8g*0.8-->
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>30720</value>
        <discription>每個節點可用記憶體,預設8192M(8G)</discription>
    </property>
    <!--nodemanager的CPU核數，預設設定為8個-->
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>96</value>
        <discription>預設為8。每個節點可分配多少虛擬核給YARN使用,通常設為該節點定義的總虛擬核數即可。</discription>
    </property>
    <!--容器最小記憶體，預設1g-->
    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>1024</value>
        <discription>單個任務可申請最少記憶體,預設1024MB</discription>
    </property>
    <!--容器最大記憶體，預設8g-->
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>4096</value>
        <discription>單個任務可申請最大記憶體,預設8192M(8G)</discription>
    </property>
    <!--容器最小CPU核數，預設2個-->
    <property>
        <name>yarn.scheduler.minimum-allocation-vcores</name>
        <value>1</value>
        <discription>指定RM可以為每個container分配的最小/最大虛擬核數，低 於或高於該限制的核申請，會按最小或最大核數來進行分配。預設值適合 一般叢集使用。</discription>
    </property>
    <!--容器最大CPU核數，預設4個-->
    <property>
        <name>yarn.schedluler.maximum-allocation-vcores</name>
        <value>6</value>
        <discription>指定RM可以為每個container分配的最小/最大虛擬核數，低 於或高於該限制的核申請，會按最小或最大核數來進行分配。預設值適合 一般叢集使用。</discription>
    </property>
    <!--虛擬記憶體和實體記憶體設定比例，預設2.1-->
    <property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>1</value>
        <discription>實體記憶體不足時,使用的虛擬記憶體,預設是2.1,表示每使用1MB的實體記憶體,最多可以使用2.1MB的虛擬記憶體總量。</discription>
    </property>
    <property>
        <name>yarn.nodemanager.vcores-pcores-ratio</name>
        <value>16</value>
        <discription>每使用一個物理cpu,可以使用的虛擬cpu的比例,預設為2</discription>
    </property>

    <property>
        <!-- Maximum resources to allocate to application masters
        If this is too high application masters can crowd out actual work -->
        <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
        <description>應用程式主控器（Application Master，簡稱 AM）可以使用的最大資源百分比</description>
        <value>0.01</value>
    </property>

</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!-- 設定MR程式預設執行模式，yarn叢集模式，local本地模式 -->
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>

  <property>
    <name>yarn.app.mapreduce.am.env</name>
    <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
  </property>

  <!-- 歷史伺服器端地址 -->
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>bigdata01:10020</value>
  </property>
  
  <!-- 歷史伺服器web端地址 -->
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>bigdata01:19888</value>
  </property>

  <!-- map環境變數 -->
  <property>
    <name>mapreduce.map.env</name>
    <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
  </property>

  <!-- reduce環境變數 -->
  <property>
    <name>mapreduce.reduce.env</name>
    <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
  </property>

  
</configuration>

fair-site.xml

此檔案根據yarn-site.xml進行配置

<?xml version="1.0"?>
<allocations>
  <!-- 預設每個使用者的最大應用程式數 -->
  <userMaxAppsDefault>100</userMaxAppsDefault>

  <!-- 根佇列配置 -->
  <queue name="root">
    <!-- 最小資源分配: 512MB 記憶體, 1 個 vcore -->
    <minResources>1024mb,1vcores</minResources>
    <!-- 最大資源分配: 1024MB 記憶體, 2 個 vcore -->
    <maxResources>153600mb,480vcores</maxResources>
    <!-- 最大執行中的應用程式數 -->
    <maxRunningApps>200</maxRunningApps>
    <!-- 權重 -->
    <weight>1.0</weight>
    <!-- 排程模式: fair 表示公平排程 -->
    <schedulingMode>fair</schedulingMode>
    <!-- 提交應用程式的 ACL (Access Control List) -->
    <aclSubmitApps>*</aclSubmitApps>
    <!-- 管理應用程式的 ACL -->
    <aclAdministerApps>*</aclAdministerApps>

    <!-- default 子佇列配置 -->
    <queue name="default">
      <!-- 最小資源分配: 256MB 記憶體, 1 個 vcore -->
      <minResources>1024mb,1vcores</minResources>
      <!-- 最大資源分配: 512MB 記憶體, 1 個 vcore -->
      <maxResources>107520mb,300vcores</maxResources>
      <!-- 最大執行中的應用程式數 -->
      <maxRunningApps>100</maxRunningApps>
      <!-- 排程模式: fair 表示公平排程 -->
      <schedulingMode>fair</schedulingMode>
      <!-- 權重 -->
      <weight>1.0</weight>
      <!-- 提交應用程式的 ACL: * 表示任何人都可以提交應用程式 -->
      <aclSubmitApps>*</aclSubmitApps>
      
      <!-- 新增 maxAMResourcePerApp 以限制 AM 的最大資源 -->
      <maxAMResourcePerApp>60000mb,150vcores</maxAMResourcePerApp>
    </queue>

    <!-- taobao 子佇列配置 -->
    <queue name="taobao">
      <!-- 最小資源分配: 1024MB 記憶體, 1 個 vcore -->
      <minResources>1024mb,1vcores</minResources>
      <!-- 最大資源分配: 46080MB 記憶體, 1 個 vcore -->
      <maxResources>46080mb,180vcores</maxResources>
      <!-- 最大執行中的應用程式數 -->
      <maxRunningApps>100</maxRunningApps>
      <!-- 排程模式: fair 表示公平排程 -->
      <schedulingMode>fair</schedulingMode>
      <!-- 權重 -->
      <weight>1.0</weight>
      <aclSubmitApps>*</aclSubmitApps>
      
      <!-- 新增 maxAMResourcePerApp 以限制 AM 的最大資源 -->
      <maxAMResourcePerApp>23040mb,180vcores</maxAMResourcePerApp>
    </queue>

  </queue>
</allocations>

使命令生效

source /etc/profile

修改配置檔案(詳見伺服器)

建立對應資料存放目錄

mkdir /opt/service/hadoop-3.2.4/tmp
mkdir /opt/service/hadoop-3.2.4/data
mkdir /opt/service/hadoop-3.2.4/data/journalnode
mkdir /opt/service/hadoop-3.2.4/data/namenode
mkdir /opt/service/hadoop-3.2.4/data/datanode

初始化叢集

#bigdata01執行
hdfs zkfc -formatZK

### 每個節點分別啟動 journalnode ###
hdfs --daemon start journalnode
### 初始化主namenode，此次選擇bigdata01為主namenode ###
hdfs namenode -format
#啟動主namenode
hdfs --daemon start namenode
#初始化從namenode，此次bigdata02為從namenode
hdfs namenode -bootstrapStandby
#啟動所有程序
start-all.sh
  • https://www.cnblogs.com/lenmom/p/11285273.html Yarn****排程器有關，後續視情況選擇具體排程器

叢集擴縮容（無需停機）

https://blog.csdn.net/andy_wcl/article/details/104558092

預設各個節點與叢集總的儲存使用率相差不超過10%，此叢集磁碟空間較小，建議設定為5%，避免資料塊傾斜

Yarn調優

如果一個伺服器是32核，虛擬後為64核，128G記憶體，我們該如何設定上面的6個引數呢？即如何做到資源最大化利用，生產上我們一般要預留15-20%的記憶體，那麼可用記憶體就是128*0.8=102.4G，去除其他元件的使用，我們設定成90G就可以了。就是說,yarn總可用資源是64核,90G；

設定單個任務可用的最大及最小核數

yarn.sheduler.maximum-allocation-vcores

一般就設定成4個，cloudera公司做過效能測試，如果CPU大於等於5之後，CPU的利用率反而不是很好。這個引數可以根據生成伺服器決定，比如公司伺服器很富裕，那就直接設定成1:1；設定成32，如果不是很富裕，可以直接設定成1:2。我們以1:2來計算。

yarn.scheduler.minimum-allocation-vcores

如果設定vcoure = 1，那麼最大可以跑64/1=64個container，如果設定成這樣，最小container是64/4=16個。

設定單個任務可用的最大及最小記憶體

yarn.scheduler.minmum-allocation-mb

預設是1G,如果設定成2G，那麼90/2=45最多可以跑45個container，如果設定成4G，那麼最多可以跑24個；vcore有些浪費。

yarn.scheduler.maximum-allocation-mb

Hadoop日誌清理

已配置自動清理，如需手動清理，三臺機器執行以下命令

rm -rf /opt/service/hadoop-3.2.4/logs/*.log.*
rm -rf /opt/service/hadoop-3.2.4/logs/*.out.*

Flink(On Yarn)

--參考地址

https://blog.csdn.net/ASN_forever/article/details/106234893

https://blog.csdn.net/weixin_52134189/article/details/139332965

埠：6123/8081

Flink On Yarn 情況下，單節點放flink即可

修改配置檔案·yarn-site.xml·

<property>
  <name>yarn.resourcemanager.am.max-attempts</name>
  <value>4</value>
  <description>
    The maximum number of application master execution attempts.
  </description>
</property>

修改配置檔案·flink-conf.xml·

# jobmanager和taskmanager、其他client的RPC通訊IP地址，TaskManager用於連線到JobManager/ResourceManager 。HA模式不用配置此項，在master檔案中配置，由zookeeper選出leader與standby
jobmanager.rpc.address: 0.0.0.0
 
# jobmanager和taskmanager、其他client的RPC通訊埠，TaskManager用於連線到JobManager/ResourceManager 。HA模式不用配置此項，在master檔案中配置，由zookeeper選出leader與standby
jobmanager.rpc.port: 6123
 
# jobmanager JVM heap 記憶體大小
jobmanager.memory.process.size: 1024m
 
# taskmanager JVM heap 記憶體大小
taskmanager.memory.process.size: 2048m
 
# 每個taskmanager提供的任務slots數量
taskmanager.numberOfTaskSlots: 4
 
# 平行計算個數
parallelism.default: 4

 
# 高可用模式
high-availability.type: zookeeper
 
# # JobManager後設資料保留在檔案系統storageDir中 指向此狀態的指標儲存在ZooKeeper中
high-availability.storageDir: hdfs:///dinky/flink/recovery_demo
 
# # Zookeeper叢集
high-availability.zookeeper.quorum: bigdata01:2181,bigdata02:2181,bigdata03:2181
 
# # 在zookeeper下的根目錄
high-availability.zookeeper.path.root: /flink
 
# 單個flink job重啟次數 必須小於等於yarn-site.xml中Application Master配置的嘗試次數
yarn.application-attempts: 4
 
#==============================================================================
# Fault tolerance and checkpointing
#==============================================================================
# jobmanager (MemoryStateBackend), filesystem (FsStateBackend), rocksdb (RocksDBStateBackend)

# 檢查點之間的時間間隔為180秒
execution.checkpointing.interval: 180000
# 兩次系統檢查點間最短間隔180秒
execution.checkpointing.min-pause: 180000
# 使用精確一次（EXACTLY_ONCE）模式進行檢查點
execution.checkpointing.mode: EXACTLY_ONCE
# 狀態後端使用檔案系統（filesystem）
state.backend: filesystem
# 檢查點儲存目錄
state.checkpoints.dir: hdfs:///dinky/flink/checkpoint
# Savepoint（儲存點）
state.savepoints.dir: hdfs:///dinky/flink/savepoint
# 檢查點超時時間為600000毫秒（10分鐘）
execution.checkpointing.timeout: 600000
# 允許的最大併發檢查點數為1個
execution.checkpointing.max-concurrent-checkpoints: 1
# 保留的最大檢查點數為2個
state.checkpoints.num-retained: 2
# 在作業取消時刪除外部化的檢查點
execution.checkpointing.externalized-checkpoint-retention: DELETE_ON_CANCELLATION

# 是否應該建立增量檢查點。對於增量檢查點，只儲存前一個檢查點的差異，而不儲存完整的檢查點狀態。一些狀態後端可能不支援增量檢查點並忽略此選項。
state.backend.incremental: false
# jobmanager故障恢復策略，指定作業計算如何從任務失敗中恢復 full重新啟動所有任務以恢復作業 region重新啟動可能受任務失敗影響的所有任務，region在目前（flink1.11）只對批處理有效，實時計算任然時full
jobmanager.execution.failover-strategy: region

#OSS相關
# fs.oss.endpoint: https://oss-cn-beijing-internal.aliyuncs.com
# fs.oss.accessKeyId: LTAI5tN3omJRJwcNyzUEo34v
# fs.oss.accessKeySecret: QgRfeoQ8TprkQPA0yknFvGcStTgQ4D

Dinky(編寫入口)

--參考地址

https://dinky.org.cn/docs/1.0/deploy_guide/normal_deploy

埠：8888

預設賬密：admin/dinky123!@#

叢集執行時候與本地jar包無關，只需有config檔案及hdfs中lib檔案即可

Dinky使用教程

補充

單個taskmanager/****jobmanager 並行度****，slot 上限為4，建議低於4或者配置4的倍數； taskmanager/jobmanager記憶體，上限為4096m，預設為1024m，建議1024m倍數修改或者預設； 並行度配置為slot的N倍，即可均勻增加taskmanager個數；

DolphinScheduler(待定-排程&監控叢集)

--參考地址

https://github.com/apache/dolphinscheduler/blob/3.1.5-release/docs/docs/zh/guide/howto/datasource-setting.md

埠：12345

預設賬密： admin/dolphinscheduler123

問題記錄

  1. 存在頻繁報錯任務，在yarn中不斷列印日誌，導致執行節點當機，擬定關閉某些日誌的列印；

  2. 存在資源濫用情況，即slot與並行度的配置，任務提交上去後及時觀察並設定合理資源，然後釋出並配置告警；

V_2.0

  1. 建議資源隔離，任務隔離，用容器進行部署

相關文章