第二週進度總結

（1）本週做了什麼，花在學習上多長時間，花在程式碼時間上多長時間，花在解決問題用了多長時間。

本週完成了yarn叢集和hive資料庫的部署

使用的命令如下

su - aaa
cd /export/server/hadoop/etc/hadoop/
ll
vim mapred-env.sh
export JAVA_HOME=/export/server/jdk
export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=1000
export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA

vim mapred-site.xml

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description></description>
</property>

<property>
<name>mapreduce.jobhistory.address</name>
<value>node1:10020</value>
<description></description>
</property>

<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>node1:19888</value>
<description></description>
</property>

<property>
<name>mapreduce.jobhistory.intermediate-done-dir</name>
<value>/data/mr-history/tmp</value>
<description></description>
</property>

<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/data/mr-history/done</value>
<description></description>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>

vim yarn-env.sh

export JAVA_HOME=/export/server/jdk
export HADOOP_HOME=/export/server/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_LOG_DIR=$HADOOP_HOME/logs

vim yarn-site.xml

<property>
<name>yarn.log.server.url</name>
<value>http://node1:19888/jobhistory/logs</value>
<description></description>
</property>

<property>
<name>yarn.web-proxy.address</name>
<value>node1:8089</value>
<description>proxy server hostname and port</description>
</property>

<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
<description>Configuration to enable or disable log aggregation</description>
</property>

<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/tmp/logs</value>
<description>Configuration to enable or disable log aggregation</description>
</property>

<property>
<name>yarn.resourcemanager.hostname</name>
<value>node1</value>
<description></description>
</property>

<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
<description></description>
</property>

<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/data/nm-local</value>
<description>Comma-separated list of paths on the local filesystem where intermediate data is written.</description>
</property>

<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/data/nm-log</value>
<description>Comma-separated list of paths on the local filesystem where logs are written.</description>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>10800</value>
<description>Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled.</description>
</property>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>Shuffle service that needs to be set for Map Reduce applications.</description>
</property>

scp mapred-env.sh mapred-site.xml yarn-env.sh yarn-site.xml node2:`pwd`/
scp mapred-env.sh mapred-site.xml yarn-env.sh yarn-site.xml node3:`pwd`/

start-yarn.sh
mapred --daemon start historyserver

#更新金鑰

rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022

#安裝MysqL yum庫

rpm -Uvh http://repo.mysql.com//mysql57-community-release-el7-7.noarch.rpm

#yum安裝Mysql

yum -y install mysql-community-server

#啟動MySqL 設定開機啟動

systemctl start mysqld

systemctl enable mysqld

#檢查MysqL服務狀態

systemctl status mysqld

#第一次啟動mysql，會在日誌檔案中生成root使用者的一個隨機密碼，使用下面命令檢視該密碼

grep 'temporary password' /var/log/mysqld.log

#如果你想設定簡單密碼，需要降低MySqL的密碼安全級別

set global validate_password_policy=LOW；#密碼安全級別低

set global validate_password_length=4;#密碼長度最低4位即可

#然後就可以用簡單密碼了（課程中使用簡單密碼，為了方便，生產中不要這樣）

ALTER USER 'root'@'LocaLhost'IDENTIFIED BY '123456';

grant all privileges on *.* to root@"%" identified by '123456' with grant option;

flush privileges;

cd /export/server/hadoop

cd etx/hadoop/

vim core-site.xml

<name>hadoop.proxyuser.hadoop.hosts</name>

</property>

<name>hadoop.proxyuser.hadoop.groups</name>

</property>

tar -zxvf apache-hive-3.1.3-bin.tar.gz -C /export/server/

mv mysql-connector-java-5.1.34.jar /export/server/apache-hive-3.1.3-bin/lib/

cd /export/server/apache-hive-3.1.3-bin/lib

cd ..

ln -s /export/server/apache-hive-3.1.3-bin hive

cd hive/

cd conf

ll

mv hive-env.sh.template hive-env.sh

ll

vim hive-env.sh

export HADOOP_HOME=/export/server/hadoop

export HIVE_CONF_DIR=/export/server/hive/conf

export HIVE_AUX_JARS_PATH=/export/server/hive/lib

vim hive-site.xml

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://node1:3306/hive?createDatabaseIfNotExist=true&useSSL=false&useUnicode=true&characterEncoding=UTF-8</value></property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

</property>

<name>javax.jdo.option.ConnectionUserName</name>

</property>

<name>javax.jdo.option.ConnectionPassword</name>

</property>

<name>hive.server2.thrift.bind.host</name>

</property>

<name>hive.metastore.uris</name>

<value>thrift://node1:9083</value>

</property>

<name>hive.metastore.event.db.notification.api.auth</name>

<value>false</value>

</property>

</configuration>

CREATE DATABASE hive CHARSET UTF8;

cd /export/server/hive

./schematool -initSchema -dbType mysql -verbos

chown -R leap:leap apache-hive-3.1.3-bin hive

cd /export/server/hive

mkdir logs

bin/hive --service metastore

nohup bin/hive --service metastore >>logs/metastore.log 2>&1 &

bin/hive

nohup bin/hive --service hiveserver2 >>logs/hiveserver2.log 2>&1 &

（2）下週準備做什麼？

下週準備使用jdbc連線hive資料庫

（3）本週遇到的問題？

本週遇到了金鑰無法更新的問題，透過更換源映象解決了

第二週進度總結

相關文章