Hadoop2.7實戰v1.0之Hive-2.0.0+MySQL遠端模式安裝
環境:Apache Hadoop2.7分散式叢集環境(HDFS HA,Yarn HA,HBase HA)
後設資料庫mysql部署在hadoop-01機器上
user:hive
password:hive
database:hive_remote_meta
hive服務端部署在hadoop-01機器上
hive客戶端部署在hadoop-02機器上
1.Install MySQL5.6.23 on hadoop-01
2.Create db and user
hadoop-01:mysqladmin:/usr/local/mysql:>mysql -uroot -p
mysql> create database hive_remote_meta;
Query OK, 1 row affected (0.04 sec)
mysql> create user 'hive' identified by 'hive';
Query OK, 0 rows affected (0.05 sec)
mysql> grant all privileges on hive_remote_meta.* to 'hive'@'%';
Query OK, 0 rows affected (0.03 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.01 sec)
3.在安裝Install hive-2.0.0
[root@hadoop-01 tmp]# wget
[root@hadoop-01 tmp]# tar zxvf apache-hive-2.0.0-bin.tar.gz
[root@hadoop-01 tmp]# mv apache-hive-2.0.0-bin /hadoop/hive-remote-server
[root@hadoop-01 tmp]# cd /hadoop/hive-remote-server
[root@hadoop-01 hive-remote-server]# ll
total 588
drwxr-xr-x 3 root root 4096 Mar 29 23:19 bin
drwxr-xr-x 2 root root 4096 Mar 29 23:19 conf
drwxr-xr-x 4 root root 4096 Mar 29 23:19 examples
drwxr-xr-x 7 root root 4096 Mar 29 23:19 hcatalog
drwxr-xr-x 4 root root 12288 Mar 29 23:19 lib
-rw-r--r-- 1 root root 26335 Jan 22 12:28 LICENSE
-rw-r--r-- 1 root root 513 Jan 22 12:28 NOTICE
-rw-r--r-- 1 root root 4348 Feb 10 09:50 README.txt
-rw-r--r-- 1 root root 527063 Feb 10 09:56 RELEASE_NOTES.txt
drwxr-xr-x 4 root root 4096 Mar 29 23:19 scripts
[root@hadoop-01 hive-remote-server]#
4.Configure profile on hadoop-01
[root@hadoop-01 ~]# vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
export HADOOP_HOME=/hadoop/hadoop-2.7.2
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export HBASE_HOME=/hadoop/hbase-1.2.0
export ZOOKEEPER_HOME=/hadoop/zookeeper
export HIVE_HOME=/hadoop/hive-remote-server
export PATH=.:$HADOOP_HOME/bin:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:$PATH
[root@hadoop-01 ~]# source /etc/profile
[root@hadoop-01 ~]#
5.configure jdbc jar
[root@hadoop-01 tmp]# wget
[root@hadoop-01 tmp]# tar zxvf mysql-connector-java-5.1.36.tar.gz
[root@hadoop-01 tmp]# cd mysql-connector-java-5.1.36
[root@hadoop-01 mysql-connector-java-5.1.36]# ll
total 1428
-rw-r--r-- 1 root root 90430 Jun 20 2015 build.xml
-rw-r--r-- 1 root root 235082 Jun 20 2015 CHANGES
-rw-r--r-- 1 root root 18122 Jun 20 2015 COPYING
drwxr-xr-x 2 root root 4096 Mar 29 23:35 docs
-rw-r--r-- 1 root root 972009 Jun 20 2015 mysql-connector-java-5.1.36-bin.jar
-rw-r--r-- 1 root root 61423 Jun 20 2015 README
-rw-r--r-- 1 root root 63674 Jun 20 2015 README.txt
drwxr-xr-x 8 root root 4096 Jun 20 2015 src
[root@hadoop-01 mysql-connector-java-5.1.36]# cp mysql-connector-java-5.1.36-bin.jar $HIVE_HOME/lib/
6.Configure Hive服務端
[root@hadoop-01 ~]# cd $HIVE_HOME/conf
[root@hadoop-01 conf]# cp hive-default.xml.template hive-default.xml
# "hdfs://mycluster"是指$HADOOP_HOME/etc/hadoop/core-site.xml檔案的fs.defaultFS的值(NameNode HA URI)
#格式:jdbc:mysql:///?createDatabaseIfNotExist=true
點選(此處)摺疊或開啟
-
[root@hadoop-01 conf]# vi hive-site.xml
-
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-
-
<configuration>
-
-
<property>
-
<name>hive.metastore.warehouse.dir</name>
-
<value>hdfs://mycluster/user/hive_remote/warehouse</value>
-
</property>
-
-
<property>
-
<name>javax.jdo.option.ConnectionURL</name>
-
<value>jdbc:mysql://hadoop-01:3306/hive_remote_meta?createDatabaseIfNotExist=true</value>
-
<description>JDBC connect string for a JDBC metastore</description>
-
</property>
-
<property>
-
<name>javax.jdo.option.ConnectionDriverName</name>
-
<value>com.mysql.jdbc.Driver</value>
-
<description>Driver class name for a JDBC metastore</description>
-
</property>
-
-
<property>
-
<name>javax.jdo.option.ConnectionUserName</name>
-
<value>hive</value>
-
<description>username to use against metastore database</description>
-
</property>
-
<property>
-
<name>javax.jdo.option.ConnectionPassword</name>
-
<value>hive</value>
-
<description>password to use against metastore database</description>
-
</property>
-
<property>
-
<name>hive.hwi.war.file</name>
-
<value>${HIVE_HOME}/lib/hive-hwi-2.0.0.jar</value>
-
<description>This sets the path to the HWI war file, relative to ${HIVE_HOME}. </description>
-
</property>
-
</configuration>
-
-
"hive-site.xml" 26L, 1056C written
- [root@hadoop-01 conf]#
7.scp hive to client
[root@hadoop-01 hadoop]# pwd
/hadoop
[root@hadoop-01 hadoop]# scp -r hive-remote-server root@hadoop-02:/hadoop/hive-remote-client
8.Configure profile on hadoop-01
[root@hadoop-01 ~]# vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
export HADOOP_HOME=/hadoop/hadoop-2.7.2
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export HBASE_HOME=/hadoop/hbase-1.2.0
export ZOOKEEPER_HOME=/hadoop/zookeeper
export HIVE_HOME=/hadoop/hive-remote-client
export PATH=.:$HADOOP_HOME/bin:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:$PATH
[root@hadoop-01 ~]# source /etc/profile
[root@hadoop-01 ~]#
9.Configure Hive客戶端
點選(此處)摺疊或開啟
-
[root@hadoop-02 conf]# vi hive-site.xml
-
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-
-
<configuration>
-
<!-- thrift://<host_name>:<port> 預設埠是9083 -->
-
<property>
-
<name>hive.metastore.uris</name>
-
<value>thrift://hadoop-01:9083</value>
-
<description>Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.</description>
-
</property>
-
-
<!-- hive表的預設儲存路徑 -->
-
<property>
-
<name>hive.metastore.warehouse.dir</name>
-
<value>hdfs://mycluster/user/hive_remote/warehouse</value>
-
</property>
-
</configuration>
-
-
"hive-site.xml" 26L, 1056C written
- [root@hadoop-01 conf]#
10.hive服務端如果是第一次需要執行初始化命令:schematool -initSchema -dbType mysql
[root@hadoop-01 bin]# schematool -initSchema -dbType mysql
Metastore connection URL: jdbc:mysql://hadoop-01.telenav.cn:3306/hive_remote_meta?createDatabaseIfNotExist=true
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: hive
Starting metastore schema initialization to 2.0.0
Initialization script hive-schema-2.0.0.mysql.sql
Initialization script completed
schemaTool completed
[root@hadoop-01 bin]#
11.啟動hive服務端和客戶端
【服務端】:
hive --service metastore -p [port]
如果不加埠預設啟動:hive --service metastore,則預設監聽埠是:9083 ,注意客戶端中的埠配置需要和啟動監聽的埠一致。服務端啟動正常後,客戶端就可以執行hive操作了
### & 以後臺方式啟動
[root@hadoop-01 bin]# hive --service metastore &
Starting Hive Metastore Server
【客戶端】:
[root@hadoop-02 hive-remote-client]# cd bin
[root@hadoop-02 bin]# hive
Logging initialized using configuration in jar:file:/hadoop/hive-remote-client/lib/hive-common-2.0.0.jar!/hive-log4j2.properties
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
hive>
12.建立表,load資料驗證
## "tab製表符"分隔
[root@hadoop-02 bin]# vi /tmp/studentInfo.txt
1 a 26 110
1 a 26 113
2 b 11 222
[root@hadoop-02 bin]# hive
Logging initialized using configuration in jar:file:/hadoop/hive-remote-client/lib/hive-common-2.0.0.jar!/hive-log4j2.properties
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
hive>
> create table studentinfo (id int,name string, age int,tel string)
> row format delimited fields terminated by '\t'
> stored as textfile;
hive> load data local inpath '/tmp/studentInfo.txt' overwrite into table studentinfo;
Loading data to table default.studentinfo
Moved: 'hdfs://mycluster/user/hive_remote/warehouse/studentinfo/studentInfo.txt' to trash at: hdfs://mycluster/user/root/.Trash/Current
OK
Time taken: 2.941 seconds
hive> select * from studentinfo;
OK
1 a 26 113
2 b 11 222
Time taken: 1.607 seconds, Fetched: 2 row(s)
hive
13.檢視hdfs檔案系統
[root@hadoop-01 bin]# ps -ef|grep hive
root 11629 4509 1 19:39 pts/0 00:00:21 /usr/java/jdk1.7.0_67-cloudera/bin/java -Xmx256m -Djava.library.path=/hadoop/hadoop-2.7.2/lib -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/hadoop/hadoop-2.7.2/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/hadoop/hadoop-2.7.2 -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx512m -Dlog4j.configurationFile=hive-log4j2.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /hadoop/hive-remote-server/lib/hive-service-2.0.0.jar org.apache.hadoop.hive.metastore.HiveMetaStore
root 13351 4509 0 19:57 pts/0 00:00:00 grep hive
[root@hadoop-01 bin]# hadoop fs -ls /user/hive_remote/warehouse/studentinfo
Found 1 items
-rwx------ 3 root root 22 2016-04-16 19:54 /user/hive_remote/warehouse/studentinfo/studentInfo.txt
[root@hadoop-01 bin]# hadoop fs -cat /user/hive_remote/warehouse/studentinfo/studentInfo.txt
1 a 26 113
2 b 11 222
[root@hadoop-01 bin]#
14.參考
來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/30089851/viewspace-2082805/,如需轉載,請註明出處,否則將追究法律責任。
相關文章
- Hadoop2.7實戰v1.0之Hive-2.0.0+MySQL本地模式安裝HadoopHiveMySql模式
- Hadoop2.7實戰v1.0之YARN HAHadoopYarn
- Hadoop2.7實戰v1.0之HDFS HAHadoop
- Hadoop2.7實戰v1.0之JVM引數調優HadoopJVM
- Hadoop2.7實戰v1.0之Hive-2.0.0的Hiveserver2服務和beeline遠端除錯HadoopHiveServer除錯
- Hadoop2.7實戰v1.0之Linux引數調優HadoopLinux
- Hadoop2.7實戰v1.0之HBase1.1.5 HA分散式搭建Hadoop分散式
- Hive遠端模式安裝配置Hive模式
- Hadoop2.7實戰v1.0之start-balancer.sh與hdfs balancer資料均衡Hadoop
- Hadoop2.7實戰v1.0之Flume1.6.0搭建(Http Source-->Memory Chanel --> Hdfs Sink)HadoopHTTP
- Hadoop2.7實戰v1.0之Eclipse+Hive2.0.0的JDBC案例(最詳細搭建)HadoopEclipseHiveJDBC
- Hadoop2.7實戰v1.0之動態刪除DataNode(含NodeManager)節點(修改dfs.replication)Hadoop
- Hadoop2.7實戰v1.0之動態新增、刪除DataNode節點及複製策略導向Hadoop
- Hadoop2.7實戰v1.0之新增DataNode節點後,更改檔案複製策略dfs.replicationHadoop
- 八、hive3.1.2 安裝及其配置(本地模式和遠端模式)Hive模式
- vnc遠端安裝,在Linux中vnc遠端安裝的教程VNCLinux
- 【zabbix】zabbix遠端安裝部署
- Hadoop之Hive本地與遠端mysql資料庫管理模式安裝手冊HadoopHiveMySql資料庫模式
- vnc遠端安裝,10個步驟在Linux中遠端安裝VNC並連線VNCLinux
- kubernetes實戰篇之helm安裝
- Ubuntu安裝向日葵遠端控制Ubuntu
- Webkit遠端除錯協議實戰WebKit除錯協議
- unserialize() 實戰之 vBulletin 5.x.x 遠端程式碼執行
- 遠端建立SP模式模式
- PXE遠端CentOS安裝anaconda報錯CentOS
- linux安裝mongoDB與遠端連線LinuxMongoDB
- linux安裝redis和遠端連線LinuxRedis
- Redis 入門 - 0(安裝、遠端訪問)Redis
- Linux VNC遠端控制安裝日記LinuxVNC
- SSH遠端埠轉發實戰詳解
- 如何在命令列下遠端安裝終端服務命令列
- 【Python實戰】在阿里雲伺服器上安裝MongoDB,並實現遠端視覺化連線Python阿里伺服器MongoDB視覺化
- loadrunner12.6快速實戰之安裝指南
- Centos7安裝Xrdp遠端桌面服務CentOS
- vs安裝Remote-SSH 遠端登入主機REM
- ubunbu 12.04中安裝xrdp建立遠端桌面
- Mysql安裝和遠端登入--Centos7MySqlCentOS
- Redis安裝之叢集-哨兵模式(sentinel)模式Redis模式