高可用Hadoop平臺－應用JAR部署

哥不是小蘿莉發表於2015-05-12

1.概述

　　今天在觀察叢集時，發現NN節點的負載過高，雖然對NN節點的資源進行了調整，同時對NN節點上的應用程式進行重新打包調整，負載問題暫時得到緩解。但是，我想了想，這樣也不是長久之計。通過這個問題，我重新分析了一下以前應用部署架構圖，發現了一些問題的所在，之前的部署架構是，將打包的應用直接部署在Hadoop叢集上，雖然這沒什麼不好，但是我們分析得知，若是將應用部署在DN節點，那麼時間長了應用程式會不會搶佔DN節點的資源，那麼如果我們部署在NN節點上，又對NN節點計算任務時造成影響，於是，經過討論後，我們覺得應用程式不應該對Hadoop叢集造成干擾，他們應該是屬於一種鬆耦合的關係，所有的應用應該部署在一個AppServer叢集上。下面，我就為大家介紹今天的內容。

2.應用部署剖析

　　由於之前的應用程式直接部署在Hadoop叢集上，這堆叢集或多或少造成了一些影響。我們知道在本地開發Hadoop應用的時候，都可以直接執行相關Hadoop程式碼，這裡我們只用到了Hadoop的HDFS的地址，那我們為什麼不能直接將應用單獨部署呢？其實本地開發就可以看作是AppServer叢集的一個節點，藉助這個思路，我們將應用單獨打包後，部署在一個獨立的AppServer叢集，只需要用到Hadoop叢集的HDFS地址即可，這裡需要注意的是，保證AppServer叢集與Hadoop叢集在同一個網段。下面我給出解耦後應用部署架構圖，如下圖所示：

　　從圖中我們可以看出，AppServer叢集想Hadoop叢集提交作業，兩者之間的資料互動，只需用到Hadoop的HDFS地址和Java API。在AppServer上的應用不會影響到Hadoop叢集的正常執行。

3.示例

　　下面為大家演示相關示例，以WordCountV2為例子，程式碼如下所示：

package cn.hadoop.hdfs.main;

import java.io.IOException;
import java.util.Random;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import cn.hadoop.hdfs.util.SystemConfig;

/**
 * @Date Apr 23, 2015
 *
 * @Author dengjie
 *
 * @Note Wordcount的例子是一個比較經典的mapreduce例子，可以叫做Hadoop版的hello world。
 *       它將檔案中的單詞分割取出，然後shuffle，sort（map過程），接著進入到彙總統計
 *       （reduce過程），最後寫道hdfs中。基本流程就是這樣。
 */
public class WordCountV2 {

    private static Logger logger = LoggerFactory.getLogger(WordCountV2.class);
    private static Configuration conf;

    /**
     * 設定高可用叢集連線資訊
     */
    static {
        String tag = SystemConfig.getProperty("dev.tag");
        String[] hosts = SystemConfig.getPropertyArray(tag + ".hdfs.host", ",");
        conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://cluster1");
        conf.set("dfs.nameservices", "cluster1");
        conf.set("dfs.ha.namenodes.cluster1", "nna,nns");
        conf.set("dfs.namenode.rpc-address.cluster1.nna", hosts[0]);
        conf.set("dfs.namenode.rpc-address.cluster1.nns", hosts[1]);
        conf.set("dfs.client.failover.proxy.provider.cluster1",
                "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");    
    }

    public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {

        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();

        /**
         * 原始檔：a b b
         * 
         * map之後：
         * 
         * a 1
         * 
         * b 1
         * 
         * b 1
         */
        public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
            StringTokenizer itr = new StringTokenizer(value.toString());// 整行讀取
            while (itr.hasMoreTokens()) {
                word.set(itr.nextToken());// 按空格分割單詞
                context.write(word, one);// 每次統計出來的單詞+1
            }
        }
    }

    /**
     * reduce之前：
     * 
     * a 1
     * 
     * b 1
     * 
     * b 1
     * 
     * reduce之後:
     * 
     * a 1
     * 
     * b 2
     */
    public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
        private IntWritable result = new IntWritable();

        public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException,
                InterruptedException {
            int sum = 0;
            for (IntWritable val : values) {
                sum += val.get();// 分組累加
            }
            result.set(sum);
            context.write(key, result);// 按相同的key輸出
        }
    }

    public static void main(String[] args) {
        try {
            if (args.length < 1) {
                logger.info("args length is 0");
                run("hello.txt");
            } else {
                logger.info("args length is not 0");
                run(args[0]);
            }
        } catch (Exception ex) {
            ex.printStackTrace();
            logger.error(ex.getMessage());
        }
    }

    private static void run(String name) throws Exception {
        long randName = new Random().nextLong();// 重定向輸出目錄
        logger.info("output name is [" + randName + "]");

        Job job = Job.getInstance(conf);
        job.setJarByClass(WordCountV2.class);
        job.setMapperClass(TokenizerMapper.class);// 指定Map計算的類
        job.setCombinerClass(IntSumReducer.class);// 合併的類
        job.setReducerClass(IntSumReducer.class);// Reduce的類
        job.setOutputKeyClass(Text.class);// 輸出Key型別
        job.setOutputValueClass(IntWritable.class);// 輸出值型別

        String sysInPath = SystemConfig.getProperty("hdfs.input.path.v2");
        String realInPath = String.format(sysInPath, name);
        String syOutPath = SystemConfig.getProperty("hdfs.output.path.v2");
        String realOutPath = String.format(syOutPath, randName);

        FileInputFormat.addInputPath(job, new Path(realInPath));// 指定輸入路徑
        FileOutputFormat.setOutputPath(job, new Path(realOutPath));// 指定輸出路徑

        System.exit(job.waitForCompletion(true) ? 0 : 1);// 執行完MR任務後退出應用
    }
}

　　在本地IDE中執行正常，截圖如下所示：

4.應用打包部署

　　然後，我們將WordCountV2應用打包後部署到AppServer1節點，這裡由於工程是基於Maven結構的，我們使用Maven命令直接打包，打包命令如下所示：

mvn assembly:assembly

　　然後，我們使用scp命令將打包後的JAR檔案上傳到AppServer1節點，上傳命令如下所示：

scp hadoop-ubas-1.0.0-jar-with-dependencies.jar hadoop@apps:~/

　　接著，我們在AppServer1節點上執行我們打包好的應用，執行命令如下所示：

java -jar hadoop-ubas-1.0.0-jar-with-dependencies.jar

　　但是，這裡卻很無奈的報錯了，錯誤資訊如下所示：

java.io.IOException: No FileSystem for scheme: hdfs
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:354)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
    at cn.hadoop.hdfs.main.WordCountV2.run(WordCountV2.java:134)
    at cn.hadoop.hdfs.main.WordCountV2.main(WordCountV2.java:108)
2015-05-10 23:31:21 ERROR [WordCountV2.main] - No FileSystem for scheme: hdfs

5.錯誤分析

　　首先，我們來定位下問題原因，我將打包後的JAR在Hadoop叢集上執行，是可以完成良好的執行，並計算出結果資訊的，為什麼在非Hadoop叢集卻報錯呢？難道是這種架構方式不對？經過仔細的分析錯誤資訊，和我們的Maven依賴環境，問題原因定位出來了，這裡我們使用了Maven的assembly外掛來打包應用。只是因為當我們使用Maven元件時，它將所有的JARS合併到一個檔案中，所有的META-INFO/services/org.apache.hadoop.fs.FileSystem被互相覆蓋，僅保留最後一個加入的，在這種情況下FileSystem的列表從Hadoop-Commons重寫到Hadoop-HDFS的列表，而DistributedFileSystem就會找不到相應的宣告資訊。因而，就會出現上述錯誤資訊。在原因找到後，我們剩下的就是去找到解決方法，這裡通過分析，我找到的解決辦法如下，在Loading相關Hadoop的Configuration時，我們設定相關FileSystem即可，配置程式碼如下所示：

conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
conf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());

　　接下來，我們重新打包應用，然後在AppServer1節點執行該應用，執行正常，並正常統計結果，執行日誌如下所示：

[hadoop@apps example]$ java -jar hadoop-ubas-1.0.0-jar-with-dependencies.jar 
2015-05-11 00:08:15 INFO  [SystemConfig.main] - Successfully loaded default properties.
2015-05-11 00:08:15 INFO  [WordCountV2.main] - args length is 0
2015-05-11 00:08:15 INFO  [WordCountV2.main] - output name is [6876390710620561863]
2015-05-11 00:08:16 WARN  [NativeCodeLoader.main] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-05-11 00:08:17 INFO  [deprecation.main] - session.id is deprecated. Instead, use dfs.metrics.session-id
2015-05-11 00:08:17 INFO  [JvmMetrics.main] - Initializing JVM Metrics with processName=JobTracker, sessionId=
2015-05-11 00:08:17 WARN  [JobSubmitter.main] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2015-05-11 00:08:17 INFO  [FileInputFormat.main] - Total input paths to process : 1
2015-05-11 00:08:18 INFO  [JobSubmitter.main] - number of splits:1
2015-05-11 00:08:18 INFO  [JobSubmitter.main] - Submitting tokens for job: job_local519626586_0001
2015-05-11 00:08:18 INFO  [Job.main] - The url to track the job: http://localhost:8080/
2015-05-11 00:08:18 INFO  [Job.main] - Running job: job_local519626586_0001
2015-05-11 00:08:18 INFO  [LocalJobRunner.Thread-14] - OutputCommitter set in config null
2015-05-11 00:08:18 INFO  [LocalJobRunner.Thread-14] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2015-05-11 00:08:18 INFO  [LocalJobRunner.Thread-14] - Waiting for map tasks
2015-05-11 00:08:18 INFO  [LocalJobRunner.LocalJobRunner Map Task Executor #0] - Starting task: attempt_local519626586_0001_m_000000_0
2015-05-11 00:08:18 INFO  [Task.LocalJobRunner Map Task Executor #0] -  Using ResourceCalculatorProcessTree : [ ]
2015-05-11 00:08:18 INFO  [MapTask.LocalJobRunner Map Task Executor #0] - Processing split: hdfs://cluster1/home/hdfs/test/in/hello.txt:0+24
2015-05-11 00:08:19 INFO  [MapTask.LocalJobRunner Map Task Executor #0] - (EQUATOR) 0 kvi 26214396(104857584)
2015-05-11 00:08:19 INFO  [MapTask.LocalJobRunner Map Task Executor #0] - mapreduce.task.io.sort.mb: 100
2015-05-11 00:08:19 INFO  [MapTask.LocalJobRunner Map Task Executor #0] - soft limit at 83886080
2015-05-11 00:08:19 INFO  [MapTask.LocalJobRunner Map Task Executor #0] - bufstart = 0; bufvoid = 104857600
2015-05-11 00:08:19 INFO  [MapTask.LocalJobRunner Map Task Executor #0] - kvstart = 26214396; length = 6553600
2015-05-11 00:08:19 INFO  [MapTask.LocalJobRunner Map Task Executor #0] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-05-11 00:08:19 INFO  [LocalJobRunner.LocalJobRunner Map Task Executor #0] - 
2015-05-11 00:08:19 INFO  [MapTask.LocalJobRunner Map Task Executor #0] - Starting flush of map output
2015-05-11 00:08:19 INFO  [MapTask.LocalJobRunner Map Task Executor #0] - Spilling map output
2015-05-11 00:08:19 INFO  [MapTask.LocalJobRunner Map Task Executor #0] - bufstart = 0; bufend = 72; bufvoid = 104857600
2015-05-11 00:08:19 INFO  [MapTask.LocalJobRunner Map Task Executor #0] - kvstart = 26214396(104857584); kvend = 26214352(104857408); length = 45/6553600
2015-05-11 00:08:19 INFO  [MapTask.LocalJobRunner Map Task Executor #0] - Finished spill 0
2015-05-11 00:08:19 INFO  [Task.LocalJobRunner Map Task Executor #0] - Task:attempt_local519626586_0001_m_000000_0 is done. And is in the process of committing
2015-05-11 00:08:19 INFO  [LocalJobRunner.LocalJobRunner Map Task Executor #0] - map
2015-05-11 00:08:19 INFO  [Task.LocalJobRunner Map Task Executor #0] - Task 'attempt_local519626586_0001_m_000000_0' done.
2015-05-11 00:08:19 INFO  [LocalJobRunner.LocalJobRunner Map Task Executor #0] - Finishing task: attempt_local519626586_0001_m_000000_0
2015-05-11 00:08:19 INFO  [LocalJobRunner.Thread-14] - map task executor complete.
2015-05-11 00:08:19 INFO  [LocalJobRunner.Thread-14] - Waiting for reduce tasks
2015-05-11 00:08:19 INFO  [LocalJobRunner.pool-6-thread-1] - Starting task: attempt_local519626586_0001_r_000000_0
2015-05-11 00:08:19 INFO  [Task.pool-6-thread-1] -  Using ResourceCalculatorProcessTree : [ ]
2015-05-11 00:08:19 INFO  [ReduceTask.pool-6-thread-1] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@16769723
2015-05-11 00:08:19 INFO  [MergeManagerImpl.pool-6-thread-1] - MergerManager: memoryLimit=177399392, maxSingleShuffleLimit=44349848, mergeThreshold=117083600, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-05-11 00:08:19 INFO  [EventFetcher.EventFetcher for fetching Map Completion Events] - attempt_local519626586_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-05-11 00:08:19 INFO  [LocalFetcher.localfetcher#1] - localfetcher#1 about to shuffle output of map attempt_local519626586_0001_m_000000_0 decomp: 50 len: 54 to MEMORY
2015-05-11 00:08:19 INFO  [InMemoryMapOutput.localfetcher#1] - Read 50 bytes from map-output for attempt_local519626586_0001_m_000000_0
2015-05-11 00:08:19 INFO  [MergeManagerImpl.localfetcher#1] - closeInMemoryFile -> map-output of size: 50, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->50
2015-05-11 00:08:19 INFO  [EventFetcher.EventFetcher for fetching Map Completion Events] - EventFetcher is interrupted.. Returning
2015-05-11 00:08:19 INFO  [LocalJobRunner.pool-6-thread-1] - 1 / 1 copied.
2015-05-11 00:08:19 INFO  [MergeManagerImpl.pool-6-thread-1] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-05-11 00:08:19 INFO  [Merger.pool-6-thread-1] - Merging 1 sorted segments
2015-05-11 00:08:19 INFO  [Merger.pool-6-thread-1] - Down to the last merge-pass, with 1 segments left of total size: 46 bytes
2015-05-11 00:08:19 INFO  [MergeManagerImpl.pool-6-thread-1] - Merged 1 segments, 50 bytes to disk to satisfy reduce memory limit
2015-05-11 00:08:19 INFO  [MergeManagerImpl.pool-6-thread-1] - Merging 1 files, 54 bytes from disk
2015-05-11 00:08:19 INFO  [MergeManagerImpl.pool-6-thread-1] - Merging 0 segments, 0 bytes from memory into reduce
2015-05-11 00:08:19 INFO  [Merger.pool-6-thread-1] - Merging 1 sorted segments
2015-05-11 00:08:19 INFO  [Merger.pool-6-thread-1] - Down to the last merge-pass, with 1 segments left of total size: 46 bytes
2015-05-11 00:08:19 INFO  [LocalJobRunner.pool-6-thread-1] - 1 / 1 copied.
2015-05-11 00:08:19 INFO  [deprecation.pool-6-thread-1] - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2015-05-11 00:08:19 INFO  [Job.main] - Job job_local519626586_0001 running in uber mode : false
2015-05-11 00:08:19 INFO  [Job.main] -  map 100% reduce 0%
2015-05-11 00:08:19 INFO  [Task.pool-6-thread-1] - Task:attempt_local519626586_0001_r_000000_0 is done. And is in the process of committing
2015-05-11 00:08:19 INFO  [LocalJobRunner.pool-6-thread-1] - 1 / 1 copied.
2015-05-11 00:08:19 INFO  [Task.pool-6-thread-1] - Task attempt_local519626586_0001_r_000000_0 is allowed to commit now
2015-05-11 00:08:19 INFO  [FileOutputCommitter.pool-6-thread-1] - Saved output of task 'attempt_local519626586_0001_r_000000_0' to hdfs://cluster1/home/hdfs/test/out/6876390710620561863/_temporary/0/task_local519626586_0001_r_000000
2015-05-11 00:08:19 INFO  [LocalJobRunner.pool-6-thread-1] - reduce > reduce
2015-05-11 00:08:19 INFO  [Task.pool-6-thread-1] - Task 'attempt_local519626586_0001_r_000000_0' done.
2015-05-11 00:08:19 INFO  [LocalJobRunner.pool-6-thread-1] - Finishing task: attempt_local519626586_0001_r_000000_0
2015-05-11 00:08:19 INFO  [LocalJobRunner.Thread-14] - reduce task executor complete.
2015-05-11 00:08:20 INFO  [Job.main] -  map 100% reduce 100%
2015-05-11 00:08:20 INFO  [Job.main] - Job job_local519626586_0001 completed successfully
2015-05-11 00:08:20 INFO  [Job.main] - Counters: 38
    File System Counters
        FILE: Number of bytes read=77813788
        FILE: Number of bytes written=78928898
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=48
        HDFS: Number of bytes written=24
        HDFS: Number of read operations=13
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=4
    Map-Reduce Framework
        Map input records=2
        Map output records=12
        Map output bytes=72
        Map output materialized bytes=54
        Input split bytes=108
        Combine input records=12
        Combine output records=6
        Reduce input groups=6
        Reduce shuffle bytes=54
        Reduce input records=6
        Reduce output records=6
        Spilled Records=12
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=53
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
        Total committed heap usage (bytes)=241442816
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=24
    File Output Format Counters 
        Bytes Written=24

6.總結

　　這裡需要注意的是，我們應用部署架構沒問題，思路是正確的，問題出在打包上，在打包的時候需要特別注意，另外，有些同學使用IDE的Export匯出時也要注意一下，相關依賴是否存在，還有常見的第三方打包工具Fat，這個也是需要注意的。

7.結束語

　　這篇部落格就和大家分享到這裡，如果大家在研究學習的過程當中有什麼問題，可以加群進行討論或傳送郵件給我，我會盡我所能為您解答，與君共勉！

高可用Hadoop平臺－Ganglia安裝部署
2015-05-21
Hadoop
高可用Hadoop平臺－探索
2015-03-18
Hadoop
高可用Hadoop平臺－Hue In Hadoop
2015-05-25
Hadoop
高可用Hadoop平臺－實戰
2015-04-01
Hadoop
高可用Hadoop平臺－啟航
2015-03-13
Hadoop
配置高可用的Hadoop平臺
2015-02-25
Hadoop
高可用Hadoop平臺－答疑篇
2015-04-07
Hadoop
高可用Hadoop平臺－整合Hive HAProxy
2015-03-26
HadoopHive
高可用Hadoop平臺－執行MapReduce程式
2015-03-24
Hadoop
高可用Hadoop平臺－HBase叢集搭建
2015-05-19
Hadoop
高可用Hadoop平臺－Oozie工作流
2015-08-20
Hadoop
高可用Hadoop平臺－實戰尾聲篇
2016-04-19
Hadoop
高可用Hadoop平臺－Oozie工作流之Hadoop排程
2015-08-25
Hadoop
高可用Hadoop平臺－Flume NG實戰圖解篇
2015-04-30
Hadoop圖解
基於linux-HA 的pg 高可用平臺正式部署
2013-04-25
Linux
LAMP平臺部署及應用（一）
2017-11-15
LAMP
揭秘貨拉拉大模型應用平臺的應用部署
2023-12-29
大模型
假期平臺子系統高可用方案
2015-03-21
Hadoop大資料平臺之HBase部署
2020-11-24
Hadoop大資料
Hadoop大資料平臺之Kafka部署
2020-11-24
Hadoop大資料Kafka
LAMP平臺服務簡介、部署及應用
2020-12-15
LAMP
高可用架構之高可用的應用和服務
2017-10-28
架構
DCOS雲平臺之業務多應用部署改造方案
2020-09-24
springcloud-高可用部署
2019-07-19
SpringGCCloud
Canal高可用架構部署
2021-03-25
架構
Spark Standalone模式高可用部署
2020-11-12
Spark模式
在 SAP 雲平臺上部署和執行 Docker 應用
2022-07-22
Docker
構建高併發高可用的電商平臺架構實踐
2015-03-26
架構
高效能、高可用平臺架構演變史
2018-07-29
架構
微信平臺應用
2019-05-11
keeplied + nginx +tomcat 高可用部署
2020-03-15
NginxTomcat
mysql MHA 高可用架構部署
2016-03-01
MySql架構
Hadoop框架：HDFS高可用環境配置
2020-10-27
Hadoop框架
將SAP UI5應用部署在Heroku雲平臺裡
2020-09-28
UI
Nginx + Keepalived 高可用叢集部署
2023-03-09
Nginx
Rabbitmq叢集高可用部署詳細
2017-03-11
MQ
Redis之Sentinel高可用安裝部署
2017-02-23
Redis
MySQL高可用MMM方案安裝部署
2014-02-18
MySql