### 歡迎訪問我的GitHub
> 這裡分類和彙總了欣宸的全部原創(含配套原始碼)：[https://github.com/zq2599/blog_demos](https://github.com/zq2599/blog_demos)

### 本篇概覽
- 本文是[《docker下，極速搭建spark叢集(含hdfs叢集)》](https://blog.51cto.com/u_13674465/5584009)的續篇，前文將spark叢集搭建成功並進行了簡單的驗證，但是存在以下幾個小問題：
1.  spark只有一個work節點，只適合處理小資料量的任務，遇到大量資料的任務要消耗更多時間；
2. hdfs的檔案目錄和docker安裝目錄在一起，如果要儲存大量檔案，很可能由於磁碟空間不足導致上傳失敗；
3. master的4040和work的8080埠都沒有開放，看不到job、stage、executor的執行情況；

- 今天就來調整系統引數，解決上述問題；

### 最初的docker-compose.yml內容
- 最佳化前的docker-compose.yml內容如下所示：

```yml
version: "2.2"
services:
  namenode:
    image: bde2020/hadoop-namenode:1.1.0-hadoop2.7.1-java8
    container_name: namenode
    volumes:
      - hadoop_namenode:/hadoop/dfs/name
      - ./input_files:/input_files
    environment:
      - CLUSTER_NAME=test
    env_file:
      - ./hadoop.env
    ports:
      - 50070:50070
  
  resourcemanager:
    image: bde2020/hadoop-resourcemanager:1.1.0-hadoop2.7.1-java8
    container_name: resourcemanager
    depends_on:
      - namenode
      - datanode1
      - datanode2
    env_file:
      - ./hadoop.env
  
  historyserver:
    image: bde2020/hadoop-historyserver:1.1.0-hadoop2.7.1-java8
    container_name: historyserver
    depends_on:
      - namenode
      - datanode1
      - datanode2
    volumes:
      - hadoop_historyserver:/hadoop/yarn/timeline
    env_file:
      - ./hadoop.env
  
  nodemanager1:
    image: bde2020/hadoop-nodemanager:1.1.0-hadoop2.7.1-java8
    container_name: nodemanager1
    depends_on:
      - namenode
      - datanode1
      - datanode2
    env_file:
      - ./hadoop.env
  
  datanode1:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    container_name: datanode1
    depends_on:
      - namenode
    volumes:
      - hadoop_datanode1:/hadoop/dfs/data
    env_file:
      - ./hadoop.env
  
  datanode2:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    container_name: datanode2
    depends_on:
      - namenode
    volumes:
      - hadoop_datanode2:/hadoop/dfs/data
    env_file:
      - ./hadoop.env
  
  datanode3:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    container_name: datanode3
    depends_on:
      - namenode
    volumes:
      - hadoop_datanode3:/hadoop/dfs/data
    env_file:
      - ./hadoop.env

master:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: master
    command: bin/spark-class org.apache.spark.deploy.master.Master -h master
    hostname: master
    environment:
      MASTER: spark://master:7077
      SPARK_CONF_DIR: /conf
      SPARK_PUBLIC_DNS: localhost
    links:
      - namenode
    expose:
      - 7001
      - 7002
      - 7003
      - 7004
      - 7005
      - 7077
      - 6066
    ports:
      - 6066:6066
      - 7077:7077
      - 8080:8080
    volumes:
      - ./conf/master:/conf
      - ./data:/tmp/data
      - ./jars:/root/jars

worker:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: worker
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
    hostname: worker
    environment:
      SPARK_CONF_DIR: /conf
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 1g
      SPARK_WORKER_PORT: 8881
      SPARK_WORKER_WEBUI_PORT: 8081
      SPARK_PUBLIC_DNS: localhost
    links:
      - master
    expose:
      - 7012
      - 7013
      - 7014
      - 7015
      - 8881
    ports:
      - 8081:8081
    volumes:
      - ./conf/worker:/conf
      - ./data:/tmp/data

volumes:
  hadoop_namenode:
  hadoop_datanode1:
  hadoop_datanode2:
  hadoop_datanode3:
  hadoop_historyserver:
```

- 接下來開始最佳化；

### 實戰環境資訊
- 本次實戰所用的電腦是聯想筆記本：
1. CPU：i5-6300HQ（四核四執行緒）
2. 記憶體：16G
3. 硬碟：256G的NVMe再加500G機械硬碟
4. 系統：Deepin15
5. docker：18.09.1
6. docker-compose：1.17.1
7. spark：2.3.0
8. hdfs：2.7.1

### 調整work節點數量
- 由於記憶體有16G，於是打算將work節點數從1個調整到6個，調整後work容器的配置如下：

```yml
worker1:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: worker1
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
    hostname: worker1
    environment:
      SPARK_CONF_DIR: /conf
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 2g
      SPARK_WORKER_PORT: 8881
      SPARK_WORKER_WEBUI_PORT: 8081
      SPARK_PUBLIC_DNS: localhost
    links:
      - master
    expose:
      - 7012
      - 7013
      - 7014
      - 7015
      - 8881
    volumes:
      - ./conf/worker1:/conf
      - ./data/worker1:/tmp/data
worker2:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: worker2
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
    hostname: worker2
    environment:
      SPARK_CONF_DIR: /conf
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 2g
      SPARK_WORKER_PORT: 8881
      SPARK_WORKER_WEBUI_PORT: 8082
      SPARK_PUBLIC_DNS: localhost
    links:
      - master
    expose:
      - 7012
      - 7013
      - 7014
      - 7015
      - 8881
    volumes:
      - ./conf/worker2:/conf
      - ./data/worker2:/tmp/data
```
- 如上所示，注意volumes引數，都對映在了docker-compose.yml同一層級的conf和data兩個目錄下，這裡只貼出了worker1和worker2的內容，worker3-worker6的內容都是類似的；

### hdfs的檔案目錄導致的磁碟空間不足問題
- 先來看下hdfs的檔案目錄配置：

```yml
volumes:
      - hadoop_datanode1:/hadoop/dfs/data
```

- 上面的**hadoop_datanode1**資料卷的配置在docker-compose.yml的最底部，是預設宣告，如下：

```yml
volumes:
  hadoop_namenode:
  hadoop_datanode1:
  hadoop_datanode2:
  hadoop_datanode3:
  hadoop_historyserver:
```

- 在容器執行狀態，執行命令**docker inspect datanode1**檢視容器資訊，和資料卷相關的資訊如下所示：

```shell
"Mounts": [
            {
                "Type": "volume",
                "Name": "temp_hadoop_datanode1",
                "Source": "/var/lib/docker/volumes/temp_hadoop_datanode1/_data",
                "Destination": "/hadoop/dfs/data",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            }
        ]
```
- 可見hdfs容器的檔案目錄對應的是宿主機的**/var/lib/docker/volumes**；
- 用**df -m**看看磁碟空間情況，如下所示，"/var/lib/docker/volumes"所在的"/dev/nvme0n1p3"裝置可用空間只有20多G（29561），顯然在儲存大量檔案時這個空間是不夠的，而且hdfs的預設副本數為3：

```shell
root@willzhao-deepin:/data/work/spark/temp# df -m
檔案系統        1M-塊   已用   可用 已用% 掛載點
udev             7893      0   7893    0% /dev
tmpfs            1584      4   1581    1% /run
/dev/nvme0n1p3  43927  12107  29561   30% /
tmpfs            7918      0   7918    0% /dev/shm
tmpfs               5      1      5    1% /run/lock
tmpfs            7918      0   7918    0% /sys/fs/cgroup
/dev/nvme0n1p4  87854    181  83169    1% /home
/dev/nvme0n1p1    300      7    293    3% /boot/efi
/dev/sda1      468428 109152 335430   25% /data
tmpfs            1584      1   1584    1% /run/user/108
tmpfs            1584      0   1584    0% /run/user/0
```

- 上面的磁碟資訊顯示裝置**/dev/sda1**還有300G，所以hdfs的檔案目錄對映到/dev/sda1就能緩解磁碟空間問題了，於是修改docker-compose.yml檔案中hdfs的三個資料節點的配置，修改後如下：

```yml
datanode1:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    container_name: datanode1
    depends_on:
      - namenode
    volumes:
      - ./hadoop/datanode1:/hadoop/dfs/data
    env_file:
      - ./hadoop.env
  
  datanode2:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    container_name: datanode2
    depends_on:
      - namenode
    volumes:
      - ./hadoop/datanode2:/hadoop/dfs/data
    env_file:
      - ./hadoop.env
  
  datanode3:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    container_name: datanode3
    depends_on:
      - namenode
    volumes:
      - ./hadoop/datanode3:/hadoop/dfs/data
    env_file:
      - ./hadoop.env
```
- 再將下面這段配置刪除：

```yml
volumes:
  hadoop_namenode:
  hadoop_datanode1:
  hadoop_datanode2:
  hadoop_datanode3:
  hadoop_historyserver:
```

### 開發master的4040和work的8080埠
- 任務執行過程中，如果有UI頁面來觀察詳情，可以幫助我們更全面直觀的瞭解執行情況，所以需要修改配置開放埠；
- 如下所示，**expose**引數增加4040，表示對外暴露4040埠，**ports**引數增加4040:4040，表示容器的4040對映到宿主機的4040埠：

```yml
  master:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: master
    command: bin/spark-class org.apache.spark.deploy.master.Master -h master
    hostname: master
    environment:
      MASTER: spark://master:7077
      SPARK_CONF_DIR: /conf
      SPARK_PUBLIC_DNS: localhost
    links:
      - namenode
    expose:
      - 4040
      - 7001
      - 7002
      - 7003
      - 7004
      - 7005
      - 7077
      - 6066
    ports:
      - 4040:4040
      - 6066:6066
      - 7077:7077
      - 8080:8080
    volumes:
      - ./conf/master:/conf
      - ./data:/tmp/data
      - ./jars:/root/jars
```

- worker的web埠同樣需要開啟，訪問worker的web頁面可以觀察worker的狀態，並且檢視任務日誌（這個很重要），這裡要注意的是由於有多個worker，所以要對映到宿主機的多個埠，如下配置，workder1的environment.SPARK_WORKER_WEBUI_PORT配置為8081，並且暴露8081，再將容器的8081對映到宿主機的8081，workder2的environment.SPARK_WORKER_WEBUI_PORT配置為8082，並且暴露8082，再將容器的8082對映到宿主機的8082：

```yml
 worker1:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: worker1
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
    hostname: worker1
    environment:
      SPARK_CONF_DIR: /conf
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 2g
      SPARK_WORKER_PORT: 8881
      SPARK_WORKER_WEBUI_PORT: 8081
      SPARK_PUBLIC_DNS: localhost
    links:
      - master
    expose:
      - 7012
      - 7013
      - 7014
      - 7015
      - 8881
      - 8081
    ports:
      - 8081:8081
    volumes:
      - ./conf/worker1:/conf
      - ./data/worker1:/tmp/data

worker2:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: worker2
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
    hostname: worker2
    environment:
      SPARK_CONF_DIR: /conf
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 2g
      SPARK_WORKER_PORT: 8881
      SPARK_WORKER_WEBUI_PORT: 8082
      SPARK_PUBLIC_DNS: localhost
    links:
      - master
    expose:
      - 7012
      - 7013
      - 7014
      - 7015
      - 8881
      - 8082
    ports:
      - 8082:8082
    volumes:
      - ./conf/worker2:/conf
      - ./data/worker2:/tmp/data  
```
- worker3-worker6的配置與上面類似，注意用不同的埠號；

- 至此，修改已經完成，最終版的docker-compose.yml內容如下：

```yml
version: "2.2"
services:
  namenode:
    image: bde2020/hadoop-namenode:1.1.0-hadoop2.7.1-java8
    container_name: namenode
    volumes:
      - ./hadoop/namenode:/hadoop/dfs/name
      - ./input_files:/input_files
    environment:
      - CLUSTER_NAME=test
    env_file:
      - ./hadoop.env
    ports:
      - 50070:50070
  
  resourcemanager:
    image: bde2020/hadoop-resourcemanager:1.1.0-hadoop2.7.1-java8
    container_name: resourcemanager
    depends_on:
      - namenode
      - datanode1
      - datanode2
    env_file:
      - ./hadoop.env
  
  historyserver:
    image: bde2020/hadoop-historyserver:1.1.0-hadoop2.7.1-java8
    container_name: historyserver
    depends_on:
      - namenode
      - datanode1
      - datanode2
    volumes:
      - ./hadoop/historyserver:/hadoop/yarn/timeline
    env_file:
      - ./hadoop.env
  
  nodemanager1:
    image: bde2020/hadoop-nodemanager:1.1.0-hadoop2.7.1-java8
    container_name: nodemanager1
    depends_on:
      - namenode
      - datanode1
      - datanode2
    env_file:
      - ./hadoop.env
  
  datanode1:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    container_name: datanode1
    depends_on:
      - namenode
    volumes:
      - ./hadoop/datanode1:/hadoop/dfs/data
    env_file:
      - ./hadoop.env
  
  datanode2:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    container_name: datanode2
    depends_on:
      - namenode
    volumes:
      - ./hadoop/datanode2:/hadoop/dfs/data
    env_file:
      - ./hadoop.env
  
  datanode3:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    container_name: datanode3
    depends_on:
      - namenode
    volumes:
      - ./hadoop/datanode3:/hadoop/dfs/data
    env_file:
      - ./hadoop.env

master:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: master
    command: bin/spark-class org.apache.spark.deploy.master.Master -h master
    hostname: master
    environment:
      MASTER: spark://master:7077
      SPARK_CONF_DIR: /conf
      SPARK_PUBLIC_DNS: localhost
    links:
      - namenode
    expose:
      - 4040
      - 7001
      - 7002
      - 7003
      - 7004
      - 7005
      - 7077
      - 6066
    ports:
      - 4040:4040
      - 6066:6066
      - 7077:7077
      - 8080:8080
    volumes:
      - ./conf/master:/conf
      - ./data:/tmp/data
      - ./jars:/root/jars

worker1:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: worker1
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
    hostname: worker1
    environment:
      SPARK_CONF_DIR: /conf
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 2g
      SPARK_WORKER_PORT: 8881
      SPARK_WORKER_WEBUI_PORT: 8081
      SPARK_PUBLIC_DNS: localhost
    links:
      - master
    expose:
      - 7012
      - 7013
      - 7014
      - 7015
      - 8881
      - 8081
    ports:
      - 8081:8081
    volumes:
      - ./conf/worker1:/conf
      - ./data/worker1:/tmp/data

worker2:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: worker2
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
    hostname: worker2
    environment:
      SPARK_CONF_DIR: /conf
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 2g
      SPARK_WORKER_PORT: 8881
      SPARK_WORKER_WEBUI_PORT: 8082
      SPARK_PUBLIC_DNS: localhost
    links:
      - master
    expose:
      - 7012
      - 7013
      - 7014
      - 7015
      - 8881
      - 8082
    ports:
      - 8082:8082
    volumes:
      - ./conf/worker2:/conf
      - ./data/worker2:/tmp/data

worker3:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: worker3
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
    hostname: worker3
    environment:
      SPARK_CONF_DIR: /conf
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 2g
      SPARK_WORKER_PORT: 8881
      SPARK_WORKER_WEBUI_PORT: 8083
      SPARK_PUBLIC_DNS: localhost
    links:
      - master
    expose:
      - 7012
      - 7013
      - 7014
      - 7015
      - 8881
      - 8083
    ports:
      - 8083:8083
    volumes:
      - ./conf/worker3:/conf
      - ./data/worker3:/tmp/data

worker4:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: worker4
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
    hostname: worker4
    environment:
      SPARK_CONF_DIR: /conf
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 2g
      SPARK_WORKER_PORT: 8881
      SPARK_WORKER_WEBUI_PORT: 8084
      SPARK_PUBLIC_DNS: localhost
    links:
      - master
    expose:
      - 7012
      - 7013
      - 7014
      - 7015
      - 8881
      - 8084
    ports:
      - 8084:8084
    volumes:
      - ./conf/worker4:/conf
      - ./data/worker4:/tmp/data

worker5:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: worker5
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
    hostname: worker5
    environment:
      SPARK_CONF_DIR: /conf
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 2g
      SPARK_WORKER_PORT: 8881
      SPARK_WORKER_WEBUI_PORT: 8085
      SPARK_PUBLIC_DNS: localhost
    links:
      - master
    expose:
      - 7012
      - 7013
      - 7014
      - 7015
      - 8881
      - 8085
    ports:
      - 8085:8085
    volumes:
      - ./conf/worker5:/conf
      - ./data/worker5:/tmp/data

worker6:
    image: gettyimages/spark:2.3.0-hadoop-2.8
    container_name: worker6
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
    hostname: worker6
    environment:
      SPARK_CONF_DIR: /conf
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 2g
      SPARK_WORKER_PORT: 8881
      SPARK_WORKER_WEBUI_PORT: 8086
      SPARK_PUBLIC_DNS: localhost
    links:
      - master
    expose:
      - 7012
      - 7013
      - 7014
      - 7015
      - 8881
      - 8086
    ports:
      - 8086:8086
    volumes:
      - ./conf/worker6:/conf
      - ./data/worker6:/tmp/data
```

- 接下來我們執行一個例項來驗證；

### 驗證

- 在docker-compose.yml所在目錄建立hadoop.env檔案，內容如下：

```shell
CORE_CONF_fs_defaultFS=hdfs://namenode:8020
CORE_CONF_hadoop_http_staticuser_user=root
CORE_CONF_hadoop_proxyuser_hue_hosts=*
CORE_CONF_hadoop_proxyuser_hue_groups=*

HDFS_CONF_dfs_webhdfs_enabled=true
HDFS_CONF_dfs_permissions_enabled=false

YARN_CONF_yarn_log___aggregation___enable=true
YARN_CONF_yarn_resourcemanager_recovery_enabled=true
YARN_CONF_yarn_resourcemanager_store_class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
YARN_CONF_yarn_resourcemanager_fs_state___store_uri=/rmstate
YARN_CONF_yarn_nodemanager_remote___app___log___dir=/app-logs
YARN_CONF_yarn_log_server_url=http://historyserver:8188/applicationhistory/logs/
YARN_CONF_yarn_timeline___service_enabled=true
YARN_CONF_yarn_timeline___service_generic___application___history_enabled=true
YARN_CONF_yarn_resourcemanager_system___metrics___publisher_enabled=true
YARN_CONF_yarn_resourcemanager_hostname=resourcemanager
YARN_CONF_yarn_timeline___service_hostname=historyserver
YARN_CONF_yarn_resourcemanager_address=resourcemanager:8032
YARN_CONF_yarn_resourcemanager_scheduler_address=resourcemanager:8030
YARN_CONF_yarn_resourcemanager_resource___tracker_address=resourcemanager:8031
```

- 修改好docker-composes.yml後，執行以下命令啟動容器：

```shell
docker-compose up -d
```

- 此次驗證所用的spark應用的功能是分析維基百科的網站統計資訊，找出訪問量最大的網頁，本次實戰用現成的jar包，不涉及編碼，該應用的原始碼和開發詳情請參照[《spark實戰之：分析維基百科網站統計資料(java版)》](https://blog.51cto.com/u_13674465/5591775)；

- 從github下載已經構建好的spark應用jar檔案：

```shell
wget https://raw.githubusercontent.com/zq2599/blog_demos/master/files/sparkdemo-1.0-SNAPSHOT.jar
```

- 從github下載維基百科的網站統計資訊大資料集，這裡只下載了一個檔案，建議您參照[《尋找海量資料集用於大資料開發實戰(維基百科網站統計資料)》](https://blog.51cto.com/u_13674465/5587921)下載更多檔案用來實戰：

```shell
wget https://raw.githubusercontent.com/zq2599/blog_demos/master/files/pagecounts-20160801-000000
```

- 將下載的sparkdemo-1.0-SNAPSHOT.jar檔案放在docker-compose.xml所在目錄的jars目錄下；
- 在docker-compose.xml所在目錄的input_files目錄內建立input目錄，再將下載的pagecounts-20160801-000000檔案放在這個input目錄下；
- 執行以下命令，將整個input目錄放入hdfs：

```shell
docker exec namenode hdfs dfs -put /input_files/input /
```

- 執行以下命令，提交一個任務，使用了12個executor，每個1G記憶體：

```shell
docker exec -it master spark-submit \
--class com.bolingcavalry.sparkdemo.app.WikiRank \
--executor-memory 1g \
--total-executor-cores 12 \
/root/jars/sparkdemo-1.0-SNAPSHOT.jar \
namenode \
8020
```

- 宿主機的狀態如下所示，CPU和記憶體都被榨乾：
![在這裡插入圖片描述](https://img-blog.csdnimg.cn/20190216161742366.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70)

- 宿主機的IP地址是192.168.1.102，以下是狀態資訊，地址：http://192.168.1.102:8080/
![在這裡插入圖片描述](https://img-blog.csdnimg.cn/20190216161936490.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70)

- 檢視job的Stage情況，如下圖，這些資訊對學習和掌握spark至關重要，地址：http://192.168.1.102:4040
![在這裡插入圖片描述](https://img-blog.csdnimg.cn/20190216162315332.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70)

- 檢視worker1的基本情況，如下圖，地址是：http://192.168.1.102:8081
![在這裡插入圖片描述](https://img-blog.csdnimg.cn/20190216162801657.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70)

- 如果想檢視worker1上的業務日誌，請點選下圖紅框中的連結，但此時會提示頁面訪問失敗，對應的url是"http://localhost:8081/logPage?appId=app-20190216081637-0002&executorId=5&logType=stdout"，這個地址是頁面生成的，我們只要把其中的"localhost"改成宿主機的IP地址就好了：
![在這裡插入圖片描述](https://img-blog.csdnimg.cn/20190216163002753.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70)

- 修改後的連結可以訪問，看到的業務日誌如下圖，紅框中就是業務程式碼輸出的日誌：
![在這裡插入圖片描述](https://img-blog.csdnimg.cn/20190216163351925.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70)

- 以上就是最佳化和驗證的全部過程，您可以根據自己機器的實際情況來調整引數，將電腦的效能充分的利用起來；

- 後來我用24個300M的檔案做資料集，大約1.5億條記錄，在上述硬體環境執行上述命令，最終耗時30分鐘完成，如下圖：
![在這裡插入圖片描述](https://img-blog.csdnimg.cn/20190216171125764.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70)

### 歡迎關注ITpub：程式設計師欣宸
> [學習路上，你不孤單，欣宸原創一路相伴...](http://blog.itpub.net/70017844/)

docker下的spark叢集，調整引數榨乾硬體

相關文章