檢視spark程式執行狀態以及安裝spark

6、移動命令

#hadoop dfs –mv /user/test.txt  /user/test/


7、拷貝命令

#hadoop dfs –copytolocal /user/test.txt   /opt/

檢視spark上面的執行情況；
http://192.168.31.10:8080/# 

wc -l judgedoc_litigant.txt 檢視檔案有多少行；

htop檢視當前spark佔用的cpu,退出按q


命令：
vim ~/.bashrc

source ~/.bashrc

ps aux | grep spark

pkill -f "spark"


sudo chown -R sc:sc  spark-2.3.1-bin-hadoop2.7/

sudo mv /home/sc/Downloads/spark-2.3.1-bin-hadoop2.7 /opt/


$SPARK_HOME 檢視spark的路徑
network_all_link_cnt.csv
network_all_link_judgedoc_cnt.csv  *
network_share_com_inv_com_judgedoc_cnt.csv 
network_c_inv_c_revoke_cnt.csv
network_c_inv_c_shixin_cnt.csv
network_c_inv_c_zhixing_cnt.csv
network_share_established_years_mean.csv
network_share_judgedoc_cnt.csv  *
network_share_registered_capital_mean.csv
network_share_revoke_cnt.csv
network_share_shixin_cnt.csv
network_share_zhixing_cnt.csv

/home/sc/PycharmProjects/risk-model/scpirt/test_build_data_scripts/shixin_cox/shixin_cox_get_features/shixin_cox_biz/sub/shixin_the_last_evevt_time_merge_shixin_time_and_shixin_cnt.py

http://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.7.tgz

drwxr-xr-x   - scdata supergroup          0 2018-08-08 14:16 /hdfs/riskModelAuto/2018-07-24/feature/feature_extract_bidding.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-08 14:15 /hdfs/riskModelAuto/2018-07-24/feature/feature_extract_courtnotice.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-09 16:14 /hdfs/riskModelAuto/2018-07-24/feature/feature_extract_judgedoc.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-08 14:16 /hdfs/riskModelAuto/2018-07-24/feature/feature_extract_shixin.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-08 14:15 /hdfs/riskModelAuto/2018-07-24/feature/feature_extract_trademark.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-09 10:39 /hdfs/riskModelAuto/2018-07-24/feature/feature_extract_sue_by_rank_judgedoc.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-06 23:04 /hdfs/riskModelAuto/2018-07-24/feature/network_all_link_cnt.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-09 11:14 /hdfs/riskModelAuto/2018-07-24/feature/network_c_inv_c_revoke_cnt.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-09 11:29 /hdfs/riskModelAuto/2018-07-24/feature/network_c_inv_c_shixin_cnt.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-09 11:25 /hdfs/riskModelAuto/2018-07-24/feature/network_c_inv_c_zhixing_cnt.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-09 15:24 /hdfs/riskModelAuto/2018-07-24/feature/network_share_com_inv_com_judgedoc_cnt.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-09 14:08 /hdfs/riskModelAuto/2018-07-24/feature/network_share_registered_capital_mean.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-09 14:31 /hdfs/riskModelAuto/2018-07-24/feature/network_share_revoke_cnt.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-08 19:07 /hdfs/riskModelAuto/2018-07-24/feature/network_share_shixin_cnt.csv
drwxr-xr-x   - scdata supergroup          0 2018-08-07 12:32 /hdfs/riskModelAuto/2018-07-24/feature/network_share_zhixing_cnt.csv