hadoop hdfs 命令使用

lcz393537415發表於2016-06-12
Hadoop

參考連結：

在HDFS中，提供了fsck命令，用於檢查HDFS上檔案和目錄的健康狀態、獲取檔案的block資訊和位置資訊等。

fsck命令必須由HDFS超級使用者來執行，普通使用者無許可權。


  1. [hadoop@dev~]$ hdfs fsck
  2. Usage:DFSck[-list-corruptfileblocks|[-move|-delete|-openforwrite][-files[-blocks[-locations|-racks]]]]
  3. start checkingfromthispath
  4. -move move corrupted files to/lost+found
  5. -deletedeletecorrupted files
  6. -filesprintoutfiles beingchecked
  7. -openforwriteprintoutfiles openedforwrite
  8. -includeSnapshots include snapshot dataifthe given path indicates a snapshottable directoryorthere are snapshottable directories under it
  9. -list-corruptfileblocksprintoutlist of missing blocksandfiles they belong to
  10. -blocksprintoutblock report
  11. -locationsprintoutlocationsforevery block
  12. -racksprintoutnetwork topologyfordata-node locations


下面介紹每一個選項的含義及用法。

檢視檔案中損壞的塊（-list-corruptfileblocks）


  1. [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/-list-corruptfileblocks
  2. Thefilesystem under path'/hivedata/warehouse/liuxiaowen.db/lxw_product_names/'has0CORRUPT files


將損壞的檔案移動至/lost+found目錄（-move）


  1. [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168-move
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168atThuAug1309:36:35CST2015
  3. .Status:HEALTHY
  4. Totalsize:13497058B
  5. Totaldirs:0
  6. Totalfiles:1
  7. Totalsymlinks:0
  8. Totalblocks(validated):1(avg.block size13497058B)
  9. Minimallyreplicated blocks:1(100.0%)
  10. Over-replicated blocks:0(0.0%)
  11. Under-replicated blocks:0(0.0%)
  12. Mis-replicated blocks:0(0.0%)
  13. Defaultreplication factor:2
  14. Averageblock replication:2.0
  15. Corruptblocks:0
  16. Missingreplicas:0(0.0%)
  17. Numberof data-nodes:15
  18. Numberof racks:1
  19. FSCK ended atThuAug1309:36:35CST2015in1milliseconds
  20.  
  21.  
  22. Thefilesystem under path'/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168'isHEALTHY


刪除損壞的檔案（-delete）


  1. [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168-delete
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168atThuAug1309:37:58CST2015
  3. .Status:HEALTHY
  4. Totalsize:13497058B
  5. Totaldirs:0
  6. Totalfiles:1
  7. Totalsymlinks:0
  8. Totalblocks(validated):1(avg.block size13497058B)
  9. Minimallyreplicated blocks:1(100.0%)
  10. Over-replicated blocks:0(0.0%)
  11. Under-replicated blocks:0(0.0%)
  12. Mis-replicated blocks:0(0.0%)
  13. Defaultreplication factor:2
  14. Averageblock replication:2.0
  15. Corruptblocks:0
  16. Missingreplicas:0(0.0%)
  17. Numberof data-nodes:15
  18. Numberof racks:1
  19. FSCK ended atThuAug1309:37:58CST2015in1milliseconds
  20.  
  21.  
  22. Thefilesystem under path'/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168'isHEALTHY


檢查並列出所有檔案狀態（-files）


  1. [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/-files
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/hivedata/warehouse/liuxiaowen.db/lxw_product_names/atThuAug1309:39:38CST2015
  3. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/dir
  4. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/_SUCCESS0bytes,0block(s):OK
  5. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000013583807bytes,1block(s):OK
  6. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000113577427bytes,1block(s):OK
  7. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000213588601bytes,1block(s):OK
  8. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000313479213bytes,1block(s):OK
  9. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000413497012bytes,1block(s):OK
  10. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000513557451bytes,1block(s):OK
  11. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000613580267bytes,1block(s):OK
  12. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000713486035bytes,1block(s):OK
  13. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000813481498bytes,1block(s):OK
  14. ...


檢查並列印正在被開啟執行寫操作的檔案（-openforwrite）


  1. [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/-openforwrite
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/hivedata/warehouse/liuxiaowen.db/lxw_product_names/atThuAug1309:41:28CST2015
  3. ....................................................................................................
  4. ....................................................................................................
  5. .Status:HEALTHY
  6. Totalsize:2704782548B
  7. Totaldirs:1
  8. Totalfiles:201
  9. Totalsymlinks:0
  10. Totalblocks(validated):200(avg.block size13523912B)
  11. Minimallyreplicated blocks:200(100.0%)
  12. Over-replicated blocks:0(0.0%)
  13. Under-replicated blocks:0(0.0%)
  14. Mis-replicated blocks:0(0.0%)
  15. Defaultreplication factor:2
  16. Averageblock replication:2.0
  17. Corruptblocks:0
  18. Missingreplicas:0(0.0%)
  19. Numberof data-nodes:15
  20. Numberof racks:1
  21. FSCK ended atThuAug1309:41:28CST2015in10milliseconds
  22.  
  23. Thefilesystem under path'/hivedata/warehouse/liuxiaowen.db/lxw_product_names/'isHEALTHY


列印檔案的Block報告（-blocks）

需要和-files一起使用。


  1. [hadoop@dev~]$ hdfs fsck/logs/site/2015-08-08/lxw1234.log-files-blocks
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/logs/site/2015-08-08/lxw1234.log atThuAug1309:45:59CST2015
  3. /logs/site/2015-08-08/lxw1234.log7408754725bytes,56block(s):OK
  4. 0.BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728repl=2
  5. 1.BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728repl=2
  6. 2.BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728repl=2
  7. 3.BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728repl=2
  8. 4.BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728repl=2
  9. 5.BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728repl=2
  10. 6.BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728repl=2
  11. 7.BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728repl=2
  12. 8.BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728repl=2
  13. 9.BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728repl=2
  14. 10.BP-1034052771-172.16.212.130-1405595752491:blk_1075893003_2152402 len=134217728repl=2
  15. 11.BP-1034052771-172.16.212.130-1405595752491:blk_1075893004_2152403 len=134217728repl=2
  16. 12.BP-1034052771-172.16.212.130-1405595752491:blk_1075893005_2152404 len=134217728repl=2
  17. 13.BP-1034052771-172.16.212.130-1405595752491:blk_1075893006_2152405 len=134217728repl=2
  18. 14.BP-1034052771-172.16.212.130-1405595752491:blk_1075893007_2152406 len=134217728repl=2
  19. ...


其中，/logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): 表示檔案的總大小和block數；

0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2
1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2
2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2

前面的0. 1. 2.代表該檔案的block索引，56的檔案塊，就從0-55;

BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381表示block id；

len=134217728 表示該檔案塊大小；

repl=2 表示該檔案塊副本數；

列印檔案塊的位置資訊（-locations）

需要和-files -blocks一起使用。


  1. [hadoop@dev~]$ hdfs fsck/logs/site/2015-08-08/lxw1234.log-files-blocks-locations
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/logs/site/2015-08-08/lxw1234.log atThuAug1309:45:59CST2015
  3. /logs/site/2015-08-08/lxw1234.log7408754725bytes,56block(s):OK
  4. 0.BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728repl=2[172.16.212.139:50010,172.16.212.135:50010]
  5. 1.BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728repl=2[172.16.212.140:50010,172.16.212.133:50010]
  6. 2.BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728repl=2[172.16.212.136:50010,172.16.212.141:50010]
  7. 3.BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728repl=2[172.16.212.133:50010,172.16.212.135:50010]
  8. 4.BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728repl=2[172.16.212.142:50010,172.16.212.139:50010]
  9. 5.BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728repl=2[172.16.212.133:50010,172.16.212.139:50010]
  10. 6.BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728repl=2[172.16.212.141:50010,172.16.212.135:50010]
  11. 7.BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728repl=2[172.16.212.144:50010,172.16.212.142:50010]
  12. 8.BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728repl=2[172.16.212.133:50010,172.16.212.138:50010]
  13. 9.BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728repl=2[172.16.212.140:50010,172.16.212.134:50010]
  14. ...


和列印出的檔案塊資訊相比，多了一個檔案塊的位置資訊：[172.16.212.139:50010, 172.16.212.135:50010]

列印檔案塊位置所在的機架資訊（-racks）


  1. [hadoop@dev~]$ hdfs fsck/logs/site/2015-08-08/lxw1234.log-files-blocks-locations-racks
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/logs/site/2015-08-08/lxw1234.log atThuAug1309:45:59CST2015
  3. /logs/site/2015-08-08/lxw1234.log7408754725bytes,56block(s):OK
  4. 0.BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728repl=2[/default-rack/172.16.212.139:50010,/default-rack/172.16.212.135:50010]
  5. 1.BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728repl=2[/default-rack/172.16.212.140:50010,/default-rack/172.16.212.133:50010]
  6. 2.BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728repl=2[/default-rack/172.16.212.136:50010,/default-rack/172.16.212.141:50010]
  7. 3.BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728repl=2[/default-rack/172.16.212.133:50010,/default-rack/172.16.212.135:50010]
  8. 4.BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728repl=2[/default-rack/172.16.212.142:50010,/default-rack/172.16.212.139:50010]
  9. 5.BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728repl=2[/default-rack/172.16.212.133:50010,/default-rack/172.16.212.139:50010]
  10. ...


和前面列印出的資訊相比，多了機架資訊：[/default-rack/172.16.212.139:50010, /default-rack/172.16.212.135:50010]

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/31347383/viewspace-2119093/，如需轉載，請註明出處，否則將追究法律責任。

相關文章