定時將資料匯入到hive的shell指令碼

破棉襖發表於2015-05-12
Hive指令碼

每日定時匯入hive資料倉儲的自動化指令碼

建立shell指令碼，建立臨時表，裝載資料，轉換到正式的分割槽表中：


  1. #!/bin/sh
  2. # upload logs to hdfs
  4. yesterday=`date --date=\'1 days ago\' +%Y%m%d`
  6. hive -e \"
  7. use stage;
  8. create table tracklog_tmp (
  9. dateday string,
  10. datetime string,
  11. ip string ,
  12. cookieid string,
  13. userid string,
  14. logserverip string,
  15. referer string,
  16. requesturl string,
  17. remark1 string,
  18. remark2 string,
  19. alexaflag string,
  20. ua string,
  21. wirelessflag string
  22. )
  23. ROW FORMAT DELIMITED FIELDS TERMINATED BY \' \';\"
  26. hive -e \"
  27. use stage;
  28. set hive.enforce.bucketing=true;
  29. set hive.exec.compress.output=true;
  30. set mapred.output.compress=true;
  31. set mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
  32. set io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec;
  33. load data local inpath \'/diskg/logs/tracklog_192.168.1.1/${yesterday}/${yesterday}????.dat\' overwrite into table tracklog_tmp;
  34. insert into table tracklog PARTITION (day=\'${yesterday}\') select * from tracklog_tmp;
  35. load data local inpath \'/diskg/logs/tracklog_192.168.1.2/${yesterday}/${yesterday}????.dat\' overwrite into table tracklog_tmp;
  36. insert into table tracklog PARTITION (day=\'${yesterday}\') select * from tracklog_tmp;
  37. load data local inpath \'/diskg/logs/tracklog_192.168.1.3/${yesterday}/${yesterday}????.dat\' overwrite into table tracklog_tmp;
  38. insert into table tracklog PARTITION (day=\'${yesterday}\') select * from tracklog_tmp;
  39. load data local inpath \'/diskg/logs/trackloguc_192.168.1.1/${yesterday}/${yesterday}????.dat\' overwrite into table tracklog_tmp;
  40. insert into table tracklog PARTITION (day=\'${yesterday}\') select * from tracklog_tmp;
  41. load data local inpath \'/diskg/logs/trackloguc_192.168.1.2/${yesterday}/${yesterday}????.dat\' overwrite into table tracklog_tmp;
  42. insert into table tracklog PARTITION (day=\'${yesterday}\') select * from tracklog_tmp;
  43. load data local inpath \'/diskg/logs/trackloguc_192.168.1.3/${yesterday}/${yesterday}????.dat\' overwrite into table tracklog_tmp;
  44. insert into table tracklog PARTITION (day=\'${yesterday}\') select * from tracklog_tmp;
  45. \"
  47. hive -e \"
  48. use stage;
  49. drop table tracklog_tmp ;\"



在crontab中加入定時任務：


crontab -e 
加入如下程式碼：

#import tracklog
25  07 * * * /opt/bin/hive_opt/import_tracklog.sh


原文：


來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/29754888/viewspace-1649771/，如需轉載，請註明出處，否則將追究法律責任。

相關文章