Flume監聽Nginx日誌流向HDFS安裝配置

破棉襖發表於2014-07-16
Nginx

wget 

tar zxvf apache-flume-1.5.0-bin.tar.gz
 
mv apache-flume-1.5.0-bin flume
cd conf
vim flume.conf  
內容如下：
  

點選(此處)摺疊或開啟

  1. agent1.sources=source1
    agent1.sinks=sink1
    agent1.channels=channel1




    agent1.sources.source1.type=exec
    agent1.sources.source1.command=tail -f /usr/local/nginx/logs/access.log
    agent1.sources.source1.channels=channel1
    agent1.sources.source1.fileHeader=true


    agent1.sources.source1.interceptors=i
    agent1.sources.source1.interceptors.i.regex= .*\.(css|js|jpg|jpeg|png|gif|ico|swf|flv).*
    agent1.sources.source1.interceptors.i.type= org.apache.flume.interceptor.RegexFilteringInterceptor$Builder
    agent1.sources.source1.interceptors.i.excludeEvents=true


    agent1.sinks.sink1.type=hdfs
    agent1.sinks.sink1.hdfs.path=hdfs://192.168.0.237:9000/flumelog/%Y/%m/%d
    agent1.sinks.sink1.hdfs.filePrefix=nginx-%Y-%m-%d-%H
    agent1.sinks.sink1.hdfs.fileType=DataStream
    agent1.sinks.sink1.hdfs.useLocalTimeStamp=true
    agent1.sinks.sink1.hdfs.writeFormat=TEXT


    agent1.sinks.sink1.hdfs.batchSize=1000


    agent1.sinks.sink1.hdfs.rollSize=0
    agent1.sinks.sink1.hdfs.rollCount=0
    agent1.sinks.sink1.hdfs.rollInterval=0
    agent1.sinks.sink1.hdfs.threadsPoolSize=4
    agent1.sinks.sink1.hdfs.callTimeout=30000
    agent1.sinks.sink1.channel=channel1




    agent1.channels.channel1.type=file
    agent1.channels.channel1.checkpointDir=/usr/flume_temp/flume_tmp123
    agent1.channels.channel1.dataDirs=/usr/flume_temp/flume_tmp

 建立目錄
 mkdir -p /usr/flume_temp/flume_tmp123
 mkdir -p /usr/flume_temp/flume_tmp
 
 mv flume-env.sh.template flume-env.sh
 vi flume-env.sh
 修改JAVA_HOME後儲存退出
 
 修改環境變數 vi /etc/profile
    1.配置好JAVA_HOME
    2.新增export PATH=$PATH:/usr/flume/flume/bin
 source /etc/profile

進入到flume根目錄：
 flume-ng agent -c conf -f conf/flume.conf --name agent1 -Dflume.root.logger=INFO,console
(必然會報類找不到的錯誤，將相應hadoop的jar包複製到/lib目錄下再啟動即可。)

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/29754888/viewspace-1220572/，如需轉載，請註明出處，否則將追究法律責任。

相關文章