Flume安裝及簡單部署

環境：

1.flume-ng-1.6.0-cdh5.7.0.tar.gz

2.jdk 1.8

1.下載安裝包並解壓
wget

tar -xvf flume-ng-1.6.0-cdh5.7.0.tar.gz

rm flume-ng-1.6.0-cdh5.7.0.tar.gz

mv apache-flume-1.6.0-cdh5.7.0-bin    flume

2.配置環境變數
vi /etc/profile

export  FLUME_HOME=/opt/software/flume
exportPATH=.:$FLUME_HOME/bin:$KAFKA_HOME/bin::$ZOOKEEPER_HOME/bin:$PATH

source /etc/profile

3. 配置flume-env.sh檔案

cd  /opt/software/flume/conf/

cp flume-env.sh.template flume-env.sh
vi  flume-env.sh

export HADOOP_HOME=/opt/software/hadoop
export JAVA_HOME=/usr/java/jdk1.8.0_45

4.版本驗證
flume-ng version
5.Flume部署示例
cd /opt/software/flume/bin/

vi netcat.conf

# Name the components on this agent

a1.sources = r1

a1.sinks = k1

a1.channels = c1

# Describe/configure the source

a1.sources.r1.type = netcat

a1.sources.r1.bind = localhost

a1.sources.r1.port = 44444

# Describe the sink

a1.sinks.k1.type = logger

# Use a channel that buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

6.執行FlumeAgent，監聽本機的44444埠

 flume-ng agent -c conf -f  netcat.conf -n a1 -Dflume.root.logger=INFO,console

[root@hadoop001 bin]# flume-ng agent -c conf -f  netcat.conf -n a1 -Dflume.root.logger=INFO,console
Info: Including Hadoop libraries found via (/opt/software/hadoop-2.6.0-cdh5.7.0/bin/hadoop) for HDFS access
Info: Excluding /opt/software/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar from classpath
Info: Excluding /opt/software/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar from classpath
Info: Including Hive libraries found via (/opt/software/hive-1.1.0-cdh5.7.0) for Hive access

.......

18/03/25 17:52:39 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: c1 started
18/03/25 17:52:39 INFO node.Application: Starting Sink k1
18/03/25 17:52:39 INFO node.Application: Starting Source r1
18/03/25 17:52:39 INFO source.NetcatSource: Source starting
18/03/25 17:52:40 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]

7.開啟另外一個client     

telnet localhost 44444

[root@hadoop001 ~]# telnet localhost 44444
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
111

OK18/03/25 18:11:15 WARN source.NetcatSource: Client sent event exceeding the maximum length
18/03/25 18:12:50 INFO sink.LoggerSink: Event: { headers:{} body: 31 31 31 0D                                     111. }

8.完畢