logstash5.x安裝及簡單運用

wadeson發表於2017-09-20
H5

Logstash requires Java 8. Java 9 is not supported.

1、檢測是否安裝了java環境

[root@node3 ~]# java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

2、安裝logstash，這裡採用rpm安裝

　　https://artifacts.elastic.co/downloads/logstash/logstash-5.6.1.rpm

　　yum install logstash

檢視生成了哪些檔案，檢視logstash的執行檔案位置：

/etc/logstash/conf.d
/etc/logstash/jvm.options
/etc/logstash/log4j2.properties
/etc/logstash/logstash.yml
/etc/logstash/startup.options
/usr/share/logstash/CHANGELOG.md
/usr/share/logstash/CONTRIBUTORS
/usr/share/logstash/Gemfile
/usr/share/logstash/Gemfile.jruby-1.9.lock
/usr/share/logstash/LICENSE
/usr/share/logstash/NOTICE.TXT
/usr/share/logstash/bin/cpdump
/usr/share/logstash/bin/ingest-convert.sh
/usr/share/logstash/bin/logstash
/usr/share/logstash/bin/logstash-plugin
/usr/share/logstash/bin/logstash-plugin.bat
/usr/share/logstash/bin/logstash.bat
/usr/share/logstash/bin/logstash.lib.sh
/usr/share/logstash/bin/ruby
/usr/share/logstash/bin/setup.bat
/usr/share/logstash/bin/system-install
/usr/share/logstash/data

　配置檔案：

1、配置jvm

/etc/logstash/jvm.options
2、logstash的一些配置
/etc/logstash/logstash.yml
3、環境變數一些的配置
/etc/logstash/startup.options
4、日誌與log4j2的配置
/etc/logstash/log4j2.properties
 
開始第一個任務：
[root@node3 conf.d]# /usr/share/logstash/bin/logstash -e 'input { stdin {} } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

　提示warning，解決辦法：

mkdir -p /usr/share/logstash/config/
ln -s /etc/logstash/* /usr/share/logstash/config
chown -R logstash:logstash /usr/share/logstash/config/
bin/logstash -e 'input { stdin { } } output { stdout {} }'

　如果logstash不適用命令列執行，而是作為一個服務：

　　logstash啟動：
　　/etc/init.d/logstash start
　　systemctl start logstash.service
 
開始編寫配置檔案進行logstash解析：
1、input外掛中file外掛的使用
[root@node3 conf.d]# cat file.conf 
input {
    file {
        path => ["/var/log/messages"]
        start_position => "beginning"
    }
}

output {
    stdout {
        codec => rubydebug
    }
}
[root@node3 conf.d]# /usr/share/logstash/bin/logstash -f file.conf

　2、多個log日誌的輸入、

[root@node3 conf.d]# cat file_more_choose.conf 
input {
    file {
        path => ["/var/log/messages"]
        start_position => "beginning"
    }
    file {
        path => ["/var/log/elasticsearch/my-elastic.log"]
        start_position => "beginning"
    }
}

output {
    stdout {
        codec => rubydebug
    }
}
[root@node3 conf.d]# /usr/share/logstash/bin/logstash -f file_more_choose.conf

　但是發現只列印出elastic的日誌，message的日誌沒有stdout，收集的日誌是增量的，之前收集的日誌已經存在sincedb中了，所以會預設從之後開始存

Path of the sincedb database file (keeps track of the current position of monitored log files) that will be written to disk. The default will write sincedb files to <path.data>/plugins/inputs/file NOTE: it must be a file path and not a directory path，這是一段sincedb_path的解釋

 

檢查配置檔案的語法是否正確：
-t, --config.test_and_exit    Check configuration for valid syntax and then exit.
                                   (default: false)
-r, --config.reload.automatic Monitor configuration changes and reload
                                  whenever it is changed.
                                  NOTE: use SIGHUP to manually reload the config
                                   (default: false)
[root@node3 conf.d]# /usr/share/logstash/bin/logstash -f file.conf -t
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK

　3、以elasticsearch外掛輸出：

input {
    file {
        path => ["/var/log/logstash/logstash-plain.log"]
        start_position => "beginning"
        type => "logstash"
    }
}


output {
    elasticsearch {
        hosts => ["192.168.44.134:9200"]
        index => "logstash-log"
        codec => rubydebug
    }
}

  

4、根據外掛type來定義輸出外掛：

[root@node3 conf.d]# cat type.conf 
input {
    file {
       path  => ["/var/log/logstash/logstash-plain.log"]
       start_position => "beginning"
       type => "logstash_2"
    }
    file {
       path => ["/var/log/messages"]
       start_position => "beginning"
       type => "system"
    }
}


output {
    if [type] == "logstash_2" {
        elasticsearch {
            hosts => ["192.168.44.134:9200"]
            index => "logstash_2"
            codec => rubydebug
        }
    }
    if [type] == "system" {
         stdout {
            codec => rubydebug
         }
    }
}

　現在向messages日誌中echo一段話：

echo "`date +%F`" >> /var/log/messages

　然後開始執行：

[root@node3 conf.d]# /usr/share/logstash/bin/logstash -f type.conf 
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
{
      "@version" => "1",
          "host" => "node3",
          "path" => "/var/log/messages",
    "@timestamp" => 2017-09-20T08:19:05.782Z,
       "message" => "2017-09-20",                這是剛剛echo新增的內容
          "type" => "system"
}

　檢視es中的索引是否有生成：

 

相關文章