logstash簡介及基本操作

logstash常用於日誌系統中做日誌採集裝置，最常用於ELK中作為日誌收集器使用

logstash的基本流程架構：input  |  filter  |  output 如需對資料進行額外處理，filter可省略。

3.1 Input(輸入）：採集各種樣式，大小和相關來源資料，從各個伺服器中收集資料。

用於在將event透過output發出之前對其實現某些處理功能。grok。

grok：用於分析結構化文字資料。目前 是logstash中將非結構化資料日誌資料轉化為結構化的可查詢資料的不二之選

3.3 Output(輸出）：將我們過濾出的資料儲存到那些資料庫和相關儲存中， 。

inpust：必須，負責產生事件（Inputs generate events），常用：File、syslog、redis、beats（如：Filebeats）

filters：可選，負責資料處理與轉換（filters modify them），常用：grok、mutate、drop、clone、geoip

outpus：必須，負責資料輸出（outputs ship them elsewhere），常用：elasticsearch、file、graphite、statsd

input {     從哪個地方讀取，輸入資料。

   

}

filter {    依據grok模式對資料進行分析結構化

   

}

output {    將分析好的資料輸出儲存到哪些地方

  

}

2.1我們自定義gork模式對日誌進行過濾。

語法格式：

       %{SYNTAX:SEMANTIC}

               SYNTAX：預定義模式名稱；

               SEMANTIC：匹配到的文字的自定義識別符號；

[root@node1 conf.d]# vim groksimple.conf

input {

    stdin {}

}

filter {

    grok {

    match => { "message" => "%{IP:clientip} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }

 }

}

output {

  stdout {

  codec => rubydebug

  }

}

[root@node1 conf.d]# logstash -f /etc/logstash/conf.d/groksimple.conf

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults

Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

[INFO ] 2020-10-13 14:29:41.936 [main] runner - Starting Logstash {"logstash.version"=>"7.9.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 25.262-b10 on 1.8.0_262-b10 +indy +jit [linux-x86_64]"}

[WARN ] 2020-10-13 14:29:42.412 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified

[INFO ] 2020-10-13 14:29:44.025 [Converge PipelineAction::Create<main>] Reflections - Reflections took 42 ms to scan 1 urls, producing 22 keys and 45 values 

[INFO ] 2020-10-13 14:29:44.995 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/groksimple.conf"], :thread=>"#<Thread:0x4ca2f74b run>"}

[INFO ] 2020-10-13 14:29:45.749 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.74}

[INFO ] 2020-10-13 14:29:45.820 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}

The stdin plugin is now waiting for input:

[INFO ] 2020-10-13 14:29:45.902 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

[INFO ] 2020-10-13 14:29:46.098 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}

1.1.1.1 get /index.html 30 0.23  我們標準輸入一些日誌資訊。

{

    "@timestamp" => 2020-10-13T06:30:11.973Z,

          "host" => "node1",

      "@version" => "1",

       "request" => "/index.html",

       "message" => "1.1.1.1 get /index.html 30 0.23",

      "duration" => "0.23",

      "clientip" => "1.1.1.1",

        "method" => "get",

         "bytes" => "30"

}

推薦閱讀：

https://blog.csdn.net/yurun_house/article/details/109025588