Elasticsearch、Logstash、Kibana、Filebeat的使用總結

ELK是什麼？
ELK Stack是軟體集合Elasticsearch、Logstash、Kibana的簡稱，由這三個軟體及其相關的元件可以打造大規模日誌實時處理系統。

ElasticSearch：是一個基於Lucene的搜尋伺服器。它提供了一個分散式多使用者能力的全文搜尋引擎，基於RESTful web介面。Elasticsearch是用Java開發的，並作為Apache許可條款下的開放原始碼釋出，是當前流行的企業級搜尋引擎。設計用於雲端計算中，能夠達到實時搜尋，穩定，可靠，快速，安裝使用方便。

Logstash：是一款強大的資料處理工具，它可以實現資料傳輸，格式處理，格式化輸出，還有強大的外掛功能，常用於日誌處理。

Kibana：是一個視覺化工具，主要負責查詢 Elasticsearch 的資料並以視覺化的方式展現給業務方，比如各類餅圖、直方圖、區域圖等。

關於Elasticsearch的版本，我是從5.x版本開始學起的，聽說之前的版本跳躍2.x--->5.x,貌似變化還挺大的。關於head外掛在5.x之後變成獨立服務了。

關於ES叢集的安裝配置以及head外掛的安裝參考：http://blog.csdn.net/chenxun_2010/article/details/78437852

logstash最佳實踐：https://doc.yonyoucloud.com/doc/logstash-best-practice-cn/get_start/hello_world.html

Logstash使用：
https://www.elastic.co/guide/en/logstash/current/pipeline.html

Logstash 工作的三個階段： input、filter、output

安裝logstash： 其安裝非常簡單，只需要下載安裝包解壓開箱即用。 只要寫配置檔案即可。
啟動方式：

./bin/logstash   -f    your_config.file

配置檔案的寫法格式：參考http://blog.csdn.net/chenxun_2010/article/details/78605934

input {
    file {

        path => ["/home/elk_test/logstash-5.6.3/logfile"]
        codec => json {
            charset => "UTF-8"
        }
    }
}

output {

    stdout {
        codec => rubydebug
    }

    elasticsearch {
        hosts => "192.168.0.153:9200"

    }
}

Kibana的安裝也非常簡單，下載安裝包解壓修改一下配置檔案就可以：

server.host: "192.168.2.181"

#elasticsearch.username: "elastic"
#elasticsearch.password: "changeme"

elasticsearch.url: "http://192.168.2.181:9200"

kibana.index: ".kibana"

Filebeat：的使用
filebeat：部署在具體的業務機器上，通過定時監控的方式獲取增量的日誌，並轉發到logstash、elasticsearch、kafka等等。

配置vim filebeat.yml

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /home/elk/logfile
    #- c:\programdata\elasticsearch\logs\*

輸出到logstash：

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.2.181:5044"]

logstash的配置

input {
    beats {
        host => "192.168.2.181"
        port => 5044 
    	codec => json
	}
}

output {

    stdout {
        codec => rubydebug
    }

    elasticsearch {
        hosts => "192.168.2.181:9200"
    }
}

啟動filebeat：

./filebeat

啟動logstash：

./bin/logstash   -f   beat.config

把準備好的json資料輸入到filebeat監控的檔案中去：

{ "firstName": "1", "lastName":"McLaughlin", "email": "aaaa" }
{ "firstName": "2", "lastName":"Hunter", "email": "bbbb"}
{ "firstName": "3", "lastName":"Harold", "email": "cccc" }

在logstash端我們看到輸出：成功解析json欄位

{
     "firstName" => "1",
      "lastName" => "McLaughlin",
    "@timestamp" => 2017-12-01T08:38:37.480Z,
        "offset" => 63,
      "@version" => "1",
          "beat" => {
            "name" => "Ubuntu-20170424",
        "hostname" => "Ubuntu-20170424",
         "version" => "5.6.3"
    },
    "input_type" => "log",
          "host" => "Ubuntu-20170424",
        "source" => "/home/elk/filebeat-5.6.3-linux-x86_64/request",
          "type" => "log",
         "email" => "aaaa",
          "tags" => [
        [0] "beats_input_codec_json_applied"
    ]
}

{
     "firstName" => "2",
      "lastName" => "Hunter",
    "@timestamp" => 2017-12-01T08:39:02.482Z,
        "offset" => 121,
      "@version" => "1",
          "beat" => {
            "name" => "Ubuntu-20170424",
        "hostname" => "Ubuntu-20170424",
         "version" => "5.6.3"
    },
    "input_type" => "log",
          "host" => "Ubuntu-20170424",
        "source" => "/home/elk/filebeat-5.6.3-linux-x86_64/request",
          "type" => "log",
         "email" => "bbbb",
          "tags" => [
        [0] "beats_input_codec_json_applied"
    ]
}
{
     "firstName" => "3",
      "lastName" => "Harold",
    "@timestamp" => 2017-12-01T08:39:02.482Z,
        "offset" => 180,
      "@version" => "1",
          "beat" => {
            "name" => "Ubuntu-20170424",
        "hostname" => "Ubuntu-20170424",
         "version" => "5.6.3"
    },
    "input_type" => "log",
          "host" => "Ubuntu-20170424",
        "source" => "/home/elk/filebeat-5.6.3-linux-x86_64/request",
          "type" => "log",
         "email" => "cccc",
          "tags" => [
        [0] "beats_input_codec_json_applied"
    ]
}

在head外掛管理頁面看到資料：