logstash輸出到influxdb

曲珂發表於2014-09-25
UX

用了這個logstash擴充套件

https://github.com/PeterPaulH/logstash-influxdb/blob/master/src/influxdb.rb

把這個檔案放到 logstash-1.4.2/lib/logstash/outputs

看一下午logstash的文件，終於解決了自己的需求

用python描述就是這樣的

開發要求統計日誌中各種型別的數量

while True:
    line = f.readline()
    try:
        if '"type":"text","receiver_id"' in line:
            type = 'directmessage'
        elif '"subtype":"unfollow"' in line:
            type = 'unfollow'
        elif '"subtype":"follow"' in line:
            type = 'follow'
        elif '"subtype":"status"' in line:
            type = 'weibo'
        elif '"subtype":"comment"' in line:
            type = 'comment'
        else:
            type = None

        if type:
            data = [
                {"name":"pingpong_processor",
                 "columns" : ["type"],
                 "points" : [[type]]

                }
            ]

 logstash配置檔案如下

input {
  stdin {}
}

filter {
  if '"type":"text"' in [message] {
      mutate {
        add_field => { "type" => "directmessage" }
        remove_field => [ "message", "search" , "@version" ]
      }
  } else if '"subtype":"unfollow"' in [message] {
      mutate {
        add_field => { "type" => "unfollow" }
        remove_field => [ "message", "search" , "@version" ]
      }
  } else if '"subtype":"follow"' in [message] {
      mutate {
        add_field => { "type" => "follow" }
        remove_field => [ "message", "search" , "@version" ]
      }
  } else if '"subtype":"status"' in [message] {
      mutate {
        add_field => { "type" => "weibo" }
        remove_field => [ "message", "search" , "@version" ]
      }
  } else if '"subtype":"comment"' in [message] {
      mutate {
        add_field => { "type" => "comment" }
        remove_field => [ "message", "search" , "@version" ]
      }
  } else {
      drop {}
  }
  
}

output {
  influxdb {
    host => "10.75.28.180"
    port => 4444
    name => ["pingpong_processor"]
    columns => ["type", "host"]
    points => [
        "%{type}", "c",
        "%{host}", "c"
    ]

  }

  stdout {}
}

 別忘記把influxdb的配置也修改下，因為預設upd協議是沒有開啟的

# Configure the udp api
  [input_plugins.udp]
  enabled = true
  port = 4444
  database = "pingpong_processor"

 我用的influxdb版本是 influxdb-0.8.2-1.x86_64,用/etc/init.d/influxdb這個啟動報錯，無奈手動啟動的

/usr/bin/influxdb -pidfile /tmp/influxdb.pid -config config.toml

看一下udp埠是否啟動了

netstat -anup|grep influxdb
udp        0      0 :::4444                     :::*                                    27512/influxdb

 完工

相關文章