Logstash 時間轉換（YYYY-MM-dd HH:mm:ss轉Unix時間）

衣舞晨風發表於2017-03-31

適用場景
- 日誌時間轉Unix時間
示例日誌：

2017-03-21 00:00:00,291 INFO [DubboServerHandler-10.135.6.53:20885-thread-98] i.w.w.r.m.RequirementManager [RequirementManager.java:860] fetch no data from oracle
2017-03-21 00:00:00,294 INFO [DubboServerHandler-10.135.6.53:20885-thread-98] i.w.w.r.m.RequirementManager [RequirementManager.java:1549] begin fetch data from oracle
2017-03-21 00:00:00,301 ERROR [DubboServerHandler-10.135.6.53:20885-thread-98] i.w.w.h.d.testDao [testDao.java:265] find test project failed,  error code={}
java.sql.SQLException: ORA-00942: table or view does not exist

    at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
    at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
    at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
    at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
    at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
    at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:799) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
    at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1038) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
    at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:839) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1133) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3285) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
    at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3329) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
    at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) ~[commons-dbcp-1.4.jar:1.4]
    at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) ~[commons-dbcp-1.4.jar:1.4]
    at io.jiankunking.demo.haier.dao.testDao.findBigResourceMaterialByStatus(testDao.java:252) ~[demo-requirement-1.0-SNAPSHOT.jar:na]
    at io.jiankunking.demo.requirement.manager.RequirementManager.findBigResourceMaterialByStatus(RequirementManager.java:1550) [demo-requirement-1.0-SNAPSHOT.jar:na]
    at io.jiankunking.demo.requirement.manager.RequirementManager$$FastClassByCGLIB$$b97dbf5c.invoke(<generated>) [spring-core-3.2.6.RELEASE.jar:na]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:701) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96) [spring-tx-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260) [spring-tx-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) [spring-tx-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:634) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at io.jiankunking.demo.requirement.manager.RequirementManager$$EnhancerByCGLIB$$de554283.findBigResourceMaterialByStatus(<generated>) [spring-core-3.2.6.RELEASE.jar:na]
    at io.jiankunking.demo.requirement.service.RequirementServiceImpl.fetchBigResourceMaterial(RequirementServiceImpl.java:4791) [demo-requirement-1.0-SNAPSHOT.jar:na]
    at io.jiankunking.demo.requirement.service.RequirementServiceImpl$$FastClassByCGLIB$$15e55a9c.invoke(<generated>) [spring-core-3.2.6.RELEASE.jar:na]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:701) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:42) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:42) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:42) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:634) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
    at io.jiankunking.demo.requirement.service.RequirementServiceImpl$$EnhancerByCGLIB$$e80e4734.fetchBigResourceMaterial(<generated>) [spring-core-3.2.6.RELEASE.jar:na]
    at com.alibaba.dubbo.common.bytecode.Wrapper55.invokeMethod(Wrapper55.java) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:64) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:60) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:132) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_111]
    at java.lang.Thread.run(Unknown Source) [na:1.8.0_111]
2017-03-21 00:00:00,302 INFO [DubboServerHandler-10.135.6.53:20885-thread-98] i.t.s.r.m.RequirementManager [RequirementManager.java:1556] fetch no data from oracle

logstash 配置檔案：

input {
   file {
    path => [ "/home/jiankunking/software/testdata/test.log" ]
    tags => [ "test","jiankunking" ]
    start_position => beginning
    codec =>multiline {
            pattern => "^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}"
            negate => true
            what => "previous"
        }
   }
}
filter {
    grok{
       match => ["message","%{TIMESTAMP_ISO8601:time}\s{1,}%{NOTSPACE:loglevel}\s{1,}%{NOTSPACE:thread-id}\s{1,}%{NOTSPACE:packageinfo}\s{1,}%{NOTSPACE:classinfo}\s{1,}%{GREEDYDATA:msginfo}"]
       }
    date{
        match=>["time","YYYY-MM-dd HH:mm:ss,SSS"]
        target=>"logdate"
    }
    ruby{
        code => "event.set('logdateunix',event.get('logdate').to_i)"
    }
    mutate {
        remove_field => ["message"]
    }
}
output{
   elasticsearch{
        hosts => ["111.222.333.444:9200"]
        index => "logstash-demo"
        codec => rubydebug
    }
}

具體效果如下：
這裡寫圖片描述

從 Logstash-5.0 開始， LogStash::Event 改為 Java 實現，直接使用
event[“parent”][“child”] 形式獲取的不是原事件的引用而是複製品。需要改用 event.get(‘[parent][child]’) 和 event.set(‘[parent][child]’,
‘value’) 的方法。

小注：本文演示logstash用的是5.0.1版本

5.0之前版本可以參考（未測試）：
http://www.cnblogs.com/yangwenbo214/p/6189622.html

作者：jiankunking 出處：http://blog.csdn.net/jiankunking

js把時間戳轉換成時間格式yyyy-MM-dd HH:mm:ss
2020-09-25
JS時間戳
時間戳與yyyy-mm-dd hh:mm:ss格式之間的互相轉換
2024-05-26
時間戳
JS 時間戳（毫秒）轉化為標準時間格式（YYYY-MM-DD HH:mm:ss）
2018-11-15
JS時間戳
js 將時間戳轉yyyy-MM-dd HH-mm-ss工具類
2018-05-28
JS時間戳
Hive日期、時間轉換：YYYY-MM-DD與YYYYMMDD；hh.mm.ss與hhmmss的相互轉換
2020-08-23
HiveHMM
Android時間戳轉換為標準Datetime（yyyy-MM-dd hh:mm:ss)格式
2016-08-12
Android時間戳
yyyy-MM-dd hh:mm:ss時間日期格式化程式碼
2017-04-13
【問題解決】使用YYYY-MM-dd時間轉換問題
2020-12-31
js 格式化日期 yyyy-MM-dd HH:mm:ss
2020-10-20
JS
Perl中本地時間和UNIX時間戳間相互轉換
2015-04-09
時間戳
java LocalTime格式化 HH:mm:ss與hh:mm:ss
2020-12-03
Java
js獲取系統當前時間並且轉換成yyyy-MM-dd形式
2017-10-27
JS
@JsonFormat(pattern=“yyyy-MM-dd HH:mm:ss“,timezone=“GMT+8“)
2020-12-10
JSONORM
時間轉換成時間戳
2016-08-23
時間戳
C 時間轉換時間戳
2017-05-08
時間戳
時間轉換
2018-12-21
unix時間轉換為datetimedatetime轉換為unixtime
2018-07-06
C 時間戳轉換成時間
2017-05-08
時間戳
時間相減和時間轉換
2024-08-29
如何將UTC時間轉換為Unix時間戳(How to convert UTC time to unix timestamp)
2020-11-16
時間戳
MYSQL中UNIX時間戳與日期的轉換
2013-10-23
MySql時間戳
時間日期字串轉換為時間物件
2017-04-06
字串物件
utc時間轉換成標準時間
2017-02-07
oracle 中將unix/linux時間戳進行轉換（轉）
2009-02-21
OracleLinux時間戳
Excel中時間戳轉換時間
2019-01-30
Excel時間戳
python中的時間轉換,秒級時間戳轉string，string轉時間
2024-04-20
Python時間戳
如何將時間字串轉換為時間物件
2017-03-16
字串物件
javascript將時間物件轉換為時間戳
2017-04-12
JavaScript物件時間戳
[時間格式的轉換]
2020-07-06
Java之時間轉換
2018-06-26
Java
java時間的轉換
2013-08-12
Java
轉換UTC時間格式
2024-07-08
時間戳與時間字串的多時區轉換
2019-12-30
時間戳字串
Linux命令date 日期時間和Unix時間戳互轉
2017-12-08
Linux時間戳
oracle時間間隔轉換函式
2014-04-29
Oracle函式
javascript時間戳和時間格式的相互轉換
2017-03-17
JavaScript時間戳
Oracle資料庫的SCN轉換成時間和時間轉換成SCN
2017-04-26
Oracle資料庫
linux與unix時間戳互轉
2017-09-19
Linux時間戳

Logstash 時間轉換（YYYY-MM-dd HH:mm:ss轉Unix時間）

相關文章