Logstash 時間轉換(YYYY-MM-dd HH:mm:ss轉Unix時間)
適用場景
- 日誌時間轉Unix時間
示例日誌:
2017-03-21 00:00:00,291 INFO [DubboServerHandler-10.135.6.53:20885-thread-98] i.w.w.r.m.RequirementManager [RequirementManager.java:860] fetch no data from oracle
2017-03-21 00:00:00,294 INFO [DubboServerHandler-10.135.6.53:20885-thread-98] i.w.w.r.m.RequirementManager [RequirementManager.java:1549] begin fetch data from oracle
2017-03-21 00:00:00,301 ERROR [DubboServerHandler-10.135.6.53:20885-thread-98] i.w.w.h.d.testDao [testDao.java:265] find test project failed, error code={}
java.sql.SQLException: ORA-00942: table or view does not exist
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:799) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1038) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:839) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1133) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3285) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3329) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) ~[commons-dbcp-1.4.jar:1.4]
at io.jiankunking.demo.haier.dao.testDao.findBigResourceMaterialByStatus(testDao.java:252) ~[demo-requirement-1.0-SNAPSHOT.jar:na]
at io.jiankunking.demo.requirement.manager.RequirementManager.findBigResourceMaterialByStatus(RequirementManager.java:1550) [demo-requirement-1.0-SNAPSHOT.jar:na]
at io.jiankunking.demo.requirement.manager.RequirementManager$$FastClassByCGLIB$$b97dbf5c.invoke(<generated>) [spring-core-3.2.6.RELEASE.jar:na]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:701) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96) [spring-tx-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260) [spring-tx-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) [spring-tx-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:634) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at io.jiankunking.demo.requirement.manager.RequirementManager$$EnhancerByCGLIB$$de554283.findBigResourceMaterialByStatus(<generated>) [spring-core-3.2.6.RELEASE.jar:na]
at io.jiankunking.demo.requirement.service.RequirementServiceImpl.fetchBigResourceMaterial(RequirementServiceImpl.java:4791) [demo-requirement-1.0-SNAPSHOT.jar:na]
at io.jiankunking.demo.requirement.service.RequirementServiceImpl$$FastClassByCGLIB$$15e55a9c.invoke(<generated>) [spring-core-3.2.6.RELEASE.jar:na]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:701) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:42) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:42) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:42) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:634) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at io.jiankunking.demo.requirement.service.RequirementServiceImpl$$EnhancerByCGLIB$$e80e4734.fetchBigResourceMaterial(<generated>) [spring-core-3.2.6.RELEASE.jar:na]
at com.alibaba.dubbo.common.bytecode.Wrapper55.invokeMethod(Wrapper55.java) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:64) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:60) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:132) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_111]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_111]
2017-03-21 00:00:00,302 INFO [DubboServerHandler-10.135.6.53:20885-thread-98] i.t.s.r.m.RequirementManager [RequirementManager.java:1556] fetch no data from oracle
logstash 配置檔案:
input {
file {
path => [ "/home/jiankunking/software/testdata/test.log" ]
tags => [ "test","jiankunking" ]
start_position => beginning
codec =>multiline {
pattern => "^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}"
negate => true
what => "previous"
}
}
}
filter {
grok{
match => ["message","%{TIMESTAMP_ISO8601:time}\s{1,}%{NOTSPACE:loglevel}\s{1,}%{NOTSPACE:thread-id}\s{1,}%{NOTSPACE:packageinfo}\s{1,}%{NOTSPACE:classinfo}\s{1,}%{GREEDYDATA:msginfo}"]
}
date{
match=>["time","YYYY-MM-dd HH:mm:ss,SSS"]
target=>"logdate"
}
ruby{
code => "event.set('logdateunix',event.get('logdate').to_i)"
}
mutate {
remove_field => ["message"]
}
}
output{
elasticsearch{
hosts => ["111.222.333.444:9200"]
index => "logstash-demo"
codec => rubydebug
}
}
具體效果如下:
從 Logstash-5.0 開始, LogStash::Event 改為 Java 實現,直接使用
event[“parent”][“child”] 形式獲取的不是原事件的引用而是複製品。需要改用 event.get(‘[parent][child]’) 和 event.set(‘[parent][child]’,
‘value’) 的方法。
小注:本文演示logstash用的是5.0.1版本
5.0之前版本可以參考(未測試):
http://www.cnblogs.com/yangwenbo214/p/6189622.html
作者:jiankunking 出處:http://blog.csdn.net/jiankunking
相關文章
- js把時間戳轉換成時間格式yyyy-MM-dd HH:mm:ssJS時間戳
- 時間戳與yyyy-mm-dd hh:mm:ss格式之間的互相轉換時間戳
- JS 時間戳(毫秒)轉化為標準時間格式(YYYY-MM-DD HH:mm:ss)JS時間戳
- js 將時間戳轉yyyy-MM-dd HH-mm-ss工具類JS時間戳
- Hive日期、時間轉換:YYYY-MM-DD與YYYYMMDD;hh.mm.ss與hhmmss的相互轉換HiveHMM
- Android時間戳轉換為標準Datetime(yyyy-MM-dd hh:mm:ss)格式Android時間戳
- yyyy-MM-dd hh:mm:ss時間日期格式化程式碼
- 【問題解決】使用YYYY-MM-dd時間轉換問題
- js 格式化日期 yyyy-MM-dd HH:mm:ssJS
- Perl中本地時間和UNIX時間戳間相互轉換時間戳
- java LocalTime格式化 HH:mm:ss與hh:mm:ssJava
- js獲取系統當前時間並且轉換成yyyy-MM-dd形式JS
- @JsonFormat(pattern=“yyyy-MM-dd HH:mm:ss“,timezone=“GMT+8“)JSONORM
- 時間轉換成時間戳時間戳
- C 時間轉換時間戳時間戳
- 時間轉換
- unix時間轉換為datetimedatetime轉換為unixtime
- C 時間戳轉換成時間時間戳
- 時間相減和時間轉換
- 如何將UTC時間轉換為Unix時間戳(How to convert UTC time to unix timestamp)時間戳
- MYSQL中UNIX時間戳與日期的轉換MySql時間戳
- 時間日期字串轉換為時間物件字串物件
- utc時間轉換成標準時間
- oracle 中將unix/linux時間戳進行轉換(轉)OracleLinux時間戳
- Excel中時間戳轉換時間Excel時間戳
- python中的時間轉換,秒級時間戳轉string,string轉時間Python時間戳
- 如何將時間字串轉換為時間物件字串物件
- javascript將時間物件轉換為時間戳JavaScript物件時間戳
- [時間格式的轉換]
- Java之時間轉換Java
- java時間的轉換Java
- 轉換UTC時間格式
- 工具方法:時間轉換
- 時間戳與時間字串的多時區轉換時間戳字串
- Linux命令date 日期時間和Unix時間戳互轉Linux時間戳
- oracle時間間隔轉換函式Oracle函式
- javascript時間戳和時間格式的相互轉換JavaScript時間戳
- Oracle資料庫的SCN轉換成時間和時間轉換成SCNOracle資料庫