logstash採集Java日誌文字檔案配合grok收集到elasticsearch簡單示例

Castle125發表於2020-10-06

這裡是引用

logstash採集Java日誌文字檔案配合grok收集到elasticsearch簡單示例

logstash 配置

input {
  file {
    path => "C:/Users/Administrator/Desktop/info*.log"
    type => "log-info"
    start_position => "beginning"
    # 多行文字換行處理 (日誌檔案以日期時間格式開頭,所以按日期處理換行)
    codec => multiline {
    pattern => "^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}\s"
    negate => true
    what => "previous"
  }}
}
filter {
  grok {
  # 使用正則將日誌拆分成多列,方便查詢
  # 原日誌檔案使用log4j生成,格式為 : %d{yyyy-MM-dd HH:mm:ss.SSS} %-5level %class{36} %L %M - %msg%xEx%n
    match => { "message" => "(?<datetime>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}) %{NOTSPACE:level} (?<className> (?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]*) %{NUMBER:line} %{NOTSPACE:method} - %{GREEDYDATA:msg}" }
	overwrite => ["message"]
  }
}
output {
	elasticsearch {
    hosts => "localhost:9200"
    index => "log4j-info-%{+YYYY.MM.dd}"
	codec => "json"
  }
}

日誌檔案示例

2020-08-10 14:47:56.204 INFO  org.springframework.boot.StartupInfoLogger 50 logStarting - Starting xxxxx on pc-111with PID 14520 (started by Administrator in E:\worker\XXXX\XXX\XXXX)
2020-08-10 14:47:56.214 INFO  org.springframework.boot.SpringApplication 679 logStartupProfileInfo - The following profiles are active: dev
2020-08-10 14:47:57.723 INFO  org.springframework.data.repository.config.RepositoryConfigurationDelegate 244 multipleStoresDetected - Multiple Spring Data modules found, entering strict repository configuration mode!
2020-08-10 14:47:57.733 INFO  org.springframework.data.repository.config.RepositoryConfigurationDelegate 126 registerRepositoriesIn - Bootstrapping Spring Data repositories in DEFAULT mode.
2020-08-10 14:47:57.803 INFO  org.springframework.data.repository.config.RepositoryConfigurationDelegate 182 registerRepositoriesIn - Finished Spring Data repository scanning in 50ms. Found 0 repository interfaces.
2020-08-10 14:47:57.813 INFO  org.springframework.data.repository.config.RepositoryConfigurationDelegate 244 multipleStoresDetected - Multiple Spring Data modules found, entering strict repository configuration mode!
2020-08-10 14:47:57.813 INFO  org.springframework.data.repository.config.RepositoryConfigurationDelegate 126 registerRepositoriesIn - Bootstrapping Spring Data repositories in DEFAULT mode.
2020-08-10 14:47:57.853 INFO  org.springframework.data.repository.config.RepositoryConfigurationDelegate 182 registerRepositoriesIn - Finished Spring Data repository scanning in 30ms. Found 0 repository interfaces.
2020-08-10 14:47:58.602 INFO  org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker 330 postProcessAfterInitialization - Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$da77a69e] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-08-10 14:47:59.152 INFO  org.springframework.boot.web.embedded.tomcat.TomcatWebServer 90 initialize - Tomcat initialized with port(s): 8090 (http)
2020-08-10 14:47:59.162 INFO  org.apache.juli.logging.DirectJDKLog 173 log - Initializing ProtocolHandler ["http-nio-8090"]
2020-08-10 14:47:59.182 INFO  org.apache.juli.logging.DirectJDKLog 173 log - Starting service [Tomcat]
2020-08-10 14:47:59.182 INFO  org.apache.juli.logging.DirectJDKLog 173 log - Starting Servlet Engine: Apache Tomcat/9.0.13
2020-08-10 14:47:59.182 INFO  org.apache.juli.logging.DirectJDKLog 173 log - The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [D:\Program Files\java\jdk1.8.0_91\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;D:\Program Files\java\jdk1.8.0_91\bin;D:\Program Files\java\mysql-5.7.19-winx64\bin;D:\Program Files\Git\cmd;D:\Program Files\java\apache-maven-3.6.0\bin;D:\Program Files\TortoiseSVN\bin;D:\Program Files\nodejs\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files\python\Scripts\;C:\Program Files\python\;D:\Program Files\Microsoft VS Code\bin;C:\Users\Administrator\AppData\Roaming\npm;E:\IlikeStudy\hexo\node_modules\.bin;C:\Users\Administrator\AppData\Local\Microsoft\WindowsApps;C:\Users\Administrator\.dotnet\tools;.]
2020-08-10 14:47:59.362 INFO  org.apache.juli.logging.DirectJDKLog 173 log - Initializing Spring embedded WebApplicationContext
2020-08-10 14:47:59.362 INFO  org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext 296 prepareWebApplicationContext - Root WebApplicationContext: initialization completed in 3078 ms
2020-08-10 14:47:59.992 INFO  com.zaxxer.hikari.HikariDataSource 110 getConnection - HikariPool-1 - Starting...
2020-08-10 14:48:00.282 INFO  com.zaxxer.hikari.pool.PoolBase 525 getAndSetNetworkTimeout - HikariPool-1 - Driver does not support get/set network timeout for connections. (oracle.jdbc.driver.T4CConnection.getNetworkTimeout()I)
2020-08-10 14:48:00.292 INFO  com.zaxxer.hikari.HikariDataSource 123 getConnection - HikariPool-1 - Start completed.
2020-08-10 14:48:00.521 INFO  org.hibernate.Version 46 logVersion - HHH000412: Hibernate Core {5.3.7.Final}
2020-08-10 14:48:00.521 INFO  org.hibernate.cfg.Environment 213 <clinit> - HHH000206: hibernate.properties not found
2020-08-10 14:48:00.711 INFO  org.hibernate.annotations.common.reflection.java.JavaReflectionManager 49 <clinit> - HCANN000001: Hibernate Commons Annotations {5.0.4.Final}
2020-08-10 14:48:01.131 INFO  org.hibernate.dialect.Dialect 157 <init> - HHH000400: Using dialect: org.hibernate.dialect.Oracle12cDialect
2020-08-10 14:48:01.421 INFO  org.hibernate.type.BasicTypeRegistry 159 register - HHH000270: Type registration [byte[]] overrides previous : org.hibernate.type.BinaryType@47eae91d
2020-08-10 14:48:01.421 INFO  org.hibernate.type.BasicTypeRegistry 159 register - HHH000270: Type registration [[B] overrides previous : org.hibernate.type.BinaryType@47eae91d
2020-08-10 14:48:01.421 INFO  org.hibernate.type.BasicTypeRegistry 159 register - HHH000270: Type registration [Byte[]] overrides previous : org.hibernate.type.WrapperBinaryType@33891646
2020-08-10 14:48:01.421 INFO  org.hibernate.type.BasicTypeRegistry 159 register - HHH000270: Type registration [[Ljava.lang.Byte;] overrides previous : org.hibernate.type.WrapperBinaryType@33891646
2020-08-10 14:48:01.651 INFO  org.springframework.orm.jpa.AbstractEntityManagerFactoryBean 415 buildNativeEntityManagerFactory - Initialized JPA EntityManagerFactory for persistence unit 'default'
2020-08-10 14:48:04.999 INFO  com.XXX.config.RedisConfig 57 cacheManager - 初始化 -> [CacheManager RedisCacheManager Start]
2020-08-10 14:48:05.009 INFO  com.xxx.config.RedisConfig 68 errorHandler - 初始化 -> [Redis CacheErrorHandler]
2020-08-10 14:48:05.089 INFO  org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean 437 buildServiceFromClass - Creating Service {http://webservice.xxx.com}xxxxx from class com.xxxx.webservice.xxxCxfService
2020-08-10 14:48:05.589 INFO  org.apache.cxf.endpoint.ServerImpl 85 initDestination - Setting the server's publish address to be /xxxWebservice
2020-08-10 14:48:05.809 INFO  org.springframework.scheduling.concurrent.ExecutorConfigurationSupport 171 initialize - Initializing ExecutorService 'applicationTaskExecutor'
2020-08-10 14:48:06.208 INFO  org.springframework.scheduling.concurrent.ExecutorConfigurationSupport 171 initialize - Initializing ExecutorService 'taskScheduler'
2020-08-10 14:48:06.268 INFO  springfox.documentation.spring.web.plugins.DocumentationPluginsBootstrapper 120 start - Context refreshed
2020-08-10 14:48:06.288 INFO  springfox.documentation.spring.web.plugins.DocumentationPluginsBootstrapper 123 start - Found 1 custom documentation plugin(s)
2020-08-10 14:48:06.308 INFO  springfox.documentation.spring.web.scanners.ApiListingReferenceScanner 41 scan - Scanning for api listing references
2020-08-10 14:48:07.398 INFO  springfox.documentation.spring.web.readers.operation.CachingOperationNameGenerator 40 startingWith - Generating unique operation named: cancelOrderUsingGET_1
2020-08-10 14:48:07.408 INFO  springfox.documentation.spring.web.readers.operation.CachingOperationNameGenerator 40 startingWith - Generating unique operation named: cancelOrderUsingGET_2
2020-08-10 14:48:07.538 INFO  springfox.documentation.spring.web.readers.operation.CachingOperationNameGenerator 40 startingWith - Generating unique operation named: getOrderInfoUsingGET_1
2020-08-10 14:48:07.588 INFO  springfox.documentation.spring.web.readers.operation.CachingOperationNameGenerator 40 startingWith - Generating unique operation named: getPayRecordUrlUsingGET_1
2020-08-10 14:48:07.698 INFO  springfox.documentation.spring.web.readers.operation.CachingOperationNameGenerator 40 startingWith - Generating unique operation named: refundOrderUsingGET_1
2020-08-10 14:48:07.738 INFO  springfox.documentation.spring.web.readers.operation.CachingOperationNameGenerator 40 startingWith - Generating unique operation named: tradeRefundUsingGET_1
2020-08-10 14:48:07.798 INFO  org.apache.juli.logging.DirectJDKLog 173 log - Starting ProtocolHandler ["http-nio-8090"]
2020-08-10 14:48:07.818 INFO  org.apache.juli.logging.DirectJDKLog 173 log - Using a shared selector for servlet write/read
2020-08-10 14:48:07.838 INFO  org.springframework.boot.web.embedded.tomcat.TomcatWebServer 204 start - Tomcat started on port(s): 8090 (http) with context path '/xxxx'
2020-08-10 14:48:07.838 INFO  org.springframework.boot.StartupInfoLogger 59 logStarted - Started TongjizizhuApplication in 12.244 seconds (JVM running for 15.08)
2020-08-10 14:48:21.568 INFO  org.apache.juli.logging.DirectJDKLog 173 log - Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-08-10 14:48:21.568 INFO  org.springframework.web.servlet.FrameworkServlet 524 initServletBean - Initializing Servlet 'dispatcherServlet'
2020-08-10 14:48:21.583 INFO  org.springframework.web.servlet.FrameworkServlet 546 initServletBean - Completed initialization in 15 ms
2020-08-10 14:51:30.565 INFO  com.wantup.common.aspect.LogAspect 59 controllerLogAround - [AOP-LOG-START]
	requestMark: 6558c294-e3c8-407b-aeb8-93de31fa6e55
	requestIP: 0:0:0:0:0:0:0:1
	contentType:null
	requestUrl: http://localhost:8090/xxx/xxx/xxx
	requestMethod: GET
	requestParams: null
	targetClassAndMethod: com.xxx.controller.xxx#xxx
2020-08-10 14:51:33.980 INFO  io.lettuce.core.EpollProvider 68 <clinit> - Starting without optional epoll library
2020-08-10 14:51:33.984 INFO  io.lettuce.core.KqueueProvider 70 <clinit> - Starting without optional kqueue library
2020-08-10 14:51:38.725 INFO  com.wantup.common.aspect.LogAspect 67 controllerLogAround - [AOP-LOG-END]
	JsonResult [rspCode=重新整理成功, rspMesg=Response Success, rspValue=null]
2020-08-10 15:04:25.017 INFO  com.wantup.common.aspect.LogAspect 59 controllerLogAround - [AOP-LOG-START]
	requestMark: 4cfd2dd0-a56b-4b21-b033-803365ad01dc
	requestIP: 0:0:0:0:0:0:0:1
	contentType:null
	requestUrl: http://localhost:8090/xxx/other/c1paperQuery
	requestMethod: GET
	requestParams: termno=xxx&printType=1
	targetClassAndMethod: com.xxx.controller.PrintController#c1paperQuery
2020-08-10 15:11:18.967 INFO  com.xxx.common.aspect.LogAspect 59 controllerLogAround - [AOP-LOG-START]
	requestMark: 0197f440-1ab0-45a7-a207-3d389c5631f9
	requestIP: 0:0:0:0:0:0:0:1
	contentType:null
	requestUrl: http://localhost:8090/xxxx
	requestMethod: GET
	requestParams: idCard=xxx&termNo=xxxx
	targetClassAndMethod: com.xxx.controller.OutPatientController#getCardInfoByIdCard
2020-08-10 15:11:45.005 INFO  com.xxx.common.aspect.LogAspect 59 controllerLogAround - [AOP-LOG-START]
	requestMark: 8e3c9001-0072-4f6c-982d-04fbd324b12f
	requestIP: 0:0:0:0:0:0:0:1
	contentType:null
	requestUrl: http://localhost:8090/xxxxx
	requestMethod: GET
	requestParams: idCard=xxx&termNo=xxxx
	targetClassAndMethod: com.xxx.controller.OutPatientController#getCardInfoByIdCard
2020-08-10 15:11:45.006 INFO  org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean 437 buildServiceFromClass - Creating Service {http://xxxx from class xxxx.config.cxfclient.WebServiceInterface
2020-08-10 15:11:45.070 INFO  com.xxx.service.impl.OutPatientServiceImpl 88 getCardInfoByIdCard - XXXXXXXX<Response><MessageHeader><MsgId>XXXXXXXX</MsgId><EventType>ACK_PCA_GET_PATIENT_ELEC_CARD</EventType><Receiver>SELFSERVICE</Receiver><Sender>S</Sender><SendTime>2020-08-10 15:12:57</SendTime></MessageHeader><MessageBody><Result><CardNo></CardNo><Code>CE</Code><Desc>失敗,null</Desc></Result></MessageBody></Response>
2020-08-10 15:14:35.228 INFO  com.xxx.common.aspect.LogAspect 59 controllerLogAround - [AOP-LOG-START]
	requestMark: 81943f2a-b42f-4bc9-b28a-d198cb42fae4
	requestIP: 0:0:0:0:0:0:0:1
	contentType:null
	requestUrl: http://localhost:8090/xxxxx
	requestMethod: GET
	requestParams: XXXXX
	targetClassAndMethod: com.xxx.controller.OutPatientController#getCardInfoByIdCard
2020-08-10 15:14:35.229 INFO  org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean 437 buildServiceFromClass - Creating Service {http://webService.eif.supcom.th.com/}WebServiceInterfaceService from class com.xxx.config.cxfclient.WebServiceInterface
2020-08-10 15:14:35.285 INFO  com.xxx.service.impl.OutPatientServiceImpl 88 getCardInfoByIdCard - XXXXXXXX<Response><MessageHeader><MsgId>XXXXXXX</MsgId><EventType>ACK_PCA_GET_PATIENT_ELEC_CARD</EventType><Receiver>SELFSERVICE</Receiver><Sender>S</Sender><SendTime>2020-08-10 15:15:47</SendTime></MessageHeader><MessageBody><Result><CardNo></CardNo><Code>CE</Code><Desc>失敗,null</Desc></Result></MessageBody></Response>

kibana結果展示:

查詢結果


參考文章:

logstash使用grok收集java日誌 ©王賽超

logstash-patterns-core/patterns/java

logstash 採集 Java log4j的日誌(全面介紹)

第六章·Logstash深入-收集java日誌

相關文章