logstash採集Java日誌文字檔案配合grok收集到elasticsearch簡單示例
這裡是引用
logstash採集Java日誌文字檔案配合grok收集到elasticsearch簡單示例
logstash 配置
input {
file {
path => "C:/Users/Administrator/Desktop/info*.log"
type => "log-info"
start_position => "beginning"
# 多行文字換行處理 (日誌檔案以日期時間格式開頭,所以按日期處理換行)
codec => multiline {
pattern => "^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}\s"
negate => true
what => "previous"
}}
}
filter {
grok {
# 使用正則將日誌拆分成多列,方便查詢
# 原日誌檔案使用log4j生成,格式為 : %d{yyyy-MM-dd HH:mm:ss.SSS} %-5level %class{36} %L %M - %msg%xEx%n
match => { "message" => "(?<datetime>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}) %{NOTSPACE:level} (?<className> (?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]*) %{NUMBER:line} %{NOTSPACE:method} - %{GREEDYDATA:msg}" }
overwrite => ["message"]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "log4j-info-%{+YYYY.MM.dd}"
codec => "json"
}
}
日誌檔案示例
2020-08-10 14:47:56.204 INFO org.springframework.boot.StartupInfoLogger 50 logStarting - Starting xxxxx on pc-111with PID 14520 (started by Administrator in E:\worker\XXXX\XXX\XXXX)
2020-08-10 14:47:56.214 INFO org.springframework.boot.SpringApplication 679 logStartupProfileInfo - The following profiles are active: dev
2020-08-10 14:47:57.723 INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate 244 multipleStoresDetected - Multiple Spring Data modules found, entering strict repository configuration mode!
2020-08-10 14:47:57.733 INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate 126 registerRepositoriesIn - Bootstrapping Spring Data repositories in DEFAULT mode.
2020-08-10 14:47:57.803 INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate 182 registerRepositoriesIn - Finished Spring Data repository scanning in 50ms. Found 0 repository interfaces.
2020-08-10 14:47:57.813 INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate 244 multipleStoresDetected - Multiple Spring Data modules found, entering strict repository configuration mode!
2020-08-10 14:47:57.813 INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate 126 registerRepositoriesIn - Bootstrapping Spring Data repositories in DEFAULT mode.
2020-08-10 14:47:57.853 INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate 182 registerRepositoriesIn - Finished Spring Data repository scanning in 30ms. Found 0 repository interfaces.
2020-08-10 14:47:58.602 INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker 330 postProcessAfterInitialization - Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$da77a69e] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-08-10 14:47:59.152 INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer 90 initialize - Tomcat initialized with port(s): 8090 (http)
2020-08-10 14:47:59.162 INFO org.apache.juli.logging.DirectJDKLog 173 log - Initializing ProtocolHandler ["http-nio-8090"]
2020-08-10 14:47:59.182 INFO org.apache.juli.logging.DirectJDKLog 173 log - Starting service [Tomcat]
2020-08-10 14:47:59.182 INFO org.apache.juli.logging.DirectJDKLog 173 log - Starting Servlet Engine: Apache Tomcat/9.0.13
2020-08-10 14:47:59.182 INFO org.apache.juli.logging.DirectJDKLog 173 log - The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [D:\Program Files\java\jdk1.8.0_91\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;D:\Program Files\java\jdk1.8.0_91\bin;D:\Program Files\java\mysql-5.7.19-winx64\bin;D:\Program Files\Git\cmd;D:\Program Files\java\apache-maven-3.6.0\bin;D:\Program Files\TortoiseSVN\bin;D:\Program Files\nodejs\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files\python\Scripts\;C:\Program Files\python\;D:\Program Files\Microsoft VS Code\bin;C:\Users\Administrator\AppData\Roaming\npm;E:\IlikeStudy\hexo\node_modules\.bin;C:\Users\Administrator\AppData\Local\Microsoft\WindowsApps;C:\Users\Administrator\.dotnet\tools;.]
2020-08-10 14:47:59.362 INFO org.apache.juli.logging.DirectJDKLog 173 log - Initializing Spring embedded WebApplicationContext
2020-08-10 14:47:59.362 INFO org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext 296 prepareWebApplicationContext - Root WebApplicationContext: initialization completed in 3078 ms
2020-08-10 14:47:59.992 INFO com.zaxxer.hikari.HikariDataSource 110 getConnection - HikariPool-1 - Starting...
2020-08-10 14:48:00.282 INFO com.zaxxer.hikari.pool.PoolBase 525 getAndSetNetworkTimeout - HikariPool-1 - Driver does not support get/set network timeout for connections. (oracle.jdbc.driver.T4CConnection.getNetworkTimeout()I)
2020-08-10 14:48:00.292 INFO com.zaxxer.hikari.HikariDataSource 123 getConnection - HikariPool-1 - Start completed.
2020-08-10 14:48:00.521 INFO org.hibernate.Version 46 logVersion - HHH000412: Hibernate Core {5.3.7.Final}
2020-08-10 14:48:00.521 INFO org.hibernate.cfg.Environment 213 <clinit> - HHH000206: hibernate.properties not found
2020-08-10 14:48:00.711 INFO org.hibernate.annotations.common.reflection.java.JavaReflectionManager 49 <clinit> - HCANN000001: Hibernate Commons Annotations {5.0.4.Final}
2020-08-10 14:48:01.131 INFO org.hibernate.dialect.Dialect 157 <init> - HHH000400: Using dialect: org.hibernate.dialect.Oracle12cDialect
2020-08-10 14:48:01.421 INFO org.hibernate.type.BasicTypeRegistry 159 register - HHH000270: Type registration [byte[]] overrides previous : org.hibernate.type.BinaryType@47eae91d
2020-08-10 14:48:01.421 INFO org.hibernate.type.BasicTypeRegistry 159 register - HHH000270: Type registration [[B] overrides previous : org.hibernate.type.BinaryType@47eae91d
2020-08-10 14:48:01.421 INFO org.hibernate.type.BasicTypeRegistry 159 register - HHH000270: Type registration [Byte[]] overrides previous : org.hibernate.type.WrapperBinaryType@33891646
2020-08-10 14:48:01.421 INFO org.hibernate.type.BasicTypeRegistry 159 register - HHH000270: Type registration [[Ljava.lang.Byte;] overrides previous : org.hibernate.type.WrapperBinaryType@33891646
2020-08-10 14:48:01.651 INFO org.springframework.orm.jpa.AbstractEntityManagerFactoryBean 415 buildNativeEntityManagerFactory - Initialized JPA EntityManagerFactory for persistence unit 'default'
2020-08-10 14:48:04.999 INFO com.XXX.config.RedisConfig 57 cacheManager - 初始化 -> [CacheManager RedisCacheManager Start]
2020-08-10 14:48:05.009 INFO com.xxx.config.RedisConfig 68 errorHandler - 初始化 -> [Redis CacheErrorHandler]
2020-08-10 14:48:05.089 INFO org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean 437 buildServiceFromClass - Creating Service {http://webservice.xxx.com}xxxxx from class com.xxxx.webservice.xxxCxfService
2020-08-10 14:48:05.589 INFO org.apache.cxf.endpoint.ServerImpl 85 initDestination - Setting the server's publish address to be /xxxWebservice
2020-08-10 14:48:05.809 INFO org.springframework.scheduling.concurrent.ExecutorConfigurationSupport 171 initialize - Initializing ExecutorService 'applicationTaskExecutor'
2020-08-10 14:48:06.208 INFO org.springframework.scheduling.concurrent.ExecutorConfigurationSupport 171 initialize - Initializing ExecutorService 'taskScheduler'
2020-08-10 14:48:06.268 INFO springfox.documentation.spring.web.plugins.DocumentationPluginsBootstrapper 120 start - Context refreshed
2020-08-10 14:48:06.288 INFO springfox.documentation.spring.web.plugins.DocumentationPluginsBootstrapper 123 start - Found 1 custom documentation plugin(s)
2020-08-10 14:48:06.308 INFO springfox.documentation.spring.web.scanners.ApiListingReferenceScanner 41 scan - Scanning for api listing references
2020-08-10 14:48:07.398 INFO springfox.documentation.spring.web.readers.operation.CachingOperationNameGenerator 40 startingWith - Generating unique operation named: cancelOrderUsingGET_1
2020-08-10 14:48:07.408 INFO springfox.documentation.spring.web.readers.operation.CachingOperationNameGenerator 40 startingWith - Generating unique operation named: cancelOrderUsingGET_2
2020-08-10 14:48:07.538 INFO springfox.documentation.spring.web.readers.operation.CachingOperationNameGenerator 40 startingWith - Generating unique operation named: getOrderInfoUsingGET_1
2020-08-10 14:48:07.588 INFO springfox.documentation.spring.web.readers.operation.CachingOperationNameGenerator 40 startingWith - Generating unique operation named: getPayRecordUrlUsingGET_1
2020-08-10 14:48:07.698 INFO springfox.documentation.spring.web.readers.operation.CachingOperationNameGenerator 40 startingWith - Generating unique operation named: refundOrderUsingGET_1
2020-08-10 14:48:07.738 INFO springfox.documentation.spring.web.readers.operation.CachingOperationNameGenerator 40 startingWith - Generating unique operation named: tradeRefundUsingGET_1
2020-08-10 14:48:07.798 INFO org.apache.juli.logging.DirectJDKLog 173 log - Starting ProtocolHandler ["http-nio-8090"]
2020-08-10 14:48:07.818 INFO org.apache.juli.logging.DirectJDKLog 173 log - Using a shared selector for servlet write/read
2020-08-10 14:48:07.838 INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer 204 start - Tomcat started on port(s): 8090 (http) with context path '/xxxx'
2020-08-10 14:48:07.838 INFO org.springframework.boot.StartupInfoLogger 59 logStarted - Started TongjizizhuApplication in 12.244 seconds (JVM running for 15.08)
2020-08-10 14:48:21.568 INFO org.apache.juli.logging.DirectJDKLog 173 log - Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-08-10 14:48:21.568 INFO org.springframework.web.servlet.FrameworkServlet 524 initServletBean - Initializing Servlet 'dispatcherServlet'
2020-08-10 14:48:21.583 INFO org.springframework.web.servlet.FrameworkServlet 546 initServletBean - Completed initialization in 15 ms
2020-08-10 14:51:30.565 INFO com.wantup.common.aspect.LogAspect 59 controllerLogAround - [AOP-LOG-START]
requestMark: 6558c294-e3c8-407b-aeb8-93de31fa6e55
requestIP: 0:0:0:0:0:0:0:1
contentType:null
requestUrl: http://localhost:8090/xxx/xxx/xxx
requestMethod: GET
requestParams: null
targetClassAndMethod: com.xxx.controller.xxx#xxx
2020-08-10 14:51:33.980 INFO io.lettuce.core.EpollProvider 68 <clinit> - Starting without optional epoll library
2020-08-10 14:51:33.984 INFO io.lettuce.core.KqueueProvider 70 <clinit> - Starting without optional kqueue library
2020-08-10 14:51:38.725 INFO com.wantup.common.aspect.LogAspect 67 controllerLogAround - [AOP-LOG-END]
JsonResult [rspCode=重新整理成功, rspMesg=Response Success, rspValue=null]
2020-08-10 15:04:25.017 INFO com.wantup.common.aspect.LogAspect 59 controllerLogAround - [AOP-LOG-START]
requestMark: 4cfd2dd0-a56b-4b21-b033-803365ad01dc
requestIP: 0:0:0:0:0:0:0:1
contentType:null
requestUrl: http://localhost:8090/xxx/other/c1paperQuery
requestMethod: GET
requestParams: termno=xxx&printType=1
targetClassAndMethod: com.xxx.controller.PrintController#c1paperQuery
2020-08-10 15:11:18.967 INFO com.xxx.common.aspect.LogAspect 59 controllerLogAround - [AOP-LOG-START]
requestMark: 0197f440-1ab0-45a7-a207-3d389c5631f9
requestIP: 0:0:0:0:0:0:0:1
contentType:null
requestUrl: http://localhost:8090/xxxx
requestMethod: GET
requestParams: idCard=xxx&termNo=xxxx
targetClassAndMethod: com.xxx.controller.OutPatientController#getCardInfoByIdCard
2020-08-10 15:11:45.005 INFO com.xxx.common.aspect.LogAspect 59 controllerLogAround - [AOP-LOG-START]
requestMark: 8e3c9001-0072-4f6c-982d-04fbd324b12f
requestIP: 0:0:0:0:0:0:0:1
contentType:null
requestUrl: http://localhost:8090/xxxxx
requestMethod: GET
requestParams: idCard=xxx&termNo=xxxx
targetClassAndMethod: com.xxx.controller.OutPatientController#getCardInfoByIdCard
2020-08-10 15:11:45.006 INFO org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean 437 buildServiceFromClass - Creating Service {http://xxxx from class xxxx.config.cxfclient.WebServiceInterface
2020-08-10 15:11:45.070 INFO com.xxx.service.impl.OutPatientServiceImpl 88 getCardInfoByIdCard - XXXXXXXX<Response><MessageHeader><MsgId>XXXXXXXX</MsgId><EventType>ACK_PCA_GET_PATIENT_ELEC_CARD</EventType><Receiver>SELFSERVICE</Receiver><Sender>S</Sender><SendTime>2020-08-10 15:12:57</SendTime></MessageHeader><MessageBody><Result><CardNo></CardNo><Code>CE</Code><Desc>失敗,null</Desc></Result></MessageBody></Response>
2020-08-10 15:14:35.228 INFO com.xxx.common.aspect.LogAspect 59 controllerLogAround - [AOP-LOG-START]
requestMark: 81943f2a-b42f-4bc9-b28a-d198cb42fae4
requestIP: 0:0:0:0:0:0:0:1
contentType:null
requestUrl: http://localhost:8090/xxxxx
requestMethod: GET
requestParams: XXXXX
targetClassAndMethod: com.xxx.controller.OutPatientController#getCardInfoByIdCard
2020-08-10 15:14:35.229 INFO org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean 437 buildServiceFromClass - Creating Service {http://webService.eif.supcom.th.com/}WebServiceInterfaceService from class com.xxx.config.cxfclient.WebServiceInterface
2020-08-10 15:14:35.285 INFO com.xxx.service.impl.OutPatientServiceImpl 88 getCardInfoByIdCard - XXXXXXXX<Response><MessageHeader><MsgId>XXXXXXX</MsgId><EventType>ACK_PCA_GET_PATIENT_ELEC_CARD</EventType><Receiver>SELFSERVICE</Receiver><Sender>S</Sender><SendTime>2020-08-10 15:15:47</SendTime></MessageHeader><MessageBody><Result><CardNo></CardNo><Code>CE</Code><Desc>失敗,null</Desc></Result></MessageBody></Response>
kibana結果展示:
參考文章:
logstash使用grok收集java日誌 ©王賽超
logstash-patterns-core/patterns/java
相關文章
- Logtail檔案日誌採集之完整正則模式AI模式
- 日誌採集/分析
- Logstash收集json格式日誌檔案如何寫配置檔案JSON
- 手把手教程:使用 Fluentbit 採集夜鶯日誌寫入 ElasticSearchElasticsearch
- 日誌採集框架Flume框架
- Kibana+Logstash+Elasticsearch 日誌查詢系統Elasticsearch
- Logstash 配置Java日誌格式的方法Java
- Elasticsearch+kibana+logstash 搭建日誌收集分析平臺Elasticsearch
- Kubernetes日誌採集
- Logstash日誌蒐集
- java專案日誌配置檔案Java
- 日誌服務之使用Nginx模式採集日誌Nginx模式
- ELK(ElasticSearch, Logstash, Kibana)搭建實時日誌分析平臺Elasticsearch
- Elasticsearch、Logstash、Kibana搭建統一日誌分析平臺Elasticsearch
- flume日誌採集,hbase資料儲存,hive查詢輸出(簡單整合)Hive
- K8S的日誌採集,沒有我們想的那麼簡單!K8S
- 日誌服務 HarmonyOS NEXT 日誌採集最佳實踐
- ELK太重?試試KFC日誌採集
- KubeSphere 多行日誌採集方案深度探索
- ELK叢集搭建(ElasticSearch Logstash Kinaba)Elasticsearch
- 簡單ELK配置實現生產級別的日誌採集和查詢實踐
- 【Oracle日誌】- 日誌檔案重建Oracle
- 日誌檔案
- Java實時讀取日誌檔案Java
- Oracle使用備份檔案集恢復歸檔日誌Oracle
- 日誌檔案和歸檔日誌檔案的關係以及如何切換日誌
- 2023最新ELK日誌平臺(elasticsearch+logstash+kibana)搭建Elasticsearch
- 使用Logstash工具匯入sqlserver資料到elasticSearch及elk分散式日誌中心SQLServerElasticsearch分散式
- 應用日誌採集是什麼意思?批次採集應用日誌軟體用哪個?怎麼操作?應用日誌
- 刪除日誌檔案組與日誌檔案成員
- PDF轉文字檔案的最簡單方法
- logstash收集springboot日誌Spring Boot
- (四)Logstash收集、解析日誌方法
- logstash kafka output 日誌處理Kafka
- 登錄檔檔案修改開啟程式的簡單示例
- 管理日誌檔案
- dump日誌檔案
- APACHE日誌檔案Apache