Logging with ElasticSearch, Kibana, ASP.NET Core and Docker

有態度的小碼甲發表於2020-07-20
ElasticsearchASP.NETDocker

好久不見，前兩週經歷了人生第一次"偽牛市"，基金和股市大起大落，更加堅信“你永遠賺不到超出你認知範圍之外的錢,除非靠著運氣”，老韭菜誠不欺我也。
當能力與野心不匹配，只能多看書，收割那些不求甚解的韭菜。

言歸正傳，本文聊一聊在ASP.NET Core3.1中整合ElasticSearch、Kibana日誌查詢系統。

我們為什麼需要ElasticSearch、Kibana？

成熟的應用程式，會在各個階段以各種姿勢記錄日誌，這些日誌包括(但不限於)nginx日誌、pod/Container、業務日誌。
ElasticSearch是一個高度可擴充套件的全文搜尋和分析引擎，使任何型別的日誌記錄變得容易，可訪問和可搜尋。ElasticSearch令人難以置信的速度和簡單的查詢語言，再加上Kibana的介面和圖形，構成了功能強大的日誌儲存和查詢搭檔。

1.在Docker中啟動ElasticSearch、Kibana

docker中啟動Elastcisearch、kibana最簡單的方式是建立docker-compose檔案，將原服務偵聽埠對映到宿主機埠。

version: '3.1'

services:

  elasticsearch:
   container_name: elasticsearch
   image: elasticsearch:7.6.2
   ports:
    - 9200:9200
   volumes:
    - elasticsearch-data:/usr/share/elasticsearch/data
   environment:
    - xpack.monitoring.enabled=true
    - xpack.watcher.enabled=false
    - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    - discovery.type=single-node
   networks:
    - elastic

  kibana:
   container_name: kibana
   image: kibana:7.6.2
   ports:
    - 5601:5601
   depends_on:
    - elasticsearch
   environment:
    - ELASTICSEARCH_URL=http://localhost:9200
   networks:
    - elastic
  
networks:
  elastic:
    driver: bridge

volumes:
  elasticsearch-data:

在後臺拉取映象，啟動容器：

docker-compose  up -d

確認ElasticSearch,kibana啟動成功：

2. 使用ASP.NET Core和NLog向ES傳送日誌

VS Code建立.NetCore程式

dotnet new mvc --no-https -o Elastic.Kibana.NLog
cd Elastic.Kibana.NLog

新增NLog依賴包

dotnet add package NLog
dotnet add package NLog.Web.AspNetCore
dotnet add package NLog.Targets.ElasticSearch

新增nlog.config檔案

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true"
      internalLogLevel="Info"
      internalLogFile="internal-nlog.txt">

  <!-- enable asp.net core layout renderers -->
  <extensions>
    <add assembly="NLog.Web.AspNetCore"/>
  </extensions>

  <!-- the targets to write to -->
  <targets>
    <!-- write logs to file  -->
    <target xsi:type="File" name="allfile" fileName="/home/root/nlog-all-${shortdate}.log"
            layout="${longdate}|${event-properties:item=EventId_Id}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}" />

    <!-- another file log, only own logs. Uses some ASP.NET core renderers -->
    <target xsi:type="File" name="ownFile-web" fileName="/home/root/nlog-own-${shortdate}.log"
            layout="${longdate}|${event-properties:item=EventId_Id}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}|url: ${aspnet-request-url}|action: ${aspnet-mvc-action}" />

    <target name="elastic" xsi:type="BufferingWrapper" flushTimeout="5000">
      <target xsi:type="ElasticSearch" uri="http://localhost:9200/" />
    </target>

  </targets>

  <!-- rules to map from logger name to target -->
  <rules>
    <!--All logs, including from Microsoft-->
    <logger name="*" minlevel="Trace" writeTo="allfile" />

    <!--Skip non-critical Microsoft logs and so log only own logs-->
    <logger name="Microsoft.*" maxlevel="Info" final="true" />
    <!-- BlackHole without writeTo -->
    <logger name="*" minlevel="Trace" writeTo="ownFile-web" />

    <logger name="*" minlevel="Info" writeTo="elastic" />
  </rules>
</nlog>

ASP.NET Core新增Nlog並輸出日誌

    public class Program
    {
        public static void Main(string[] args)
        {
            var webHost = WebHost.CreateDefaultBuilder(args)
                     .ConfigureLogging((hostingContext, loggingBuilder) =>
                     {
                         loggingBuilder.AddConsole(x => x.IncludeScopes = true).AddDebug();
                     })
                     .UseNLog()
                     .UseStartup<Startup>()
                     .Build();
            webHost.Run();
        }
    }

Controller自定義日誌：

public class HomeController : Controller
    {
        private readonly ILogger<HomeController> _logger;

        public HomeController(ILogger<HomeController> logger)
        {
            _logger = logger;
        }

        public IActionResult Index()
        {

           _logger.LogInformation("HomeController Index executed at {date}", DateTime.UtcNow);
            return View();
        }

        public IActionResult Privacy()
        {
            return View();
        }

        [ResponseCache(Duration = 0, Location = ResponseCacheLocation.None, NoStore = true)]
        public IActionResult Error()
        {
            return View(new ErrorViewModel { RequestId = Activity.Current?.Id ?? HttpContext.TraceIdentifier });
        }
    }

3.在kibana中建立Index Pattern

dotnet run啟動ASP.NET Core程式，現在開始產生日誌。
Kibana使用index pattern從ElasticSearch indices
中獲取資料。要在Kibana中顯示日誌，需要先定義index pattern:

之後選擇@timestamp時間過濾。

回到discovery介面，可以看到所有日誌。

ok,上面便於ASP.NETCOre+NLog整合ES、Kibana的基本操作流程，
看起來很簡單，其實只是九牛一毛，Elastic Stack的內功博大精深，關注我，解鎖各種姿勢。

相關文章