Linux下rsyslog日誌收集服務環境部署記錄

散盡浮華發表於2016-06-08

 

rsyslog 可以理解為多執行緒增強版的syslog。 在syslog的基礎上擴充套件了很多其他功能,如資料庫支援(MySQL、PostgreSQL、Oracle等)、日誌內容篩選、定義日誌格式模板等。目前大多數Linux發行版預設也是使用rsyslog進行日誌記錄。rsyslog提供了三種遠端傳輸協議:

UDP 傳輸協議 
基於傳統UDP協議進行遠端日誌傳輸,也是傳統syslog使用的傳輸協議; 可靠性比較低,但效能損耗最少, 在網路情況比較差, 或者接收伺服器壓力比較高情況下,
可能存在丟日誌情況。 在對日誌完整性要求不是很高,在可靠的區域網環境下可以使用。

TCP 傳輸協議 
基於傳統TCP協議明文傳輸,需要回傳進行確認,可靠性比較高; 但在接收伺服器當機或者兩者之間網路出問題的情況下,會出現丟日誌情況。 這種協議相比於UDP在
可靠性方面已經好很多,並且rsyslog原生支援,配置簡單, 同時針對可能丟日誌情況,可以進行額外配置提高可靠性,因此使用比較廣。

RELP 傳輸協議 
RELP(Reliable Event Logging Protocol)是基於TCP封裝的可靠日誌訊息傳輸協議; 是為了解決TCP 與 UDP 協議的缺點而在應用層實現的傳輸協議,也是三者
之中最可靠的。 需要多安裝一個包rsyslog-relp以支援該協議。

對於線上伺服器,為了日誌安全起見,建議使用還是使用 RELP 協議進行傳輸。 

rsyslog的簡單配置記錄(如下將公司防火牆上的日誌(UDP)打到IDC的rsyslog日誌伺服器上)

一、rsyslog服務端的部署
安裝rsyslog 程式(rsyslog預設已經在各發行版安裝,如果系統中沒有的話,可以用yum 進行安裝,如下:)
[root@zabbix ~]# yum install rsyslog -y

配置:
[root@zabbix ~]# cat /etc/rsyslog.conf
# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp                                          #開啟udp的514埠。也可以開啟tcp的514埠,這裡只接受udp的
$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

$WorkDirectory /var/lib/rsyslog
$AllowedSender udp, 192.168.17.0/8                    #僅僅接收來自192.168.17.0/8網段的主機的udp日誌(這個是公司防火牆的ip地址)
#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$template Remote,"/data/fw_logs/%fromhost-ip%/%fromhost-ip%_%$YEAR%-%$MONTH%-%$DAY%.log"           #定義模板,接受日誌檔案路徑,區分了不同主機的日誌
:fromhost-ip, !isequal, "127.0.0.1" ?Remote                                                        # 過濾server 本機的日誌
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog
local4.*                                                /data/fw.log

# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###


[root@zabbix ~]# mkdir /data/fw_logs/

[root@zabbix ~]# /etc/init.d/rsyslog restart


二、在公司防火牆(192.168.17.41/42)上配置udp日誌輸出策略(在防火牆新增rsyslog服務端的ip和514埠)

三、過一會兒,在rsyslog日誌伺服器上設定的日誌目錄下就能看到防火牆的日誌輸出了
[root@zabbix ~]# ll /data/fw_logs/
total 4.0K
drwxrwxrwx   4 root root   46 Jul 28 10:40 .
drwxr-xr-x. 18 root root 4.0K Jul 28 10:38 ..
drwx------   2 root root   41 Jul 28 10:37 192.168.17.41
drwx------   2 root root   41 Jul 28 10:40 192.168.17.42
[root@zabbix ~]# ll /data/fw_logs/192.168.17.41
total 16K
drwx------ 2 root root  41 Jul 28 10:37 .
drwxrwxrwx 4 root root  46 Jul 28 10:40 ..
-rw------- 1 root root 13K Jul 28 14:02 192.168.17.41_2017-07-28.log


------------------------------------------------------------------------------------
可以將上面rsyslog服務端的rsyslog.conf裡的ip白名單設定為客戶機的ip端,比如:
$AllowedSender tcp, 172.18.0.0/16                  #表示接收172.18.0.0/16網段的客戶機的tcp日誌輸入,前提是開啟tcp的514埠

客戶機的配置:
只需要在rsyslog.conf檔案裡新增下面一行:
*.*                               @172.18.10.20                     #後面的ip是rsyslog服務端的ip地址

啟動rsyslog日誌即可!

====================再看一例=======================
以上配置的是將公司防火牆的日誌打到rsyslog裡。現在有這麼一個需求:
公司IDC的另外兩臺伺服器172.19.10.24和172.19.10.25上部署了gitlab、nexus、jenkins、jira和wiki,上面的許可權設定的比較雜,很多人都有登入需求。現在需要將登入到這兩臺伺服器上的使用者的所有操作過程記錄下來,記錄達到rsyslog日誌裡,相當於做使用者操作記錄的審計工作。

配置如下(結合上面的安裝配置)(服務端的ip是172.19.16.21):
1)rsyslog服務端配置  (相比於上面的配置,這裡去掉了AllowedSender的來源ip的白名單限制。即允許接收所有機器的日誌;上面的防火牆日誌還是能繼續收集)
[root@zabbix ~]# cat /etc/rsyslog.conf|grep -v "#"|grep -v "^$"
$ModLoad imudp
$UDPServerRun 514
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$template Remote,"/data/fw_logs/%fromhost-ip%/%fromhost-ip%_%$YEAR%-%$MONTH%-%$DAY%.log"
:fromhost-ip, !isequal, "127.0.0.1" ?Remote
$IncludeConfig /etc/rsyslog.d/*.conf
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 *
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
local5.*                                              /var/log/history.log

[root@zabbix ~]# /etc/init.d/rsyslog restart

2)在172.19.10.24上的配置
[root@gitlab ~]# cat /etc/rsyslog.conf|grep -v "#"|grep -v "^$"
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 *
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
local5.*    @172.19.16.21

[root@gitlab ~]# /etc/init.d/rsyslog restart

[root@gitlab ~]# cat /etc/profile                  #在該檔案的底部新增下面內容
.......
export HISTTIMEFORMAT
export PROMPT_COMMAND='{ command=$(history 1 | { read x y; echo $y; }); logger -p local5.notice -t bash -i "user=$USER,ppid=$PPID,from=$SSH_CLIENT,pwd=$PWD,command:$command"; }'

3)在另一臺172.19.10.25上做類似配置配置
[root@nexus ~]# cat /etc/rsyslog.conf |grep -v "#"|grep -v "^$"
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 *
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
local5.*   @172.19.16.21 

[root@nexus ~]# /etc/init.d/rsyslog restart

[root@nexus ~]# cat /etc/profile
.......
export HISTTIMEFORMAT
export PROMPT_COMMAND='{ command=$(history 1 | { read x y; echo $y; }); logger -p local5.notice -t bash -i "user=$USER,ppid=$PPID,from=$SSH_CLIENT,pwd=$PWD,command:$command"; }'

4)過一段時間,發現在rsyslog服務端的日誌目錄/data/fw_logs下面已經有收集到的日誌了
[root@zabbix fw_logs]# pwd
/data/fw_logs
[root@zabbix fw_logs]# cd
[root@zabbix ~]# cd /data/fw_logs/
[root@zabbix fw_logs]# ll
total 12K
drwxrwxrwx   6 root root   84 Aug 16 18:28 .
drwxr-xr-x. 18 root root 4.0K Aug 16 17:58 ..
drwx------   2 root root   74 Aug 17 09:50 172.19.10.24
drwx------   2 root root   74 Aug 17 10:00 172.19.10.25
drwx------   2 root root 4.0K Aug 17 00:01 192.168.17.41
drwx------   2 root root 4.0K Aug 17 00:01 192.168.17.42
[root@zabbix fw_logs]# cd 172.19.10.24/
[root@zabbix 172.19.10.24]# ll
total 20K
drwx------ 2 root root  74 Aug 17 09:50 .
drwxrwxrwx 6 root root  84 Aug 16 18:28 ..
-rw------- 1 root root 14K Aug 16 20:45 172.19.10.24_2017-08-16.log
-rw------- 1 root root 771 Aug 17 10:03 172.19.10.24_2017-08-17.log
[root@zabbix 172.19.10.24]# cat 172.19.10.24_2017-08-16.log
Aug 16 18:39:56 gitlab bash[138413]: user=root,ppid=124297,from=172.19.16.28 29338 22,pwd=/root,command:[2017-08-16 18:39:56]root pts/5 2017-08-16 17:23 (172.19.16.28)/etc/init.d/rsyslog restart
Aug 16 18:39:56 gitlab bash[138418]: user=root,ppid=124297,from=172.19.16.28 29338 22,pwd=/root,command:[2017-08-16 18:39:56]root pts/5 2017-08-16 17:23 (172.19.16.28)/etc/init.d/rsyslog restart
Aug 16 18:39:56 gitlab bash[138422]: user=root,ppid=124297,from=172.19.16.28 29338 22,pwd=/root,command:[2017-08-16 18:39:56]root pts/5 2017-08-16 17:23 (172.19.16.28)/etc/init.d/rsyslog restart
Aug 16 18:39:57 gitlab bash[138426]: user=root,ppid=124297,from=172.19.16.28 29338 22,pwd=/root,command:[2017-08-16 18:39:56]root pts/5 2017-08-16 17:23 (172.19.16.28)/etc/init.d/rsyslog restart
Aug 16 18:40:30 gitlab bash[138610]: user=root,ppid=138586,from=172.16.255.202 52496 22,pwd=/root,command:[2017-08-16 18:40:03]root pts/0 2017-08-16 18:40 (172.16.255.202)exit
Aug 16 18:40:43 gitlab bash[138652]: user=root,ppid=138586,from=172.16.255.202 52496 22,pwd=/data,command:[2017-08-16 18:40:43]root pts/0 2017-08-16 18:40 (172.16.255.202)cd /data/
Aug 16 18:40:43 gitlab bash[138657]: user=root,ppid=138586,from=172.16.255.202 52496 22,pwd=/data,command:[2017-08-16 18:40:43]root pts/0 2017-08-16 18:40 (172.16.255.202)ls
Aug 16 18:40:47 gitlab bash[138666]: user=root,ppid=138586,from=172.16.255.202 52496 22,pwd=/data,command:[2017-08-16 18:40:47]root pts/0 2017-08-16 18:40 (172.16.255.202)mkdir hahahahah
Aug 16 18:40:48 gitlab bash[138671]: user=root,ppid=138586,from=172.16.255.202 52496 22,pwd=/data/hahahahah,command:[2017-08-16 18:40:48]root pts/0 2017-08-16 18:40 (172.16.255.202)cd hahahahah/
Aug 16 18:40:48 gitlab bash[138677]: user=root,ppid=138586,from=172.16.255.202 52496 22,pwd=/data/hahahahah,command:[2017-08-16 18:40:48]root pts/0 2017-08-16 18:40 (172.16.255.202)ls
Aug 16 18:40:54 gitlab bash[138696]: user=root,ppid=138586,from=172.16.255.202 52496 22,pwd=/data/hahahahah,command:[2017-08-16 18:40:54]root pts/0 2017-08-16 18:40 (172.16.255.202)echo "Asdfasdf" >heihei
Aug 16 18:40:54 gitlab bash[138702]: user=root,ppid=138586,from=172.16.255.202 52496 22,pwd=/data/hahahahah,command:[2017-08-16 18:40:54]root pts/0 2017-08-16 18:40 (172.16.255.202)ls
.......

有上面日誌可以看出,在172.19.10.24這臺機器上的操作記錄都被詳細記錄下來了。這樣,就能清楚地知道登入到這臺機器上的使用者都做了些什麼了.......

=====================通過rsyslog收集nginx日誌到遠端伺服器上====================
需求說明:通過rsyslog服務將192.168.10.21伺服器上的/data/nginx/logs/www.kevin.com-access.log日誌實時同步到192.168.10.52伺服器上(路徑為/data/rsyslog/nginx)

1)192.168.10.21為rsyslog客戶端,即日誌的推送端rsyslog日誌是客戶機主動將自己的日誌推送到遠端伺服器上
操作如下:
[root@nginx-server ~]# yum install rsyslog -y
[root@nginx-server ~]# cp /etc/rsyslog.conf /etc/rsyslog.conf.bak
[root@nginx-server ~]# cat /etc/rsyslog.conf
# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability
$ModLoad imfile                               ##裝載imfile模組,這一行手動新增

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;local5.none /var/log/messages             ##不記錄local5的日誌

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog


# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
user.info /var/log/history

#在檔案底部新增下面幾行內容
$InputFileName /data/nginx/logs/www.kevin.com-access.log        ##讀取日誌檔案(要監控的日誌檔案)
$InputFileTag web_access             ##日誌寫入日誌附加標籤字串
$InputFileSeverity info           ##日誌等級
$InputFileStateFile /etc/rsyslog.d/stat-access         ##記錄日誌點等資訊。(相當於msyql的master.info)檔名變了,
這個StateFile標誌必須變,否則無法傳輸。
$InputFileFacility local5         ##設施類別
$InputFilePollInterval 1          ##檢查日誌檔案間隔(秒)
$InputFilePersistStateInterval 1       ##回寫偏移量資料到檔案間隔時間(秒)
$InputRunFileMonitor                          ##啟用讀取,可以設定多組日誌讀取,每組結束時設定本引數。以示生效。
local5.* @192.168.10.52            ##代表local5設施的所有級別通過udp協議傳送到192.168.10.51

重啟rsyslog服務
[root@nginx-server ~]# /etc/init.d/rsyslog restart
關閉系統日誌記錄器: [確定]
啟動系統日誌記錄器: [確定]

由於作為日誌的推送端,rsyslog日誌不需要開啟514埠(如上在rsyslog.conf檔案裡沒有開啟dup或tcp的514埠)
[root@nginx-server ~]# lsof -i:514
[root@nginx-server ~]#

2)192.168.10.52為rsyslog服務端,即日誌的接收端。
配置如下:
[root@log-server ~]# yum install rsyslog -y
[root@log-server ~]# cp /etc/rsyslog.conf /etc/rsyslog.conf.bak
# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp                   ##載入imudp模組
$UDPServerRun 514            ##開啟udp接收並制定埠號

# Provides TCP syslog reception
$ModLoad imtcp                 ##載入imtcp模組。
$InputTCPServerRun 514             ##開啟tcp接收並制定埠號。tcp和udp兩個埠模組可以同時使用!

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

#定義一個模板用來指定接收的日誌訊息的格式(預設會在記錄的日誌前加幾個欄位)
$template  SpiceTmpl,"%msg%\n"                   ##%msg:2:$%為去掉日誌開頭的空格

#定義一個模板用來指定接收的日誌檔案的存放路徑%……%之間的是定義日誌按照年-月-日命名
$template  DynaFile,"/data/rsyslog/nginx/%$YEAR%-%$MONTH%-%$DAY%.log"

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;local5.none                /var/log/messages            ##不記錄local5設施的日誌

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog


# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

#接收客戶端local5設施傳送來的日誌並存放到指定位置(位置可用定義的模板。?代表使用動態的模板)
local5.*                       ?DynaFile;SpiceTmpl

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

編輯/etc/sysconfig/rsyslog中"SYSLOGD_OPTIONS="開啟遠端日誌接收功能
[root@log-server ~]# cat /etc/sysconfig/rsyslog
# Options for rsyslogd
# Syslogd options are deprecated since rsyslog v3.
# If you want to use them, switch to compatibility mode 2 by "-c 2"
# See rsyslogd(8) for more details
SYSLOGD_OPTIONS="-c 5"

建立日誌接收過來後定義的存放目錄
[root@log-server ~]# mkdir -p /data/rsyslog/nginx

重啟rsyslog服務
[root@log-server ~]# /etc/init.d/rsyslog restart
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
[root@log-server ~]# lsof -i:514
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 24594 root 2u IPv4 38927639 0t0 TCP *:shell (LISTEN)
rsyslogd 24594 root 3u IPv4 38927635 0t0 UDP *:syslog
rsyslogd 24594 root 4u IPv6 38927636 0t0 UDP *:syslog
rsyslogd 24594 root 5u IPv6 38927640 0t0 TCP *:shell (LISTEN)

檢視日誌是否接收過來了
[root@log-server ~]# ll /data/rsyslog/nginx/
total 550876
-rw------- 1 root root 483539594 Jun 13 12:58 2018-06-13.log
[root@log-server ~]# tail -2 /data/rsyslog/nginx/2018-06-13.log
1.203.163.198 - [27/Apr/2018:00:17:53 +0800] "POST /scf/%7B%7BloginConfig.loginSubmitUrl%7D%7D HTTP/1.1" 302 0 "https://www.kevin.com/scf/login" Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.62 Safari/537.36 - 0.010 0.003 10.0.54.21:9020 302
1.203.163.198 - [27/Apr/2018:00:17:53 +0800] "POST /scf/%7B%7BloginConfig.loginSubmitUrl%7D%7D HTTP/1.1" 302 0 "https://www.kevin.com/scf/login" Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.62 Safari/537.36 - 0.012 0.003 10.0.54.21:9020 302

=========================溫馨提示========================
rsyslog也可以收集多個日誌檔案,需要注意的是:
$InputFileTag        定義的APPNAME必須唯一,同一臺主機上不同的應用應當使用不同的APPNAME,否則會導致新定義的TOKEN和TAG不生效;
$template         定義的模板名必須唯一,否則會導致新定義的TOKEN和TAG不生效;
$InputFileStateFile       定義的StateFile必須唯一,它被rsyslog用於記錄檔案上傳進度,否則會導致混亂;

如下是rsyslog收集多個日誌的配置,這裡以2個日誌檔案為例:

日誌的推送端配置

[root@external-lb01 ~]# cat /etc/rsyslog.conf
..........
$ModLoad imfile

.........
*.info;mail.none;authpriv.none;cron.none;local5.none;local4.none                /var/log/messages

.........

$InputFileName /data/nginx/logs/portal.kevin.com-access.log
$InputFileTag portal_access
$InputFileSeverity info
$InputFileStateFile /etc/rsyslog.d/stat1-access
$InputFileFacility local4
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor 
local4.*  @192.168.10.52

$InputFileName /data/nginx/logs/www.kevin.com-access.log
$InputFileTag web_access
$InputFileSeverity info
$InputFileStateFile /etc/rsyslog.d/stat-access
$InputFileFacility local5
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
local5.*  @192.168.10.52

重啟日誌傳送端的rsyslog服務
[root@external-lb01 ~]# /etc/init.d/rsyslog restart

日誌的接收端配置

[root@open-falcon01 ~]# cat /etc/rsyslog.conf
........
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

.........
$template SpiceTmpl,"%msg%\n" 
$template DynaFile,"/data/external-lb/nginx/nginx-access.log"

$template SpiceTmpl2,"%msg%\n" 
$template DynaFile2,"/data/external-lb/portal/portal-access.log"

.........
*.info;mail.none;authpriv.none;cron.none;local5.none;local4.none                /var/log/messages

.........
local5.*                                                ?DynaFile;SpiceTmpl
local4.*                                                ?DynaFile2;SpiceTmpl2

重啟日誌接收端的rsyslog服務
[root@open-falcon01 ~]# /etc/init.d/rsyslog restart

檢視,當訪問對應對應的url時,就會有轉發後的檔案產生,並實時有日誌內容轉發過來
[root@open-falcon01 ~]# ll /data/external-lb/nginx/nginx-access.log
-rw------- 1 root root 1067372 Oct  9 10:51 /data/external-lb/nginx/nginx-access.log
[root@open-falcon01 ~]# ll /data/external-lb/portal/portal-access.log 
-rw------- 1 root root 88141 Oct  9 22:26 /data/external-lb/portal/portal-access.log

==========================================================================
注意:
a)如果發現日誌還沒有接收過來,即/data/rsyslog/nginx目錄下沒有日誌產生,就同時重啟推送端和接收端的rsyslog服務。確保雙方的iptables防火牆和selinux關閉!
b)也可以自行修改接收的日誌檔案的存放路徑,如改為下面的配置:
$template DynaFile,"/data/rsyslog/nginx/nginx-access.log"
則日誌收集後存放的檔案如下:
[root@log-server ~]# ll /data/rsyslog/nginx/
total 571716
-rw------- 1 root root 483539594 Jun 13 12:58 2018-06-13.log
-rw------- 1 root root 101893593 Jun 13 13:13 nginx-access.log

相關文章