[20221103]奇怪的mail資訊(整理版本).txt

--//生產系統伺服器root登入,我發現快下班是出現如下資訊.
--//you have mail,快下班時出現嚴重影響當時的工作心態,抽時間看看到底怎麼回事.

# mail
Heirloom Mail version 12.5 7/5/10. Type ? for help.
"/var/spool/mail/root": 42060 messages 42041 unread
    1 (Cron Daemon)         Mon Nov 30 12:00 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
    2 (Cron Daemon)         Mon Nov 30 12:01 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
    3 (Cron Daemon)         Mon Nov 30 12:02 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
    4 (Cron Daemon)         Mon Nov 30 12:03 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
    5 (Cron Daemon)         Mon Nov 30 12:04 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
    6 (Cron Daemon)         Mon Nov 30 12:05 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
    7 (Cron Daemon)         Mon Nov 30 12:06 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
    8 (Cron Daemon)         Mon Nov 30 12:07 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
    9 (Cron Daemon)         Mon Nov 30 12:08 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
   10 (Cron Daemon)         Mon Nov 30 12:09 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
   11 (Cron Daemon)         Mon Nov 30 12:10 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
   12 (Cron Daemon)         Mon Nov 30 12:11 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
   13 (Cron Daemon)         Mon Nov 30 12:12 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
   14 (Cron Daemon)         Mon Nov 30 12:13 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
   15 (Cron Daemon)         Mon Nov 30 12:14 29/923   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
>U 16 (Cron Daemon)         Mon Nov 30 12:15 29/922   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
U 17 (Cron Daemon)         Mon Nov 30 12:16 29/922   "Cron <root@LIS-DB> /home/del_log/del_arc.sh"
--//每分鐘一次呼叫,正好在12點上下,怪不得正好在這個時間段出現.

& n
Message 16:
From root@LIS-DB.localdomain Mon Nov 30 12:15:01 2020
Return-Path: <root@LIS-DB.localdomain>
X-Original-To: root
Delivered-To: root@LIS-DB.localdomain
From: "(Cron Daemon)" <root@LIS-DB.localdomain>
To: root@LIS-DB.localdomain
Subject: Cron <root@LIS-DB> /home/del_log/del_arc.sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=491>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Date: Mon, 30 Nov 2020 12:15:01 +0800 (CST)
Status: RO

Message file RMAN<lang>.msb not found
Verify that ORACLE_HOME is set properly
--//很明顯,呼叫/home/del_log/del_arc.sh指令碼時以root使用者執行,缺乏一些環境變數.
--//我自己不會使用這個命令mail的介面,n好像是下一封信, n 加上數字好像是看第幾封:
& n 42060
Message 42060:
From root@LIS-DB.localdomain Mon Oct 31 12:59:01 2022
Return-Path: <root@LIS-DB.localdomain>
X-Original-To: root
Delivered-To: root@LIS-DB.localdomain
From: "(Cron Daemon)" <root@LIS-DB.localdomain>
To: root@LIS-DB.localdomain
Subject: Cron <root@LIS-DB> /home/del_log/del_arc.sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=200887>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Date: Mon, 31 Oct 2022 12:59:01 +0800 (CST)
Status: RO

Message file RMAN<lang>.msb not found
Verify that ORACLE_HOME is set properly

& n
At EOF

--//很明顯運維人員在root下建立執行了/home/del_log/del_arc.sh指令碼.而且指令碼還沒有寫在oracle目錄下.真不知道對方怎麼想的.
--//從時間間隔看這個問題一直存在,不知道運維認真測試沒有檢查沒有.
--//最後是為什麼時間間隔是1分鐘做一次(12點).為什麼呼叫這麼頻繁.

# ls -l /home/del_log/del_arc.sh
-rwxrwxrwx. 1 root root 193 2021-04-09 10:48:04 /home/del_log/del_arc.sh
--//owner,group都是root.

# cat /home/del_log/del_arc.sh
source ~/.bash_profile
exec >> /home/del_log/log/del_arch`date +%F-%H`.log
/u01/app/oracle/product/19/db_1/bin/rman target / <<EOF
delete noprompt archivelog until time 'sysdate-10';
exit;
EOF
--//很明顯指令碼一開始時root使用者執行是不行的.另外我檢查發現oracle使用者也有類似的資訊,也是報錯.我已經解決了!!

# cd /etc
# grep -r del_arc.sh *

--//恩!! 沒有相關資訊在/etc目錄下,我個人工作習慣在手工寫crontab內容放在/etc/cron.d/目錄下.

# stat /home/del_log/del_arc.sh
File: '/home/del_log/del_arc.sh'
Size: 193             Blocks: 8          IO Block: 4096   regular file
Device: f901h/63745d    Inode: 104907001   Links: 1
Access: (0777/-rwxrwxrwx) Uid: (    0/    root)   Gid: (    0/    root)
Context: unconfined_u:object_r:home_root_t:s0
Access: 2022-10-31 12:27:01.970203358 +0800
Modify: 2021-04-09 10:48:04.329182000 +0800
Change: 2021-04-09 10:48:04.329182000 +0800
Birth: -

# crontab -l
* 12 * * * /home/del_log/del_arc.sh
--//看了一下crontab的格式,實際上對方寫錯了,它相當於12點的每分鐘執行1次,13點停止執行.相當於每天12點後每分鐘執行1次.共60
--//次.

# strace -e open crontab -l
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libpam.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libaudit.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libcap-ng.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
open("/var/spool/cron/root", O_RDONLY) = 4
* 12 * * * /home/del_log/del_arc.sh
+++ exited with 0 +++
--//哦,執行指令碼放在var/spool/cron/目錄下root檔案,順便說一下我自己很少使用cronatb命令,我一般編輯好指令碼.
--//呼叫放在/etc/crond.d目錄,建立相關crontab檔案.

# ls -l /var/spool/cron
total 8
-rw-------. 1 oracle oinstall 36 2021-04-08 10:14:26 oracle
-rw-------. 1 root   root     36 2020-11-30 11:14:10 root
--//很明顯從時間看對方發現錯誤以後在2021-04-08 另外寫一個oracle使用者執行的cronatb格式的檔案.
--//以前的錯誤沒有清除,明顯工作的責任心太差了.

# cat /var/spool/cron/oracle
* 23 * * * /home/del_log/del_arc.sh
--//修改到23點,還是寫錯了!!相當於每分鐘執行1次.明顯做工作完成後缺乏必要的測試與檢查.
--//如何取消呢?我感覺直接刪除檔案應該是ok的.
# crontab -e --//編輯.
# crontab -l --//顯示.
# ls -l /var/spool/cron/
total 4
-rw-------. 1 oracle oinstall 36 2021-04-08 10:14:26 oracle
-rw-------. 1 root   root      0 2022-11-01 11:24:56 root
--//root檔案大小變成0,刪除它應該也沒有問題.

# rm /var/spool/cron/root
rm: remove regular empty file '/var/spool/cron/root'? y

--//使用V$RMAN_OUTPUT檢視:
SYS@192.168.100.235:1521/orcl> SELECT distinct sid
     , stamp
     , session_stamp
     , rman_status_stamp
FROM V$RMAN_OUTPUT
WHERE lower(output) like '%deleted archived log%'
order by 4;
       SID      STAMP SESSION_STAMP RMAN_STATUS_STAMP
---------- ---------- ------------- -----------------
      4265 1119481204    1119481202        1119481203
      4265 1119481205    1119481202        1119481203
      1140 1119567604    1119567602        1119567603
      1140 1119567605    1119567602        1119567603

SYS@192.168.100.235:1521/orcl> @ stamp 1119481203
     STAMP STAMP_CONV_TIME
---------- -------------------
1119481203 2022-10-30 23:00:03

SYS@192.168.100.235:1521/orcl> @ stamp 1119567603
     STAMP STAMP_CONV_TIME
---------- -------------------
1119567603 2022-10-31 23:00:03

--//視乎輸出僅僅執行1次,出現的時間戳也能對上,後面間隔1分鐘的呼叫,因為沒有archive log的刪除.沒有顯示.
--//注: 主要原因是有一個lower(output) like '%deleted archived log%'條件限制顯示資訊.

SYS@192.168.100.235:1521/orcl> select * from (select distinct stamp fROM V$RMAN_OUTPUT where stamp>=1119481205) where rownum<=10;
     STAMP
----------
1119481205
1119481206
1119481262
1119481263
1119481322
1119481323
1119481382
1119481383
1119481441
1119481442
10 rows selected.

SYS@192.168.100.235:1521/orcl> @ stamp 1119481322
     STAMP STAMP_CONV_TIME
---------- -------------------
1119481322 2022-10-30 23:02:02

SYS@192.168.100.235:1521/orcl> @ stamp 1119481382
     STAMP STAMP_CONV_TIME
---------- -------------------
1119481382 2022-10-30 23:03:02

SYS@192.168.100.235:1521/orcl> @ stamp 1119481441
     STAMP STAMP_CONV_TIME
---------- -------------------
1119481441 2022-10-30 23:04:01

--//相差1分鐘.基本可以確定對方設計的是12點時,分鐘呼叫1次/home/del_log/del_arc.sh執行.共60次
--//可以透過其對應的log檔案確認:
$ grep "Recovery Manager complete" del_arch2022-10-31-23.log|wc
     60     180    1620
--//正好60次!!

--//利用crontab -e編輯檔案,更正其錯誤.
$ crontab -e
$ crontab -l
5 23 * * * /home/del_log/del_arc.sh

--//順便記錄一下oracle使用者執行的錯誤:
$ mail
Heirloom Mail version 12.5 7/5/10. Type ? for help.
"/var/spool/mail/oracle": 2 messages 2 new
>N 1 (Cron Daemon)         Tue Nov 1 23:05 25/924   "Cron <oracle@LIS-DB> /home/del_log/del_arc.sh"
N 2 (Cron Daemon)         Wed Nov 2 23:05 25/924   "Cron <oracle@LIS-DB> /home/del_log/del_arc.sh"
&
Message 1:
From oracle@LIS-DB.localdomain Tue Nov 1 23:05:02 2022
Return-Path: <oracle@LIS-DB.localdomain>
X-Original-To: oracle
Delivered-To: oracle@LIS-DB.localdomain
From: "(Cron Daemon)" <oracle@LIS-DB.localdomain>
To: oracle@LIS-DB.localdomain
Subject: Cron <oracle@LIS-DB> /home/del_log/del_arc.sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=201711>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/502>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/home/oracle>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=oracle>
X-Cron-Env: <USER=oracle>
Date: Tue, 1 Nov 2022 23:05:02 +0800 (CST)
Status: R

/home/oracle/.bashrc: line 14: `==': not a valid identifier

--//在我.bashrc中我定義函式,導致oracle使用者下的執行也報錯,不知道為什麼.我的函式定義如下在/home/oracle/.bashrc檔案中.
== ()
{
    local in="$(echo "$@" | sed -e 's/\[/(/g' -e 's/\]/)/g')";
    echo $in | bc -lq | tr -d '\\\r' | sed -e "s/\.$[0-9]*[1-9]$0\+$/.\1/" -e "s/\.0\+$//"
}

--//就是一個簡易的計算器.
--//理論執行沒有問題的,我手工執行/home/del_log/del_arc.sh ok的,不知道為什麼透過crontab呼叫會報錯.
--//個人不建議source ~/.bash_profile 這樣的呼叫模式,應該直接把對應的環境變數寫入指令碼中.
--//我當前採用的方式是註解該函式,下次將等號換成js 看看,crontab呼叫是否報錯.

--//附上stamp.sql指令碼:
$ cat stamp.sql
SELECT &&1 stamp,to_date(yyyy||'/'||mm||'/'||dd||' '||hh||':'||mi||':'||ss,'yyyy-mm-dd hh24:mi:ss') stamp_conv_time from (
SELECT &&1
        ,FLOOR (&&1 / (86400*31*12))+1988 yyyy
        ,FLOOR (MOD (&&1 / (86400*31),12))+1 mm
        ,FLOOR (MOD (&&1 / 86400, 31))+1 dd
        ,FLOOR (MOD (&&1 / 3600, 24)) hh
        ,FLOOR (MOD (&&1 / 60, 60)) mi
        ,MOD (&&1, 60) ss
        from dual);

[20221103]奇怪的mail資訊(整理版本).txt

相關文章