用strace跟蹤分析oracle 10.2.0.1 rac lmd程式系列二

結論：

1,strace -p跟蹤lmd程式，是和fd為9及17的2個socket進行通訊
2,這2個socket為
    9為socket:[1361566]，而17為socket:[1361581]
3，strace的應用關鍵在於有針性性理解作業系統的函式，比如times,getrusage,poll的含義是什麼，你就明白了lmd程式到底在作什麼
4, find .|xargs grep -ri "2015-11-10 17:30:32.608" -l獲取當前目錄是哪個檔案字串2015-11-10 17:30:32.608，僅列印檔名稱出來
   或者透過strace輸出的write函式的write函式也可定位到是寫入到lmd trc file

測試

1,獲取lmd程式
[oracle@jingfa1 ~]$ ps -ef|grep lmd
oracle    4774     1  0 Nov09 ?        00:01:19 asm_lmd0_+ASM1
oracle   15271     1  0 07:40 ?        00:00:42 ora_lmd0_jingfa1
oracle   17639 17535  0 17:06 pts/3    00:00:00 grep lmd




2，用strace跟蹤lmd程式
[oracle@jingfa1 ~]$ strace -p 15271
Process 15271 attached - interrupt to quit
times({tms_utime=3783, tms_stime=445, tms_cutime=0, tms_cstime=0}) = 436476294
getrusage(RUSAGE_SELF, {ru_utime={37, 830248}, ru_stime={4, 459322}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={37, 830248}, ru_stime={4, 459322}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={37, 830248}, ru_stime={4, 459322}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={37, 830248}, ru_stime={4, 459322}, ...}) = 0
times({tms_utime=3783, tms_stime=445, tms_cutime=0, tms_cstime=0}) = 436476294
poll([{fd=9, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}, {fd=17, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 2, 80) = 0 (Timeout) --fd為9及17，會一直相同的迴圈使用poll函式
times({tms_utime=3783, tms_stime=445, tms_cutime=0, tms_cstime=0}) = 436476302
getrusage(RUSAGE_SELF, {ru_utime={37, 830248}, ru_stime={4, 459322}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={37, 830248}, ru_stime={4, 459322}, ...}) = 0
times({tms_utime=3783, tms_stime=445, tms_cutime=0, tms_cstime=0}) = 436476302
poll([{fd=9, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}, {fd=17, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 2, 80) = 0 (Timeout)


3，到對應目錄查查上述的fd到底是什麼？
可見9為socket:[1361566]，而17為socket:[1361581]
[root@jingfa1 fd]# pwd
/proc/15271/fd
[root@jingfa1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Nov 10 17:00 0 -> /dev/null
lr-x------ 1 oracle oinstall 64 Nov 10 17:00 1 -> /dev/null
lrwx------ 1 oracle oinstall 64 Nov 10 17:00 10 -> /u01/app/oracle/product/10.2.0/db_1/dbs/lkinstjingfa1 (deleted)
lr-x------ 1 oracle oinstall 64 Nov 10 17:00 11 -> /dev/zero
lrwx------ 1 oracle oinstall 64 Nov 10 17:00 12 -> /u01/app/oracle/admin/jingfa/adump/ora_15228.aud
lr-x------ 1 oracle oinstall 64 Nov 10 17:00 13 -> /u01/app/oracle/product/10.2.0/db_1/rdbms/mesg/oraus.msb
lr-x------ 1 oracle oinstall 64 Nov 10 17:00 14 -> /dev/zero
lrwx------ 1 oracle oinstall 64 Nov 10 17:00 15 -> /u01/app/oracle/product/10.2.0/db_1/dbs/hc_jingfa1.dat
lrwx------ 1 oracle oinstall 64 Nov 10 17:00 16 -> socket:[1361569]
lrwx------ 1 oracle oinstall 64 Nov 10 17:00 17 -> socket:[1361581]
lr-x------ 1 oracle oinstall 64 Nov 10 17:00 18 -> /u01/app/oracle/product/10.2.0/db_1/rdbms/mesg/oraus.msb
l-wx------ 1 oracle oinstall 64 Nov 10 17:00 2 -> /u01/app/oracle/admin/jingfa/bdump/jingfa1_lmd0_15271.trc
lr-x------ 1 oracle oinstall 64 Nov 10 17:00 3 -> /dev/null
lr-x------ 1 oracle oinstall 64 Nov 10 17:00 4 -> /dev/null
l-wx------ 1 oracle oinstall 64 Nov 10 17:00 5 -> /u01/app/oracle/admin/jingfa/udump/jingfa1_ora_15228.trc
l-wx------ 1 oracle oinstall 64 Nov 10 17:00 6 -> /u01/app/oracle/admin/jingfa/bdump/alert_jingfa1.log
lrwx------ 1 oracle oinstall 64 Nov 10 17:00 7 -> /u01/app/oracle/product/10.2.0/db_1/dbs/hc_jingfa1.dat
l-wx------ 1 oracle oinstall 64 Nov 10 17:00 8 -> /u01/app/oracle/admin/jingfa/bdump/alert_jingfa1.log
lrwx------ 1 oracle oinstall 64 Nov 10 17:00 9 -> socket:[1361566]
[root@jingfa1 fd]# 




可見如果lmd程式暫停，內部機制如下，提示資源暫不可用
SQL> oradebug setospid 15271
Oracle pid: 6, Unix process pid: 15271, image: oracle@jingfa1 (LMD0)
SQL> oradebug suspend
Statement processed.


semtimedop(1277953, 0x7fff7d684890, 1, {1, 0}) = -1 EAGAIN (Resource temporarily unavailable)
semtimedop(1277953, 0x7fff7d684890, 1, {1, 0}) = -1 EAGAIN (Resource temporarily unavailable)
semtimedop(1277953, 0x7fff7d684890, 1, {1, 0}) = -1 EAGAIN (Resource temporarily unavailable)
semtimedop(1277953, 0x7fff7d684890, 1, {1, 0}) = -1 EAGAIN (Resource temporarily unavailable)
semtimedop(1277953, 0x7fff7d684890, 1, {1, 0}) = -1 EAGAIN (Resource temporarily unavailable)
semtimedop(1277953, 0x7fff7d684890, 1, {1, 0}) = -1 EAGAIN (Resource temporarily unavailable)
semtimedop(1277953, 0x7fff7d684890, 1, {1, 0}) = -1 EAGAIN (Resource temporarily unavailable)


如果恢復LMD，資源又恢復正常，又開始了各種作業系統函式的呼叫執行
SQL> oradebug resume
Statement processed.


getrusage(RUSAGE_SELF, {ru_utime={38, 665122}, ru_stime={4, 631295}, ...}) = 0
times({tms_utime=3866, tms_stime=463, tms_cutime=0, tms_cstime=0}) = 436558478
poll([{fd=9, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}, {fd=17, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 2, 80) = 0 (Timeout)
times({tms_utime=3866, tms_stime=463, tms_cutime=0, tms_cstime=0}) = 436558486
getrusage(RUSAGE_SELF, {ru_utime={38, 665122}, ru_stime={4, 631295}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={38, 665122}, ru_stime={4, 631295}, ...}) = 0
times({tms_utime=3866, tms_stime=463, tms_cutime=0, tms_cstime=0}) = 436558486
poll([{fd=9, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}, {fd=17, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 2, 80) = 0 (Timeout)
times({tms_utime=3866, tms_stime=463, tms_cutime=0, tms_cstime=0}) = 436558494
getrusage(RUSAGE_SELF, {ru_utime={38, 665122}, ru_stime={4, 631295}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={38, 665122}, ru_stime={4, 632295}, ...}) = 0


而且lmd程式狀態變化時，會向其trace file寫入資訊


write(2, "*** 2015-11-10 17:30:32.608", 27) = 27  ---write函式含義：write(fb,buf,size)
write(2, "\n", 1)                       = 1
write(2, "SKGXPIWAIT: keepalive_reset elap"..., 86) = 86
write(2, "\n", 1)                       = 1


[root@jingfa1 fd]# locate "2015-11-10 17:30:32.608"
[oracle@jingfa1 bdump]$ pwd
/u01/app/oracle/admin/jingfa/bdump


[oracle@jingfa1 bdump]$ find .|xargs grep -ri "2015-11-10 17:30:32.608" -l
./jingfa1_lmd0_15271.trc
./jingfa1_lmd0_15271.trc






[oracle@jingfa1 bdump]$ more *|grep "2015-11-10 17:30:32.608"
*** 2015-11-10 17:30:32.608


或者透過write函式的write函式也可定位到是寫入到lmd trc file
[oracle@jingfa1 fd]$ pwd
/proc/15271/fd
[oracle@jingfa1 fd]$ ll -l 2
l-wx------ 1 oracle oinstall 64 Nov 10 17:00 2 -> /u01/app/oracle/admin/jingfa/bdump/jingfa1_lmd0_15271.trc
[oracle@jingfa1 fd]$