slp_srvr top程式，cluster.log大量日誌
一個資料庫主機/var目錄很高
[biprd1]root#df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 0.50 0.26 49% 10920 15% /
/dev/hd2 7.00 3.17 55% 56403 8% /usr
/dev/hd9var 5.00 0.31 94% 7284 8% /var
/dev/hd3 1.50 0.88 42% 20596 9% /tmp
/dev/hd1 0.12 0.07 42% 187 2% /home
/dev/hd11admin 0.12 0.12 1% 5 1% /admin
/proc - - - - - /proc
/dev/hd10opt 6.00 3.09 49% 22598 4% /opt
/dev/livedump 0.25 0.25 1% 5 1% /var/adm/ras/livedump
/dev/u01_fs 30.00 11.93 61% 235400 8% /u01
/dev/tsmlog 0.25 0.25 1% 9 1% /tsmlog
/dev/fslv00 60.00 44.19 27% 12 1% /bibak
/dev/nmonlv 0.50 0.31 38% 39 1% /nmon
/dev/backup_fs 40.00 13.00 68% 76 1% /backup
/dev/u02_fs 409.00 167.11 60% 398 1% /u02
top程式主要是兩個，cpu IDLE 7%：
Topas Monitor for host: biprd1 EVENTS/QUEUES FILE/TTY
Tue Mar 26 15:29:49 2019 Interval: 2 Cswitch 912 Readch 1750.2K
Syscall 494.2K Writech 2225.9K
CPU User% Kern% Wait% Idle% Physc Entc Reads 481 Rawin 0
ALL 45.4 47.7 0.0 6.8 0.50 99.8 Writes 80508 Ttyout 576
Forks 4 Igets 0
Network KBPS I-Pack O-Pack KB-In KB-Out Execs 4 Namei 14864
Total 2.5 20.0 5.6 1.4 1.2 Runqueue 7.2 Dirblk 0
Waitqueue 0.0
Disk Busy% KBPS TPS KB-Read KB-Writ MEMORY
Total 0.4 1024.0 8.0 0.0 1024.0 PAGING Real,MB 12544
Faults 2558 % Comp 77
FileSystem KBPS TPS KB-Read KB-Writ Steals 0 % Noncomp 5
Total 2.7K 8.3K 1.7K 1.0K PgspIn 0 % Client 5
PgspOut 0
Name PID CPU% PgSp Owner PageIn 0 PAGING SPACE
slp_srvr 3735686 50.0 128.6 root PageOut 256 Size,MB 8192
syslogd 4718742 50.0 5.4 root Sios 256 % Used 13
topas 29294718 0.0 4.6 root % Free 87
sleep 31719550 0.0 0.2 root NFS (calls/sec)
sshd 36831360 0.0 4.1 root SerV2 0 WPAR Activ 0
oracle 19136702 0.0 15.7 oracle CliV2 0 WPAR Total 0
oracle 44433504 0.0 13.7 oracle SerV3 0 Press: "h"-help
oracle 27721982 0.0 13.7 oracle CliV3 0 "q"-quit
oracle 14745714 0.0 13.9 oracle
vmmd 458766 0.0 1.2 root
oracle 10682470 0.0 18.3 oracle
lrud 262152 0.0 0.5 root
oracle 11337728 0.0 20.1 oracle
oracle 8978578 0.0 13.9 oracle
oracle 8192170 0.0 13.8 oracle
syncd 2228416 0.0 0.6 root
clcomd 8519684 0.0 4.9 root
EvMgrC 5570622 0.0 10.0 root
nfsd 2949318 0.0 1.8 root
ksh 7012396 0.0 0.5 root
報錯資訊如下：
[biprd1]root#cat /var/hacmp/adm/cluster.log
.......
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Mar 26 15:19:41 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
.....
透過重啟服務問題依舊，日誌檔案大約每三分鐘就增長100M
stopsrc -s syslogd
startsrc -s syslogd
於是kill slp_srvr程式
[biprd1]root#ps -ef|grep slp
root 2818092 1 0 Sep 24 - 0:52 /opt/ibm/director/cimom/bin/tier1slp
root 3735686 1 66 Sep 24 - 2136:34 ./slp_srvreg -D
root 16646316 16253072 0 16:40:51 pts/0 0:00 grep slp
[biprd1]root#kill -9 3735686
[biprd1]root#
馬上停止生成日誌，cpu也降了下來，再將slp_srvr程式起來
cd /usr/sbin
./slp_srvreg -D
[biprd1]root#df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 0.50 0.26 49% 10921 15% /
/dev/hd2 7.00 3.17 55% 56403 8% /usr
/dev/hd9var 5.00 1.58 69% 7296 2% /var
/dev/hd3 1.50 0.88 42% 20596 9% /tmp
/dev/hd1 0.12 0.07 42% 187 2% /home
/dev/hd11admin 0.12 0.12 1% 5 1% /admin
/proc - - - - - /proc
/dev/hd10opt 6.00 3.09 49% 22645 4% /opt
/dev/livedump 0.25 0.25 1% 5 1% /var/adm/ras/livedump
/dev/u01_fs 30.00 11.93 61% 235774 8% /u01
/dev/tsmlog 0.25 0.25 1% 9 1% /tsmlog
/dev/fslv00 60.00 44.19 27% 12 1% /bibak
/dev/nmonlv 0.50 0.31 38% 39 1% /nmon
/dev/backup_fs 40.00 13.00 68% 76 1% /backup
/dev/u02_fs 409.00 166.94 60% 400 1% /u02
[biprd1]root#iostat 1 4
System configuration: lcpu=8 drives=5 ent=0.50 paths=10 vdisks=6
tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc
0.0 108.6 0.5 3.3 95.9 0.2 0.0 7.0
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk5 1.0 25.5 6.4 16 0
hdisk1 0.0 0.0 0.0 0 0
hdisk6 0.0 0.0 0.0 0 0
hdisk7 0.0 0.0 0.0 0 0
tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc
0.0 865.0 18.5 8.7 72.9 0.0 0.2 36.0
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk5 0.0 19.0 4.8 0 12
hdisk1 0.0 0.0 0.0 0 0
hdisk6 0.0 76.0 4.8 0 48
hdisk7 0.0 0.0 0.0 0 0
tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc
0.0 873.6 0.4 2.8 96.8 0.0 0.0 5.9
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk5 0.0 0.0 0.0 0 0
hdisk1 0.0 0.0 0.0 0 0
hdisk6 0.0 0.0 0.0 0 0
hdisk7 0.0 0.0 0.0 0 0
tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc
0.0 873.6 0.4 2.9 96.8 0.0 0.0 6.0
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk5 0.0 0.0 0.0 0 0
hdisk1 0.0 0.0 0.0 0 0
hdisk6 0.0 0.0 0.0 0 0
hdisk7 0.0 0.0 0.0 0 0
[biprd1]root#ls -l
total 2099800
-rw-r--r-- 1 root system 0 Nov 28 2011 clavan.log
-rw-r--r-- 1 root system 74978079 Apr 02 09:04 cluster.log
-rw-r--r-- 1 root system 100000040 Apr 01 16:39 cluster.log.0
-rw-r--r-- 1 root system 100000078 Apr 01 16:36 cluster.log.1
-rw-r--r-- 1 root system 100000002 Apr 01 16:34 cluster.log.2
-rw-r--r-- 1 root system 100000002 Apr 01 16:31 cluster.log.3
-rw-r--r-- 1 root system 100000002 Apr 01 16:28 cluster.log.4
-rw-r--r-- 1 root system 100000040 Apr 01 16:26 cluster.log.5
-rw-r--r-- 1 root system 100000116 Apr 01 16:23 cluster.log.6
-rw-r--r-- 1 root system 100000078 Apr 01 16:21 cluster.log.7
-rw-r--r-- 1 root system 100000078 Apr 01 16:18 cluster.log.8
-rw-r--r-- 1 root system 100000078 Apr 01 16:15 cluster.log.9
drwxr-xr-x 2 root system 12288 Mar 07 2018 history
drwxr-xr-x 2 root system 256 Sep 24 2016 wpar
可見日誌不再大量生產
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-031 [804399472] Calloc call failed. errno = 804399468.
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Apr 1 16:41:10 biprd1 daemon:err|error syslog: slp: 0660-080 [804399792] SLPServiceListener -- The SA failed to compute received message: 瀟 (-21).
Apr 1 17:04:50 biprd1 daemon:err|error syslog: slp: 0660-065 [804398544] Impossible to parse attribute |^L^Am.
Apr 1 17:04:50 biprd1 daemon:err|error syslog: slp: [804398928] decode_srvreg -- __srv_reg_local failed with rc = 804398924.
Apr 1 17:04:50 biprd1 daemon:err|error syslog: slp: 0660-084 [804399280] The SA failed to decode and compute received message: 醊繼Am (-2).
Apr 1 18:04:50 biprd1 daemon:err|error syslog: slp: 0660-065 [804398544] Impossible to parse attribute |^L^Am.
Apr 1 18:04:50 biprd1 daemon:err|error syslog: slp: [804398928] decode_srvreg -- __srv_reg_local failed with rc = 804398924.
Apr 1 18:04:50 biprd1 daemon:err|error syslog: slp: 0660-084 [804399280] The SA failed to decode and compute received message: 醊繼Am (-2).
Apr 1 19:04:50 biprd1 daemon:err|error syslog: slp: 0660-065 [804398544] Impossible to parse attribute |^L^Am.
Apr 1 19:04:50 biprd1 daemon:err|error syslog: slp: [804398928] decode_srvreg -- __srv_reg_local failed with rc = 804398924.
Apr 1 19:04:50 biprd1 daemon:err|error syslog: slp: 0660-084 [804399280] The SA failed to decode and compute received message: 醊繼Am (-2).
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/26870952/viewspace-2640137/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- guava cache大量的WARN日誌的問題分析Guava
- messages 日誌裡面大量的fcoemon錯誤
- 啟動tomcat時，日誌裡大量輸出建立資料來源dataSource的日誌Tomcat
- [Java/日誌] 日誌框架列印應用程式日誌程式碼的執行情況Java框架
- 日誌管理程式
- 【MySQL】慢日誌大量生成 long-query-time “失效”問題MySql
- 程式中的日誌
- 物化檢視日誌(materialized view log)引起大量Dfs Lock Handle等待ZedView
- 11g告警日誌中大量ORA-12170錯誤
- 系統日誌中出現大量的truncating integer value to 32 bits資訊
- 物化檢視comlete重新整理會產生大量的日誌
- 11G flashback data archive 導致產生大量歸檔日誌Hive
- 應用程式日誌Sample
- ELK日誌系統之通用應用程式日誌接入方案
- python logging日誌模組以及多程式日誌Python
- .NET 8使用日誌功能以及自定義日誌提供程式
- HACMP的cluster.log報錯ACM
- Percona MySQL 日誌中出現大量Aborted connection (Got an error reading)MySqlGoError
- mysql之 日誌體系（錯誤日誌、查詢日誌、二進位制日誌、事務日誌、中繼日誌）MySql中繼
- 論加鎖否：程式日誌
- 小程式日誌監控工具
- Python監控日誌程式Python
- 記：”$ORACLE_HOME/.../sysman/recv/errors“目錄下產生大量日誌檔案OracleError
- Oralcle 10g資料庫做shrink space操作建立了大量redo日誌資料庫
- php日誌,記錄日誌PHP
- 日誌分析-apache日誌分析Apache
- 聯機重做日誌、歸檔日誌、備用重做日誌
- kafka-log日誌程式碼解析Kafka
- 如何在程式碼中打日誌
- 壓縮SQL SERVER日誌程式碼SQLServer
- Python多程式記錄日誌Python
- mysql 日誌之錯誤日誌MySql
- mysql 日誌之慢查詢日誌MySql
- 【Oracle日誌】- 日誌檔案重建Oracle
- 在被線上大量日誌輸出導致效能瓶頸，執行緒Block的坑執行緒BloC
- 日誌
- [日誌分析篇]-利用ELK分析jumpserver日誌-日誌拆分篇Server
- mysql 日誌之普通查詢日誌MySql