Nagios監控mongodb分片叢集服務實戰

mchdba發表於2014-10-10
iOSMongoDB


1，監控外掛下載 
Mongodb外掛下載地址為：git clone git://github.com/mzupan/nagios-plugin-mongodb.git，剛開始本人這裡沒有安裝gitpub環境，找網友草根幫忙下載的，之後上傳到了csdn資源頁面，新的下載地址為：http://download.csdn.net/detail/mchdba/8019077

2，新增新的mongodb監控命令

因為mongodb服務是和mysql從庫公用一臺物理機，之前已經做了基礎nagios以及mysql服務監控，所以這裡只需要在原來的基礎上新增mongodb命令和服務即可。Nagios監控mysql請參考：http://blog.itpub.net/26230597/viewspace-760141/以及http://blog.itpub.net/26230597/viewspace-1217246/。所以這裡需要新增的mongodb監控命令如下所示：

  1. [root@wgq objects]# cd /usr/local/nagios/etc/objects
  2. [root@wgq objects]# vim commands.cfg
  3. define command {
  4.     command_name check_mongodb
  5.     command_line $USER1$/nagios-plugin-mongodb/check_mongodb.py -H $HOSTADDRESS$ -A $ARG1$ -P $ARG2$ -W $ARG3$ -C $ARG4$
  6. }

  8. define command {
  9.     command_name check_mongodb_database
  10.     command_line $USER1$/nagios-plugin-mongodb/check_mongodb.py -H $HOSTADDRESS$ -A $ARG1$ -P $ARG2$ -W $ARG3$ -C $ARG4$ -d $ARG5$
  11. }

  13. define command {
  14.     command_name check_mongodb_collection
  15.     command_line $USER1$/nagios-plugin-mongodb/check_mongodb.py -H $HOSTADDRESS$ -A $ARG1$ -P $ARG2$ -W $ARG3$ -C $ARG4$ -d $ARG5$ -c $ARG6$
  16. }

  18. define command {
  19.     command_name check_mongodb_replicaset
  20.     command_line $USER1$/nagios-plugin-mongodb/check_mongodb.py -H $HOSTADDRESS$ -A $ARG1$ -P $ARG2$ -W $ARG3$ -C $ARG4$ -r $ARG5$
  21. }

  23. define command {
  24.     command_name check_mongodb_query
  25.     command_line $USER1$/nagios-plugin-mongodb/check_mongodb.py -H $HOSTADDRESS$ -A $ARG1$ -P $ARG2$ -W $ARG3$ -C $ARG4$ -q $ARG5$
  26. }

3，新增mongodb監控服務
mongodb的服務也需要單獨重新新增，如下所示：

  1. #檢測mongodb服務的連線時間，超過2秒就普通報警，5秒就嚴重報警
  2. define service{
  3.         host_name dbm1slave1
  4.         service_description Mongo Connect Check
  5.         check_command check_mongodb!connect!30000!2!5
  6.         max_check_attempts 5
  7.         normal_check_interval 3
  8.         retry_check_interval 2
  9.         check_period 24x7
  10.         notification_interval 10
  11.         notification_period 24x7
  12.         notification_options w,u,c,r
  13.         contact_groups ops
  14.         }

  16. #檢查mongodb的連線數，超過150普通報警，200嚴重報警
  17. define service{
  18.         host_name dbm1slave1
  19.         service_description Mongo Free Connections
  20.         check_command check_mongodb!connections!27017!70!80
  21.         max_check_attempts 5
  22.         normal_check_interval 3
  23.         retry_check_interval 2
  24.         check_period 24x7
  25.         notification_interval 10
  26.         notification_period 24x7
  27.         notification_options w,u,c,r
  28.         contact_groups ops
  29.         }
  30.         
  31.         
  32. #檢查mongodb複製完成的百分比率，確保primary和standby的time是一致的。
  33. define service{
  34.         host_name dbm1slave1
  35.         service_description Mongo Replication Lag
  36.         check_command check_mongodb!replication_lag!27017!15!30
  37.         max_check_attempts 5
  38.         normal_check_interval 3
  39.         retry_check_interval 2
  40.         check_period 24x7
  41.         notification_interval 10
  42.         notification_period 24x7
  43.         notification_options w,u,c,r
  44.         contact_groups ops
  45.         }
  46.         
  47. #檢查mongodb記憶體使用率，閥值與mongodb所在機器的總記憶體數相關
  48. define service{
  49.         host_name dbm1slave1
  50.         service_description Mongo Memory Usage
  51.         check_command check_mongodb!memory!27017!20!28
  52.         max_check_attempts 5
  53.         normal_check_interval 3
  54.         retry_check_interval 2
  55.         check_period 24x7
  56.         notification_interval 10
  57.         notification_period 24x7
  58.         notification_options w,u,c,r
  59.         contact_groups ops
  60.         }
  61.         
  62. #檢查mongodb Mapped的記憶體使用率，閥值與mongodb所在機器的總記憶體數相關
  63. define service{
  64.         host_name dbm1slave1
  65.         service_description Mongo Mapped Memory Usage
  66.         check_command check_mongodb!memory_mapped!27017!20!28
  67.         max_check_attempts 5
  68.         normal_check_interval 3
  69.         retry_check_interval 2
  70.         check_period 24x7
  71.         notification_interval 10
  72.         notification_period 24x7
  73.         notification_options w,u,c,r
  74.         contact_groups ops
  75.         }
  76.         
  77. #檢查Lock Time的百分率，如果lock time佔據mongo執行時間的5%就普通報警，如果超過10%就嚴重報警
  78. define service{
  79.         host_name dbm1slave1
  80.         service_description Mongo Lock Percentage
  81.         check_command check_mongodb!lock!27017!5!10
  82.         max_check_attempts 5
  83.         normal_check_interval 3
  84.         retry_check_interval 2
  85.         check_period 24x7
  86.         notification_interval 10
  87.         notification_period 24x7
  88.         notification_options w,u,c,r
  89.         contact_groups ops
  90.         }

  92. # Check Average Flush Time，檢查mongo伺服器的平均flush時間，
  93. define service{
  94.         host_name dbm1slave1
  95.         service_description Mongo Flush Average
  96.         check_command check_mongodb!flushing!27017!100!200
  97.         max_check_attempts 5
  98.         normal_check_interval 3
  99.         retry_check_interval 2
  100.         check_period 24x7
  101.         notification_interval 10
  102.         notification_period 24x7
  103.         notification_options w,u,c,r
  104.         contact_groups ops
  105.         }

  107. # Check Last Flush Time，檢查最新的flush時間，如果超過200ms就普通報警，超過400ms就嚴重報警
  108. define service{
  109.         host_name dbm1slave1
  110.         service_description Mongo Last Flush Time
  111.         check_command check_mongodb!last_flush_time!27017!200!400
  112.         max_check_attempts 5
  113.         normal_check_interval 3
  114.         retry_check_interval 2
  115.         check_period 24x7
  116.         notification_interval 10
  117.         notification_period 24x7
  118.         notification_options w,u,c,r
  119.         contact_groups ops
  120.         }
  121.         
  122. # Check status of mongodb replicaset，檢查mongo複製的狀態
  123. define service{
  124.         host_name dbm1slave1
  125.         service_description MongoDB state
  126.         check_command check_mongodb!replset_state!27017!0!0
  127.         max_check_attempts 5
  128.         normal_check_interval 3
  129.         retry_check_interval 2
  130.         check_period 24x7
  131.         notification_interval 10
  132.         notification_period 24x7
  133.         notification_options w,u,c,r
  134.         contact_groups ops
  135.         }

  137. # Check status of index miss ratio，檢查索引命中率，
  138. define service{
  139.         host_name dbm1slave1
  140.         service_description MongoDB Index Miss Ratio
  141.         check_command check_mongodb!index_miss_ratio!27017!.005!.01
  142.         max_check_attempts 5
  143.         normal_check_interval 3
  144.         retry_check_interval 2
  145.         check_period 24x7
  146.         notification_interval 10
  147.         notification_period 24x7
  148.         notification_options w,u,c,r
  149.         contact_groups ops
  150.         }
  151.         
  152. # Check number of databases and number of collections
  153. define service{
  154.         host_name dbm1slave1
  155.         service_description MongoDB Number of databases
  156.         check_command check_mongodb!databases!27017!300!500
  157.         max_check_attempts 5
  158.         normal_check_interval 3
  159.         retry_check_interval 2
  160.         check_period 24x7
  161.         notification_interval 10
  162.         notification_period 24x7
  163.         notification_options w,u,c,r
  164.         contact_groups ops
  165.         }
  166. define service{
  167.         host_name dbm1slave1
  168.         service_description MongoDB Number of collections
  169.         check_command check_mongodb!collections!27017!300!500
  170.         max_check_attempts 5
  171.         normal_check_interval 3
  172.         retry_check_interval 2
  173.         check_period 24x7
  174.         notification_interval 10
  175.         notification_period 24x7
  176.         notification_options w,u,c,r
  177.         contact_groups ops
  178.         }        
  179.         
  180. # Check size of a database，檢查庫的大小
  181. define service{
  182.         host_name dbm1slave1
  183.         service_description MongoDB Database size your-database
  184.         check_command check_mongodb_database!database_size!27017!300!500!your-database
  185.         max_check_attempts 5
  186.         normal_check_interval 3
  187.         retry_check_interval 2
  188.         check_period 24x7
  189.         notification_interval 10
  190.         notification_period 24x7
  191.         notification_options w,u,c,r
  192.         contact_groups ops
  193.         }                
  194.         
  195. # Check index size of a database，檢查庫索引的大小
  196. define service{
  197.         host_name dbm1slave1
  198.         service_description MongoDB Database index size your-database
  199.         check_command check_mongodb_database!database_indexes!27017!50!100!your-database
  200.         max_check_attempts 5
  201.         normal_check_interval 3
  202.         retry_check_interval 2
  203.         check_period 24x7
  204.         notification_interval 10
  205.         notification_period 24x7
  206.         notification_options w,u,c,r
  207.         contact_groups ops
  208.         }            
  209.         
  210. # Check index size of a collection，檢查集合collection的索引大小
  211. define service{
  212.         host_name dbm1slave1
  213.         service_description MongoDB Database index size your-database
  214.         check_command check_mongodb_collection!collection_indexes!27017!50!100!your-database!your-collection
  215.         max_check_attempts 5
  216.         normal_check_interval 3
  217.         retry_check_interval 2
  218.         check_period 24x7
  219.         notification_interval 10
  220.         notification_period 24x7
  221.         notification_options w,u,c,r
  222.         contact_groups ops
  223.         }
  224.         
  225. # Check the primary server of replicaset，檢查複製的primary服務
  226. define service{
  227.         host_name dbm1slave1
  228.         service_description MongoDB Replicaset Master Monitor: your-replicaset
  229.         check_command check_mongodb_replicaset!replica_primary!27017!0!1!your-replicaset
  230.         #示例：check_command check_mongodb_replicaset!replica_primary!27017!0!1!shard2
  231.         max_check_attempts 5
  232.         normal_check_interval 3
  233.         retry_check_interval 2
  234.         check_period 24x7
  235.         notification_interval 10
  236.         notification_period 24x7
  237.         notification_options w,u,c,r
  238.         contact_groups ops
  239.         }

  241.         
  242. # Check the number of queries per second，檢查每一秒的查詢數量
  243. define service{
  244.         host_name dbm1slave1
  245.         service_description MongoDB Updates per Second
  246.         check_command check_mongodb_query!queries_per_second!27017!200!150!update
  247.         max_check_attempts 5
  248.         normal_check_interval 3
  249.         retry_check_interval 2
  250.         check_period 24x7
  251.         notification_interval 10
  252.         notification_period 24x7
  253.         notification_options w,u,c,r
  254.         contact_groups ops
  255.         }
  256.         
  257. # Check Primary Connection，檢查複製中與primary庫的連線時間，超過2秒就普通報警，超過4秒就嚴重報警
  258. define service{
  259.         host_name dbm1slave1
  260.         service_description Mongo Connect Check
  261.         check_command check_mongodb!connect_primary!27017!2!4
  262.         max_check_attempts 5
  263.         normal_check_interval 3
  264.         retry_check_interval 2
  265.         check_period 24x7
  266.         notification_interval 10
  267.         notification_period 24x7
  268.         notification_options w,u,c,r
  269.         contact_groups ops
  270.         }

  272. # Check Collection State，檢查collection狀態，檢查mongo服務組列表的每一個主機，可以檢查重要collection的高可用性（鎖、超時、服務配置的可用性），如果發現一個查詢失敗就會報警。
  273. define service{
  274.         host_name dbm1slave1
  275.         service_description Mongo Collection State
  276.         check_command check_mongodb!collection_state!27017!your-database!your-collection
  277.         max_check_attempts 5
  278.         normal_check_interval 3
  279.         retry_check_interval 2
  280.         check_period 24x7
  281.         notification_interval 10
  282.         notification_period 24x7
  283.         notification_options w,u,c,r
  284.         contact_groups ops
  285.         }

4，檢視部分監控項效果

配置完nagios端服務，重啟下service nagios restart; 等上幾分鐘，nagios監控介面就會出現完整的mongo服務資訊，如下所示：




5，從ps中確定mongodb的架構

[root@db-m1-slave-1 ~]# ps -eaf|grep mongo

mongodb   2457     1  0  2013 ?        2-03:39:08 ./mongod --configsvr --dbpath /home/data/mongodb/config --port 20000 --logpath /home/data/mongodb/config.log --logappend --fork

mongodb   2804     1  0  2013 ?        1-10:02:33 mongos --configdb 192.168.12.62:20000,192.168.12.63:20000,192.168.12.72:20000 --port 30000 --chunkSize 64 --logpath /home/data/mongodb/mongos.log --logappend --fork

mongodb   3072     1  0  2013 ?        1-10:17:20 mongod --shardsvr --replSet shard1 --port 27017 --dbpath /home/data/mongodb/shard11 --oplogSize 2048 --logpath /home/data/mongodb/shard11.log --logappend --fork

root     11179  9391  0 11:14 pts/1    00:00:00 grep mongo

mongodb  30414     1  0 Feb14 ?        1-06:20:50 mongod --shardsvr --replSet shard2 --port 27018 --dbpath /home/data/mongodb/shard21 --oplogSize 2048 --logpath /home/data/mongodb/shard21.log --logappend --fork

[root@db-m1-slave-1 ~]#

 

看到有4個mongo程式，

a)         啟動引數有“--configdb”的就是叢集入口程式；

b)         Shard Server，啟動引數帶“--shardsvr --replSet”的是叢集分片的一個片組啟動程式，使用者儲存實際的資料塊，也就是27017埠和27018埠的mongodb服務例項。至於如何判斷27017埠中哪個是primary哪個是secondary需要去登入27107埠執行rs.status();去檢視一下。

c)         Config Server：啟動引數帶“--configsvr”的程式，儲存了整個Cluster Metadata，其中包括chunk資訊，也就是20000埠的mongodb服務例項。

d)         Route Server：啟動引數帶“mongos --configdb”的程式，前端路由，客戶端由此接入，且讓整個叢集看上去像單一資料庫，前端應用可以透明使用，也就是30000埠的mongodb例項。


6，除錯中出現過的錯誤

錯誤1：

[root@wgq nagios ~]# tail -f /usr/local/nagios/var/nagios.log

[1412819956] Warning: Return code of 13 for check of service 'Mongo Memory Usage' on host 'dbm1slave1' was out of bounds.

[1412819956] SERVICE ALERT: dbm1slave1;Mongo Memory Usage;CRITICAL;SOFT;1;(Return code of 13 is out of bounds)

[1412819975] Warning: Return code of 13 for check of service 'Mongodb Connect Check' on host 'dbm1slave1' was out of bounds.

[1412819975] SERVICE ALERT: dbm1slave1;Mongodb Connect Check;CRITICAL;SOFT;1;(Return code of 13 is out of bounds)

[1412820058] Warning: Return code of 13 for check of service 'Mongo Free Connections' on host 'dbm1slave1' was out of bounds.

 

需要賦值nagios使用者所有許可權以及r執行許可權

chmod 770 /usr/lib/nagios/plugins/check_mongodb.py

chown -R nagios.nagios /usr/lib/nagios/plugins/check_mongodb.py

 

錯誤2：

監控介面Status Information一欄出現 No module named pymongo報錯提示資訊：

出現這個提示是因為需要安裝pymongo模組，執行easy_install pymongo命令安裝即可，如下所示：

[root@wgq objects]# easy_install pymongo

Searching for pymongo

Reading http://pypi.python.org/simple/pymongo/

Best match: pymongo 2.7.2

......

zip_safe flag not set; analyzing archive contents...

Adding pymongo 2.7.2 to easy-install.pth file

 

Installed /usr/lib/python2.6/site-packages/pymongo-2.7.2-py2.6-linux-x86_64.egg

Processing dependencies for pymongo

Finished processing dependencies for pymongo

 

參考文章：https://github.com/mzupan/nagios-plugin-mongodb/blob/master/README.md

 



來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/26230597/viewspace-1293589/，如需轉載，請註明出處，否則將追究法律責任。

相關文章