Ubuntu下MongoDB failed code=exited, status=XXX問題排查 以及 Ubuntu下檢視磁碟使用情況和快速清理方法

某工程師$發表於2019-02-19
UbuntuMongoDBAI

問題描述：早上泡了杯茶，前端同學通知我服務不可用了。艾，昨天還能用的，今天怎麼涼了，應該不是程式碼出了什麼問題，感覺問題出現在外部。上伺服器檢視了下今天的日誌，今天的 error log檔案有400K，往常都是個位數的，檢視之，果然，MongoDB無法連線，頻繁報錯。

問題排查：伺服器用的是 Ubuntu，通過  service mongod status 檢視mongod的狀態如下：

 mongod.service - MongoDB Database Server
   Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2018-08-28 22:17:37 CEST; 10s ago
     Docs: https://docs.mongodb.org/manual
  Process: 392 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=48)
 Main PID: 392 (code=exited, status=48)
      CPU: 33ms

aug 28 22:17:37 staging-manager-1 systemd[1]: Started MongoDB Database Server.
aug 28 22:17:37 staging-manager-1 mongod[392]: 2018-08-28T22:17:37.389+0200 I CONTROL  [main] Automatically disabling TLS 1.0, to fo
aug 28 22:17:37 staging-manager-1 systemd[1]: mongod.service: Main process exited, code=exited, status=48/n/a
aug 28 22:17:37 staging-manager-1 systemd[1]: mongod.service: Unit entered failed state.
aug 28 22:17:37 staging-manager-1 systemd[1]: mongod.service: Failed with result 'exit-code'.

從上狀態可以看到 MongoDB的啟動命令如下

/usr/bin/mongod --config /etc/mongod.conf

接下來去檢視 mongod.conf 配置檔案，從中可以看到 MongoDB日誌的位置

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

找到了MongoDB系統日誌的位置，去檢視這個日誌就可以看到具體的報錯資訊了，如下：

2019-02-19T09:18:11.786+0800 E STORAGE  [WTCheckpointThread] WiredTiger error (28) [1550539091:784271][1123:0x7fea309b1700], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_write, 579: /var/lib/mongodb/WiredTiger.turtle.set: handle-write: pwrite: failed to write 1077 bytes at offset 0: No space left on device Raw: [1550539091:784271][1123:0x7fea309b1700], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_write, 579: /var/lib/mongodb/WiredTiger.turtle.set: handle-write: pwrite: failed to write 1077 bytes at offset 0: No space left on device

直接看最後的： No space left on device。  哦，特喵的測試服磁碟又滿了。

分析磁碟佔用、找到大檔案快速清理看這篇：https://blog.csdn.net/weixin_37882382/article/details/87704533

然後，終於清理完成了，有空間了，開始重啟 MongoDB：

service mongod restart

艾，用 status命令檢視又沒有成功，再去看日誌，報如下錯誤：

2019-02-19T10:48:35.735+0800 E STORAGE  [initandlisten] WiredTiger error (13) [1550544515:735833][8897:0x7f2977d70a40], wiredtiger_open: __posix_open_file, 715: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied Raw: [1550544515:735833][8897:0x7f2977d70a40], wiredtiger_open: __posix_open_file, 715: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied

意思為：mongod無許可權訪問 /var/lib/mongodb目錄，好的  分配許可權命令如下：

chown -R mongodb:mongodb /var/lib/mongodb

再重啟，終於  active了，over

相關文章