【案例】利用innodb_force_recovery 解決MySQL伺服器crash無法重啟問題

楊奇龍發表於2015-02-25
一 背景
   某一創業的朋友的主機因為磁碟陣列損壞機器crash,重啟MySQL服務時 報如下錯誤:
  1. InnoDB: Reading tablespace information from the .ibd files...
  2. InnoDB: Restoring possible half-written data pages from the doublewrite
  3. InnoDB: buffer...
  4. InnoDB: Doing recovery: scanned up to log sequence number 9120034833
  5. 150125 16:12:51 InnoDB: Starting an apply batch of log records to the database...
  6. InnoDB: Progress in percents: 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 150125 16:12:51 [ERROR] mysqld got signal 11 ;
  7. This could be because you hit a bug. It is also possible that this binary
  8. or one of the libraries it was linked against is corrupt, improperly built,
  9. or misconfigured. This error can also be caused by malfunctioning hardware.
  10. To report this bug, see http://kb.askmonty.org/en/reporting-bugs
  11. We will try our best to scrape up some info that will hopefully help
  12. diagnose the problem, but since we have already crashed,
  13. something is definitely wrong and this may fail.
  14. Server version: 5.5.37-MariaDB-log
  15. key_buffer_size=268435456
  16. read_buffer_size=1048576
  17. max_used_connections=0
  18. max_threads=1002
  19. thread_count=0
  20. It is possible that mysqld could use up to
  21. key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2332093 K bytes of memory
  22. 41 Hope that.
二 分析
    主要關注 mysqld got signal 11 的問題,從日誌內容分析來看,資料庫在機器crash 導致日誌檔案損壞,重啟之後無法正常恢復,更無法正常對外提供服務。

三 解決
    因為日誌已經損壞,這裡採用非常規手段,首先修改innodb_force_recovery引數,使mysqld跳過恢復步驟,將mysqld 啟動,將資料匯出來然後重建資料庫。
innodb_force_recovery可以設定為1-6,大的數字包含前面所有數字的影響。
  1. (SRV_FORCE_IGNORE_CORRUPT):忽略檢查到的corrupt頁。
  2. (SRV_FORCE_NO_BACKGROUND):阻止主執行緒的執行,如主執行緒需要執行full purge操作,會導致crash。
  3. (SRV_FORCE_NO_TRX_UNDO):不執行事務回滾操作。
  4. (SRV_FORCE_NO_IBUF_MERGE):不執行插入緩衝的合併操作。
  5. (SRV_FORCE_NO_UNDO_LOG_SCAN):不檢視重做日誌,InnoDB儲存引擎會將未提交的事務視為已提交。
  6. (SRV_FORCE_NO_LOG_REDO):不執行前滾的操作。
注意 
  a 當設定引數值大於0後,可以對錶進行select,create,drop操作,但insert,update或者delete這類操作是不允許的。
  b 當innodb_purge_threads 和 innodb_force_recovery一起設定會出現一種loop現象:   

  1. 150125 17:07:42 InnoDB: Waiting for the background threads to start
  2. 150125 17:07:43 InnoDB: Waiting for the background threads to start
  3. 150125 17:07:44 InnoDB: Waiting for the background threads to start
  4. 150125 17:07:45 InnoDB: Waiting for the background threads to start
  5. 150125 17:07:46 InnoDB: Waiting for the background threads to start
  6. 150125 17:07:47 InnoDB: Waiting for the background threads to start
在my.cnf中修改以下兩個引數
innodb_force_recovery=6
innodb_purge_thread=0

重啟MySQL 

  1. 150125 17:10:47 [Note] Crash recovery finished.
  2. 150125 17:10:47 [Note] Server socket created on IP: '0.0.0.0'.
  3. 150125 17:10:47 [Note] Event Scheduler: Loaded 0 events
  4. 150125 17:10:47 [Note] /vdata/webserver/mysql/bin/mysqld: ready for connections.
  5. Version: '5.5.37-MariaDB-log' socket: '/tmp/mysql.sock' port: 3306 Source distribution
立即對資料庫做邏輯匯出 ,完成之後將innodb_force_recovery設定為0 ,innodb_purge_thread=1 ,然後重建資料庫 。
另外 MySQL 版本 5.5以及之前 ,當innodb_purge_threads =1,innodb_force_recovery >1 的情況會出現上文提到的迴圈報warning 問題(=1 沒有問題),
原因:
MySQL 的原始碼中顯示  當innodb_purge_threads 和 innodb_force_recovery一起設定會出現loop迴圈
  1. while (srv_shutdown_state == SRV_SHUTDOWN_NONE) {
  2.       if (srv_thread_has_reserved_slot(SRV_MASTER) == ULINT_UNDEFINED
  3.           || (srv_n_purge_threads == 1
  4.           && srv_thread_has_reserved_slot(SRV_WORKER)
  5.           == ULINT_UNDEFINED)) {
  6.           ut_print_timestamp(stderr);
  7.           fprintf(stderr, " InnoDB: Waiting for the background threads to start\n");
  8.           os_thread_sleep(1000000);
  9.       } else {
  10.           break;
  11.       }
  12.   }
所以當需要設定innodb_force_recovery>1的時候需要關閉 innodb_purge_threads,設定為0(預設)。

四 小結 
   MySQL crash 或者 MySQL 資料庫伺服器 crash 會導致各種各樣的問題 ,比如主備之間的error 1594 (5.6 版本開啟crash-safe ,會最大程度上避免 error 1594的問題,以後會寫5.6新特性介紹該功能 ),error 1236 日誌損壞資料檔案損壞 ,等等,本案例只是其中的一種,細心從日誌中找的相關錯誤提示,逐步解決即可。


來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/22664653/viewspace-1441389/,如需轉載,請註明出處,否則將追究法律責任。

相關文章