小丸子學MongoDB系列之——副本集在容災環境中的故障演練
假設MongoDB副本集部署在多機房環境中，下面模擬下機房故障發生時的現象已經處理過程。
0.架構規劃
主機名 IP 角色 機房
hadoop1 10.1.245.72 primary 北京
hadoop2 10.1.245.73 secondary1 北京
hadoop3 10.1.245.74 secondary2 上海
1.檢視副本集狀態
[mgousr01@hadoop1 ~]$ mongo 10.1.245.72:37027
rstl:PRIMARY> rs.status()
{
"set" : "rstl",
"date" : ISODate("2015-12-23T03:26:38.917Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "10.1.245.72:37027",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 815,
"optime" : Timestamp(1450841104, 1),
"optimeDate" : ISODate("2015-12-23T03:25:04Z"),
"electionTime" : Timestamp(1450841106, 1),
"electionDate" : ISODate("2015-12-23T03:25:06Z"),
"configVersion" : 1,
"self" : true
},
{
"_id" : 1,
"name" : "10.1.245.73:37037",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 94,
"optime" : Timestamp(1450841104, 1),
"optimeDate" : ISODate("2015-12-23T03:25:04Z"),
"lastHeartbeat" : ISODate("2015-12-23T03:26:38.312Z"),
"lastHeartbeatRecv" : ISODate("2015-12-23T03:26:38.188Z"),
"pingMs" : 0,
"configVersion" : 1
},
{
"_id" : 2,
"name" : "10.1.245.74:37047",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 94,
"optime" : Timestamp(1450841104, 1),
"optimeDate" : ISODate("2015-12-23T03:25:04Z"),
"lastHeartbeat" : ISODate("2015-12-23T03:26:38.312Z"),
"lastHeartbeatRecv" : ISODate("2015-12-23T03:26:38.238Z"),
"pingMs" : 0,
"lastHeartbeatMessage" : "could not find member to sync from",
"configVersion" : 1
}
],
"ok" : 1
}
2.測試資料同步功能
2.1 primary執行寫入操作
rstl:PRIMARY> use test
switched to db test
rstl:PRIMARY> db.goods.insert({name:"eggs",price:38})
WriteResult({ "nInserted" : 1 })
rstl:PRIMARY> show collections;
goods
system.indexes
rstl:PRIMARY> db.goods.find();
{ "_id" : ObjectId("567a152c8d26acb133400410"), "name" : "eggs", "price" : 38 }
2.2 secondary1檢視同步資料
[mgousr01@hadoop2 ~]$ mongo 10.1.245.73:37037
rstl:SECONDARY> use test
switched to db test
rstl:SECONDARY> db.goods.find();
{ "_id" : ObjectId("567a152c8d26acb133400410"), "name" : "eggs", "price" : 38 }
2.3 secondary2檢視同步資料
rstl:SECONDARY> use test
switched to db test
rstl:SECONDARY> db.goods.find();
{ "_id" : ObjectId("567a152c8d26acb133400410"), "name" : "eggs", "price" : 38 }
注：資料同步正常
3.模擬北京機房故障
3.1 同時將primary和secondary1上的mongod程式kill掉
[mgousr01@hadoop1 ~]$ ps -ef|grep mongod|grep -v grep
mgousr01 32555 1 0 11:13 ? 00:00:04 mongod -f mongodb/conf/mg72.conf
[mgousr01@hadoop1 ~]$ kill -9 32555
[mgousr01@hadoop2 ~]$ ps -ef|grep mongod |grep -v grep
mgousr01 15981 1 0 11:13 ? 00:00:05 mongod -f mongodb/conf/mg73.conf
[mgousr01@hadoop2 ~]$ kill -9 15981
3.2 secondary2上檢視當前副本集狀態
rstl:SECONDARY> rs.status()
{
"set" : "rstl",
"date" : ISODate("2015-12-23T03:36:08.448Z"),
"myState" : 2,
"members" : [
{
"_id" : 0,
"name" : "10.1.245.72:37027",
"health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2015-12-23T03:36:07.019Z"),
"lastHeartbeatRecv" : ISODate("2015-12-23T03:34:51Z"),
"pingMs" : 0,
"lastHeartbeatMessage" : "Failed attempt to connect to 10.1.245.72:37027; couldn't connect to server 10.1.245.72:37027 (10.1.245.72), connection attempt failed", "configVersion" : -1
},
{
"_id" : 1,
"name" : "10.1.245.73:37037",
"health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2015-12-23T03:36:07.021Z"),
"lastHeartbeatRecv" : ISODate("2015-12-23T03:34:52.948Z"),
"pingMs" : 0,
"lastHeartbeatMessage" : "Failed attempt to connect to 10.1.245.73:37037; couldn't connect to server 10.1.245.73:37037 (10.1.245.73), connection attempt failed",
"configVersion" : -1
},
{
"_id" : 2,
"name" : "10.1.245.74:37047",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 1345,
"optime" : Timestamp(1450841388, 2),
"optimeDate" : ISODate("2015-12-23T03:29:48Z"),
"configVersion" : 1,
"self" : true
}
],
"ok" : 1
}
rstl:SECONDARY> db.goods.find();
{ "_id" : ObjectId("567a152c8d26acb133400410"), "name" : "eggs", "price" : 38 }
注：目前已經無法連線到primary和secondary1，當前還可以查詢資料
rstl:SECONDARY> db.goods.insert({name:"cake",price:100});
WriteResult({ "writeError" : { "code" : undefined, "errmsg" : "not master" } })
注：secondary成員是無法接受寫入請求的
3.3 secondary2上重新初始化副本集配置
rstl:SECONDARY> cfg = rs.conf()
rstl:SECONDARY> printjson(cfg) --備份當前配置
rstl:SECONDARY> cfg.members = [cfg.members[2]] --在配置中刪除不可用成員
rstl:SECONDARY> rs.reconfig(cfg, {force : true}) --讓新配置重新生效
{ "ok" : 1 }
rstl:PRIMARY>
注：這個時候提示符已經從"SECONDARY"變成了"PRIMARY"
rstl:PRIMARY> rs.status(); --檢視當前副本集狀態
{
"set" : "rstl",
"date" : ISODate("2015-12-23T05:49:39.772Z"),
"myState" : 1,
"members" : [
{
"_id" : 2,
"name" : "10.1.245.74:37047",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 9356,
"optime" : Timestamp(1450841388, 2),
"optimeDate" : ISODate("2015-12-23T03:29:48Z"),
"electionTime" : Timestamp(1450849750, 1),
"electionDate" : ISODate("2015-12-23T05:49:10Z"),
"configVersion" : 21037,
"self" : true
}
],
"ok" : 1
}
rstl:PRIMARY> db.goods.insert({name:"cake",price:100});
WriteResult({ "nInserted" : 1 })
rstl:PRIMARY> db.goods.find();
{ "_id" : ObjectId("567a152c8d26acb133400410"), "name" : "eggs", "price" : 38 }
{ "_id" : ObjectId("567a361bb72708516df2cb36"), "name" : "cake", "price" : 100 }
至此，新的primary配置成功並能夠正常接收寫入請求。
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/20801486/viewspace-1878235/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- 小丸子學MongoDB系列之——部署MongoDB副本集MongoDB
- 小丸子學MongoDB系列之——副本集Auto-FailoverMongoDBAI
- 小丸子學MongoDB系列之——安裝MongoDBMongoDB
- mongodb 容災演練操作步步驟【適用於計劃內演練】MongoDB
- MongoDB日常運維-05副本集故障切換MongoDB運維
- 2.MongoDB 4.2副本集環境基於時間點的恢復MongoDB
- mongodb 4.0副本集搭建MongoDB
- MongoDB 6.0.3副本集搭建MongoDB
- MongoDB副本集replica set (二)--副本集環境搭建MongoDB
- MongoDB 4.2副本集自動故障轉移（一主一副一仲裁）MongoDB
- 容災演練月報 | 福萊特集團核心系統完成“跨雲”容災切換演練
- 小丸子學MongoDB系列之——部署Replica Set+Sharded ClusterMongoDB
- 在ECS上自建MySQL手工容災環境MySql
- 容災演練月報 | 紹興銀行四大業務系統完成容災切換演練
- 容災演練月報 | 雅安市商業銀行四大業務系統完成容災切換演練
- MongoDB日常運維-04副本集搭建MongoDB運維
- 容災演練，一鍵切換，浙大二院實戰演練圓滿成功！
- MongoDB之副本集MongoDB
- MongoDB實戰-如何在Windows環境下管理副本集MongoDBWindows
- 小丸子學Redis系列之——安裝RedisRedis
- 小丸子學Redis系列之——Data types（一）Redis
- 小丸子學Docker系列之——實戰DockerfileDocker
- 生產環境故障處理演練-mysql資料庫主從恢復MySql資料庫
- MongoDB副本集故障恢復機制概述MongoDB
- redis的sentinel模式故障演練Redis模式
- MongoDB 4.2副本集新增/刪除副本（一主一副一仲裁）MongoDB
- Mongodb3.0.5副本集搭建及spring和java連線副本集配置MongoDBSpringJava
- 小丸子學Hadoop系列之——部署Hbase叢集Hadoop
- 修改mongodb3.0副本集使用者密碼遇到的坑MongoDB密碼
- 小丸子學Hadoop系列之——部署Hadoop叢集Hadoop
- oracle 資料庫搭建高可用環境 容災參考。Oracle資料庫
- 系統高可用之容災應急切換演練探討活動總結
- MongoDB副本集MongoDB
- 小丸子學Hadoop系列之——hbase備份與恢復Hadoop
- 小丸子學Oracle 12c系列之——Oracle Pluggable DatabaseOracleDatabase
- 小丸子學Docker系列之——安裝Docker及基本命令的使用Docker
- Flutter學習系列之Flutter上手環境準備Flutter
- MongoDB系列三：Replica Sets在生產環境中安裝配置的注意事項MongoDB