Mongodb通過一致性備份搭建SECONDARY.

該方法應用面比較窄,

適用於 : 一主 一備 一投票節點,資料庫較大,oplog 比較小,備庫需要修復而且主庫不能停機的情況.

該方法僅限於學習測試,線上環境慎用.

叢集結構：
opsdba-vbj01-1:27018 ARBITE
opsdba-vbj01-1:27019 PRIMARY
opsdba-vbj01-1:27016 SECONDARY

模擬opsdba-vbj01-1:27016 crash ,使用一致性備份搭建備庫.

1.在主庫,自建role,賦予restore oplog許可權.
use admin
db.runCommand({ createRole: “restoreoplog”,
privileges:
        [
        { resource: { anyResource: true }, actions: [ “anyAction” ] }
        ],
roles:
       []
});
db.grantRolesToUser( “root”, [“restoreoplog”] );

2.主庫一致性備份.
[root@opsdba-vbj01-1 dump]# mongodump -uroot -proot123 –port=27019 –oplog –authenticationDatabase=admin -o all_backup
2016-10-11T15:45:58.019+0800    writing admin.system.users to
2016-10-11T15:45:58.019+0800    done dumping admin.system.users (1 document)
2016-10-11T15:45:58.019+0800    writing admin.system.roles to
2016-10-11T15:45:58.020+0800    done dumping admin.system.roles (1 document)
2016-10-11T15:45:58.020+0800    writing admin.system.version to
2016-10-11T15:45:58.020+0800    done dumping admin.system.version (1 document)
2016-10-11T15:45:58.021+0800    writing test.testData to
2016-10-11T15:45:58.021+0800    writing test.tab to
2016-10-11T15:45:58.190+0800    done dumping test.tab (34056 documents)
2016-10-11T15:46:01.022+0800    [###########………….]  test.testData  451185/909000  (49.6%)
2016-10-11T15:46:04.022+0800    [####################….]  test.testData  774771/909000  (85.2%)
2016-10-11T15:46:05.024+0800    [########################]  test.testData  913877/909000  (100.5%)
2016-10-11T15:46:05.024+0800    done dumping test.testData (913877 documents)
2016-10-11T15:46:05.025+0800    writing captured oplog to
2016-10-11T15:46:05.781+0800            dumped 6470 oplog entries

3.把opsdba-vbj01-1:27016 從SECONDARY 轉為單節點空庫.

/data/mongodb/mongodb/bin/mongod -f /data/mongodb/mongodb1/conf/mongod.cnf  –shutdown
清空data目錄,單節點啟動:
修改port = 27015,
註釋掉replSet
/data/mongodb/mongodb/bin/mongod -f /data/mongodb/mongodb1/conf/mongod.cnf
 
建立管理員使用者:
[root@opsdba-vbj01-1 mongodb1]# mongo –port=27015 admin
use admin
db.createUser(
  {
    user: “root”,
    pwd: “root123”,
    roles:
    [
      {
        role: “root”,
        db: “admin”
      }
    ]
  }
);
自建role,賦予restore oplog許可權:
[root@opsdba-vbj01-1 mongodb1]# mongo -uroot -proot123 –port=27015 admin
use admin
db.runCommand({ createRole: “restoreoplog”,
privileges:
        [
        { resource: { anyResource: true }, actions: [ “anyAction” ] }
        ],
roles:
       []
});
db.grantRolesToUser( “root”, [“restoreoplog”] );

4.匯入
[root@opsdba-vbj01-1 dump]# mongorestore -uroot -proot123 –port=27015 –authenticationDatabase=admin –oplogReplay –dir=all_backup
2016-10-11T15:59:48.505+0800    building a list of dbs and collections to restore from all_backup dir
2016-10-11T15:59:48.507+0800    reading metadata for test.testData from all_backup/test/testData.metadata.json
2016-10-11T15:59:48.508+0800    reading metadata for test.tab from all_backup/test/tab.metadata.json
2016-10-11T15:59:48.556+0800    restoring test.tab from all_backup/test/tab.bson
2016-10-11T15:59:48.604+0800    restoring test.testData from all_backup/test/testData.bson
2016-10-11T15:59:49.147+0800    restoring indexes for collection test.tab from metadata
2016-10-11T15:59:49.147+0800    finished restoring test.tab (34056 documents)
2016-10-11T15:59:51.507+0800    [####………………..]  test.testData  18.5 MB/94.1 MB  (19.7%)
2016-10-11T15:59:54.507+0800    [##########…………..]  test.testData  39.8 MB/94.1 MB  (42.3%)
2016-10-11T15:59:57.507+0800    [###############………]  test.testData  59.7 MB/94.1 MB  (63.5%)
2016-10-11T16:00:00.507+0800    [####################….]  test.testData  80.1 MB/94.1 MB  (85.1%)
2016-10-11T16:00:03.288+0800    [########################]  test.testData  94.1 MB/94.1 MB  (100.0%)
2016-10-11T16:00:03.288+0800    restoring indexes for collection test.testData from metadata
2016-10-11T16:00:03.289+0800    finished restoring test.testData (913877 documents)
2016-10-11T16:00:03.289+0800    restoring users from all_backup/admin/system.users.bson
2016-10-11T16:00:03.416+0800    restoring roles from all_backup/admin/system.roles.bson
2016-10-11T16:00:03.466+0800    replaying oplog
2016-10-11T16:00:03.808+0800    done

5.獲取最後一個oplog的時間戳
[root@opsdba-vbj01-1 dump]# cd all_backup/
[root@opsdba-vbj01-1 all_backup]# bsondump oplog.bson >oplog.txt
[root@opsdba-vbj01-1 all_backup]# tail -1 oplog.txt
{“ts”:{“$timestamp”:{“t”:1476171965,”i”:805}},”t”:{“$numberLong”:”2″},”h”:{“$numberLong”:”6906152948185446623″},”v”:2,”op”:”i”,”ns”:”test.testData”,”o”:{“_id”:{“$oid”:”57fc98bddfa99af76706f721″},”x”:6470.0,”name”:”MACLEAN”,”name1″:”MACLEAN”,”name2″:”MACLEAN”,”name3″:”MACLEAN”}}

6.初始化local庫的相關表
[root@opsdba-vbj01-1 mongodb1]# mongo -uroot -proot123 –port=27015 admin
use local
db.runCommand( { create: “oplog.rs”, capped: true, size: (1* 1024 * 1024 * 1024) } );

#資料來自oplog.txt
db.oplog.rs.save({“ts” : Timestamp(1476171965, 805),”h” : NumberLong(“6906152948185446623”)});
db.db.replset.minvalid.save({“ts” : Timestamp(1476171965,805), “t” : NumberLong(2)});

#資料來自主庫的資料查詢
db.replset.election.save({ “_id” : ObjectId(“57fc5ea0cfa6486e03e975d0”), “term” : NumberLong(2), “candidateIndex” : NumberLong(2) });
db.system.replset.save({ “_id” : “myrelset”, “version” : 5, “protocolVersion” : NumberLong(1), “members” : [ { “_id” : 1, “host” : “opsdba-vbj01-1:27018”, “arbiterOnly” : false, “buildIndexes” : true, “hidden” : false, “priority” : 1, “tags” : {  }, “slaveDelay” : NumberLong(0), “votes” : 1 }, { “_id” : 2, “host” : “opsdba-vbj01-1:27019”, “arbiterOnly” : false, “buildIndexes” : true, “hidden” : false, “priority” : 1, “tags” : {  }, “slaveDelay” : NumberLong(0), “votes” : 1 }, { “_id” : 3, “host” : “opsdba-vbj01-1:27016”, “arbiterOnly” : false, “buildIndexes” : true, “hidden” : false, “priority” : 1, “tags” : {  }, “slaveDelay” : NumberLong(0), “votes” : 1 } ], “settings” : { “chainingAllowed” : true, “heartbeatIntervalMillis” : 2000, “heartbeatTimeoutSecs” : 10, “electionTimeoutMillis” : 10000, “getLastErrorModes” : {  }, “getLastErrorDefaults” : { “w” : 1, “wtimeout” : 0 }, “replicaSetId” : ObjectId(“57bfdcdcd40cbe4bf173396a”) } });

7.重啟
use admin
db.shutdownServer();

修改為原始值.
port=27016
replSet 取消註釋
啟動
/data/mongodb/mongodb/bin/mongod -f /data/mongodb/mongodb1/conf/mongod.cnf

8.檢測:
比對行數.
db.collection.count()

9.啟動日誌
2016-10-11T16:42:51.520+0800 I CONTROL  [initandlisten] MongoDB starting : pid=633 port=27016 dbpath=/data/mongodb/mongodb1/data 64-bit host=opsdba-vbj01-1
2016-10-11T16:42:51.520+0800 I CONTROL  [initandlisten] db version v3.2.8
2016-10-11T16:42:51.520+0800 I CONTROL  [initandlisten] git version: ed70e33130c977bda0024c125b56d159573dbaf0
2016-10-11T16:42:51.520+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-10-11T16:42:51.521+0800 I CONTROL  [initandlisten] allocator: tcmalloc
2016-10-11T16:42:51.521+0800 I CONTROL  [initandlisten] modules: none
2016-10-11T16:42:51.521+0800 I CONTROL  [initandlisten] build environment:
2016-10-11T16:42:51.521+0800 I CONTROL  [initandlisten]     distmod: rhel62
2016-10-11T16:42:51.521+0800 I CONTROL  [initandlisten]     distarch: x86_64
2016-10-11T16:42:51.521+0800 I CONTROL  [initandlisten]     target_arch: x86_64
2016-10-11T16:42:51.521+0800 I CONTROL  [initandlisten] options: { config: “/data/mongodb/mongodb1/conf/mongod.cnf”, net: { http: { enabled: false }, maxIncomingConnections: 3000, port: 27016, unixDomainSocket: { pathPrefix: “/data/mongodb/mongodb1/data” } }, operationProfiling: { mode: “slowOp”, slowOpThresholdMs: 800 }, processManagement: { fork: true, pidFilePath: “/data/mongodb/mongodb1/data/mongod.pid” }, replication: { replSet: “myrelset” }, security: { clusterAuthMode: “keyFile”, keyFile: “/data/mongodb/mongodb1/conf/myrelset.keyfile” }, storage: { dbPath: “/data/mongodb/mongodb1/data”, directoryPerDB: true, engine: “wiredTiger”, journal: { commitIntervalMs: 300, enabled: true }, mmapv1: { nsSize: 32 }, repairPath: “/data/mongodb/mongodb1/data”, syncPeriodSecs: 60.0, wiredTiger: { engineConfig: { cacheSizeGB: 1 } } }, systemLog: { destination: “file”, path: “/data/mongodb/mongodb1/log/mongod.log”, quiet: true, timeStampFormat: “iso8601-local” } }
2016-10-11T16:42:51.521+0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-10-11T16:42:51.944+0800 I STORAGE  [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs
2016-10-11T16:42:51.944+0800 I STORAGE  [initandlisten] The size storer reports that the oplog contains 1 records totaling to 45 bytes
2016-10-11T16:42:51.944+0800 I STORAGE  [initandlisten] Scanning the oplog to determine where to place markers for truncation
2016-10-11T16:42:51.982+0800 W STORAGE  [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger
2016-10-11T16:42:51.982+0800 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2016-10-11T16:42:51.982+0800 I CONTROL  [initandlisten]
2016-10-11T16:42:51.982+0800 I CONTROL  [initandlisten]
2016-10-11T16:42:51.982+0800 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 1024 processes, 65535 files. Number of processes should be at least 32767.5 : 0.5 times number of files.
2016-10-11T16:42:52.030+0800 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory `/data/mongodb/mongodb1/data/diagnostic.data`
2016-10-11T16:42:52.030+0800 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2016-10-11T16:42:52.031+0800 I NETWORK  [initandlisten] waiting for connections on port 27016
2016-10-11T16:42:52.066+0800 I REPL     [ReplicationExecutor] New replica set config in use: { _id: “myrelset”, version: 5, protocolVersion: 1, members: [ { _id: 1, host: “opsdba-vbj01-1:27018”, arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: “opsdba-vbj01-1:27019”, arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 3, host: “opsdba-vbj01-1:27016”, arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId(`57bfdcdcd40cbe4bf173396a`) } }
2016-10-11T16:42:52.067+0800 I REPL     [ReplicationExecutor] This node is opsdba-vbj01-1:27016 in the config
2016-10-11T16:42:52.067+0800 I REPL     [ReplicationExecutor] transition to STARTUP2
2016-10-11T16:42:52.067+0800 I REPL     [ReplicationExecutor] Starting replication applier threads
2016-10-11T16:42:52.067+0800 I REPL     [ReplicationExecutor] transition to RECOVERING
2016-10-11T16:42:52.072+0800 I REPL     [ReplicationExecutor] transition to SECONDARY
2016-10-11T16:42:52.102+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Successfully connected to opsdba-vbj01-1:27018
2016-10-11T16:42:52.102+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Successfully connected to opsdba-vbj01-1:27019
2016-10-11T16:42:52.102+0800 I REPL     [ReplicationExecutor] Member opsdba-vbj01-1:27018 is now in state SECONDARY
2016-10-11T16:42:52.103+0800 I REPL     [ReplicationExecutor] Member opsdba-vbj01-1:27019 is now in state PRIMARY
2016-10-11T16:42:58.069+0800 I REPL     [ReplicationExecutor] syncing from: opsdba-vbj01-1:27018
2016-10-11T16:42:58.086+0800 I REPL     [SyncSourceFeedback] setting syncSourceFeedback to opsdba-vbj01-1:27018
2016-10-11T16:42:58.104+0800 I ASIO     [NetworkInterfaceASIO-BGSync-0] Successfully connected to opsdba-vbj01-1:27018