GoldenGate實時投遞資料到大資料平臺(1)-MongoDB

margiex發表於2017-12-21

mongodb安裝

安裝

linux下可使用apt-get install mongodb-server 或 yum install mongodb-server 進行安裝。

也可以在windows上安裝mongodb,只要網路連通,資料就可以從關係型資料庫投遞到mongodb。詳細安裝文件可網路搜尋。

啟動mongodb

先建立一個資料目錄

mkdir /opt/mongodb-data

然後啟動DB服務, --rest是啟動restful web

mongod --dbpath /opt/mongodb-data -–reset

clip_image002

可以通過restful admin web訪問mongodb,如下

http://192.168.89.129:28017/

clip_image004

基本使用

插入示例資料並查詢

clip_image006

簡單統計test db.

clip_image007

OGG安裝測試

以下使用ogg12.3 for big data介質進行資料投遞,下載地址:

http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

ogg for bigdata需要JDK 1.8及以上,安裝完成之後,需要設定LD_LIBRARY_PATH環境變數

export LD_LIBRARY_PATH=/u01/jdk1.8.0_111/jre/lib/amd64/server

OGG for bigdata解壓即可安裝,安裝完成之後,拷貝安裝目錄下自帶的示例到dirprm目錄。

[oracle@ol73 mongodb]$ pwd

/u01/ogg4bd_12.3/AdapterExamples/big-data/mongodb

[oracle@ol73 mongodb]$ cp * /u01/ogg4bd_12.3/dirprm/

修改dirprm/mongo.props檔案,主要調整mongodb jdbc jar的路徑和client URI

gg.handlerlist=mongodb

gg.handler.mongodb.type=mongodb

#The following handler properties are optional.

#Please refer to the Oracle GoldenGate for BigData documentation

#for details about the configuration.

gg.handler.mongodb.clientURI=mongodb://192.168.89.129:27017/

goldengate.userexit.timestamp=utc

goldengate.userexit.writers=javawriter

javawriter.stats.display=TRUE

javawriter.stats.full=TRUE

gg.log=log4j

gg.log.level=INFO

gg.report.time=30sec

#Path to MongoDB Java driver.

gg.classpath=/u01/ogg4bd_12.3/mongo-java-driver-3.4.0.jar

javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm

 


OGG投遞程式引數,可直接使用示例自帶的引數檔案

GGSCI (ol73) 1> view param rmongo

REPLICAT rmongo

-- Trail file for this example is located in "AdapterExamples/trail" directory

-- Command to add REPLICAT

-- add replicat rmongo, exttrail AdapterExamples/trail/tr

TARGETDB LIBFILE libggjava.so SET property=dirprm/mongo.props

REPORTCOUNT EVERY 1 MINUTES, RATE

GROUPTRANSOPS 1000

MAP QASOURCE.*, TARGET QASOURCE.*;


使用OGG自帶的增量資料建立投遞程式

GGSCI>add replicat rmongo, exttrail AdapterExamples/trail/tr

啟動投遞程式

GGSCI (ol73) 3> start rmongo

Sending START request to MANAGER ...

REPLICAT RMONGO starting

統計變更資料

GGSCI (ol73) 9> stats rmongo, total

Sending STATS request to REPLICAT RMONGO ...

Start of Statistics at 2017-12-20 17:38:34.

Replicating from QASOURCE.TCUSTMER to QASOURCE.TCUSTMER:

*** Total statistics since 2017-12-20 17:38:18 ***

Total inserts 5.00

Total updates 1.00

Total deletes 0.00

Total discards 0.00

Total operations 6.00

Replicating from QASOURCE.TCUSTORD to QASOURCE.TCUSTORD:

*** Total statistics since 2017-12-20 17:38:18 ***

Total inserts 5.00

Total updates 3.00

Total deletes 2.00

Total discards 0.00

Total operations 10.00

End of Statistics.

可以看到變更資料中有2張表,包括有insert/update/delete等操作。

在資料庫端驗證結果

$mongo

檢視資料庫,已經看到有新插入的QASOURCE

> show dbs

QASOURCE 0.078GB

admin (empty)

local 0.078GB

test 0.078GB

> use QASOURCE

switched to db QASOURCE

簡單檢視一下資料庫的基本資訊

> db.stats()

{

"db" : "QASOURCE",

"collections" : 4,

"objects" : 15,

"avgObjSize" : 116.26666666666667,

"dataSize" : 1744,

"storageSize" : 32768,

"numExtents" : 4,

"indexes" : 2,

"indexSize" : 16352,

"fileSize" : 67108864,

"nsSizeMB" : 16,

"dataFileVersion" : {

"major" : 4,

"minor" : 5

},

"extentFreeList" : {

"num" : 0,

"totalSize" : 0

},

"ok" : 1

}

 


檢視一下資料庫中的集合

> db.getCollectionInfos()

[

{

"name" : "TCUSTMER"

},

{

"name" : "TCUSTORD"

},

{

"name" : "system.indexes"

}

]

 


> db.getCollectionNames()

[ "TCUSTMER", "TCUSTORD", "system.indexes" ]

>

> db.getCollectionNames()

[ "TCUSTMER", "TCUSTORD", "system.indexes" ]

檢視TCUSTMER中的資料

> db.TCUSTMER.find()

{ "_id" : "WILL", "CITY" : "SEATTLE", "STATE" : "WA", "CUST_CODE" : "WILL", "NAME" : "BG SOFTWARE CO." }

{ "_id" : "JANE", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "JANE", "NAME" : "ROCKY FLYER INC." }

{ "_id" : "DAVE", "CITY" : "TALLAHASSEE", "STATE" : "FL", "CUST_CODE" : "DAVE", "NAME" : "DAVE'S PLANES INC." }

{ "_id" : "BILL", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "BILL", "NAME" : "BILL'S USED CARS" }

{ "_id" : "ANN", "CITY" : "NEW YORK", "STATE" : "NY", "CUST_CODE" : "ANN", "NAME" : "ANN'S BOATS" }


查詢CITY=’DENVER’的記錄

> db.TCUSTMER.find({"CITY":"DENVER"})

{ "_id" : "JANE", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "JANE", "NAME" : "ROCKY FLYER INC." }

{ "_id" : "BILL", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "BILL", "NAME" : "BILL'S USED CARS" }

測試完成。

相關文章