mysql同步es，秒級
https://golang.org/dl/
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
1,安裝mysql映象
docker pull mysql:5.7.21
執行映象
sudo docker run --name mysql -p 33061:3306 -e MYSQL_ROOT_PASSWORD=root -d mysql:5.7.21
2、進入到mysql docker容器裡：
安裝vim
apt-get update
apt-get install -y vim
修改mysql配置檔案：vi /etc/mysql/mysql.conf.d/mysqld.cnf
將bind_address的值從127.0.0.1修改成0.0.0.0
bind_address=0.0.0.0
server-id=123454
log-bin=/var/lib/mysql/mysql-bin
binlog_format=ROW
3、登入到mysql中，執行下面的授權命令：
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%'IDENTIFIED BY 'root' WITH GRANT OPTION;
FLUSH PRIVILEGES;
4、退出資料庫，執行下面的命令重啟mysql:
service mysql restart
sudo docker restart mysql
5,檢視binlog日誌的狀態
sudo docker exec -it mysql bash
mysql -uroot -proot
show variables like '%log_bin%';
CREATE DATABASE IF NOT EXISTS RUNOOB DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
use RUNOOB;
CREATE TABLE IF NOT EXISTS `runoob_tbl`(
`runoob_id` INT UNSIGNED AUTO_INCREMENT,
`runoob_title` VARCHAR(100) NOT NULL,
`runoob_author` VARCHAR(40) NOT NULL,
`submission_date` DATE,
PRIMARY KEY ( `runoob_id` )
)ENGINE=InnoDB DEFAULT CHARSET=utf8;
INSERT INTO runoob_tbl (runoob_title, runoob_author, submission_date) VALUES ("學習 PHP", "菜鳥教程", NOW());
INSERT INTO runoob_tbl (runoob_title, runoob_author, submission_date) VALUES ("學習 MySQL", "菜鳥教程", NOW());
INSERT INTO runoob_tbl (runoob_title, runoob_author, submission_date) VALUES ("huawei cloud", "RUNOOB.COM", '2018-10-10');
221.130.253.135
221.130.253.135
proxy.cmcc
http://127.0.0.1:1080/pac?auth=hj6b8UvtvtQ1yNVKTomR&t=201811071416083924
POST nfvocatalog/_delete_by_query
{
"query": {
"range" : {
"asctime" : {
"lt" : "now-40d"
}
}
}
}
output {
stdout { codec => rubydebug }
}
convert => { "allocatedcapacity" => "float" }
convert => { "allocatedratio" => "float" }
TimeStamp=2018-11-02 09:18:16|TimeZone=UTC+8|Period=15|VendorName=Huawei|ElementType=CSCF|PmVersion=V3.2.1|ObjectType=EthernetPort
rmUID|Dn|UserLabel|StartTime
3301HWWX1EPO1003221569541|DC=www.huawei.com,SubNetwork=1,ManagedElement=1924,EthernetPort=vISBG91_CSDPU_0_VNIC1|2018-11-02 09:00:00|vISBG91_CSDPU_0_VNIC1
FROM ubuntu:16.04
sudo apt install -y golang
FROM ubuntu:16.04
RUN apt-get update && \
apt-get install -y openjdk-8-jdk && \
apt-get install -y net-tools && \
apt-get install -y vim && \
apt-get install -y wget && \
apt-get install -y unzip && \
apt-get install -y curl && \
apt-get install -y tar && \
apt-get install -y gzip && \
apt install -y golang
WORKDIR /elk
ADD . /elk
RUN wget -q -O logstash-6.4.2.tar.gz 'https://artifacts.elastic.co/downloads/logstash/logstash-6.4.2.tar.gz' && \
tar -xvf logstash-6.4.2.tar.gz && \
rm -rf logstash-6.4.2.tar.gz && \
mv logstash.yml /elk/logstash-6.4.2/config/ && \
mv c1.conf /elk/logstash-6.4.2/ && \
chmod 777 /elk/entrypoint.sh && \
rm /elk/Dockerfile && \
wget http://central.maven.org/maven2/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
EXPOSE 5044
ENTRYPOINT /elk/entrypoint.sh
CM
TimeStamp|TimeZone|VendorName|ElementType|CmVersion|ObjectType|rmUID|Dn|UserLabel|PVFlag|VNFInstanceID|VMID
TimeStamp|TimeZone|VendorName|ElementType|CmVersion|ObjectType|rmUID|Dn|UserLabel|PVFlag|VNFInstanceID
TimeStamp|TimeZone|VendorName|ElementType|CmVersion|ObjectType|rmUID|Dn|UserLabel|PVFlag
TimeStamp|TimeZone|VendorName|ElementType|CmVersion|ObjectType|rmUID|Dn|UserLabel|PVFlag|VMID
把
TimeStamp|TimeZone|VendorName|ElementType|CmVersion|ObjectType|rmUID|Dn|UserLabel|PVFlag|VMID
改成
TimeStamp|TimeZone|VendorName|ElementType|CmVersion|ObjectType|rmUID|Dn|UserLabel|PVFlag|-|-|VMID
TimeStamp=2014-01-07T00:17:01|TimeZone=UTC+8|VendorName=Huawei|ElementType=CSCF|CmVersion=V3.1.4|ObjectType=EpRpDynCxScscf|rmUID=3301HWWX1CXS1003227406336|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1924,ScscfFunction=0,EpRpDynCxScscf=21_59|UserLabel=HZVSCSCF91BHW_TO_HZHSS07BHW_01|PVFlag=HZVSCSCF91BHW_TO_HZHSS07BHW_01|-|-|VMID=hello7
TimeStamp=2014-01-08T00:17:01|TimeZone=UTC+8|VendorName=Huawei|ElementType=CSCF|CmVersion=V3.1.4|ObjectType=EpRpDynCxScscf|rmUID=3301HWWX1CXS1003227406336|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1924,ScscfFunction=0,EpRpDynCxScscf=21_59|UserLabel=HZVSCSCF91BHW_TO_HZHSS07BHW_01|PVFlag=HZVSCSCF91BHW_TO_HZHSS07BHW_01
TimeStamp=2014-01-09T00:17:01|TimeZone=UTC+8|VendorName=Huawei|ElementType=CSCF|CmVersion=V3.1.4|ObjectType=EpRpDynCxScscf|rmUID=3301HWWX1CXS1003227406336|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1924,ScscfFunction=0,EpRpDynCxScscf=21_59|UserLabel=HZVSCSCF91BHW_TO_HZHSS07BHW_01|PVFlag=HZVSCSCF91BHW_TO_HZHSS07BHW_01|VNFInstanceID=hello7
TimeStamp=2014-01-10T00:17:01|TimeZone=UTC+8|VendorName=Huawei|ElementType=CSCF|CmVersion=V3.1.4|ObjectType=EpRpDynCxScscf|rmUID=3301HWWX1CXS1003227406336|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1924,ScscfFunction=0,EpRpDynCxScscf=21_59|UserLabel=HZVSCSCF91BHW_TO_HZHSS07BHW_01|PVFlag=HZVSCSCF91BHW_TO_HZHSS07BHW_01|VNFInstanceID=hello7|VMID=hello7
TimeStamp=%{TIMESTAMP_ISO8601:timestamp}\|TimeZone=(?<timezone>[^|]*)\|VendorName=(?<vendorname>[^|]*)\|ElementType=(?<elementtype>[^|]*)\|CmVersion=(?<cmversion>[^|]*)\|ObjectType=(?<objecttype>[^|]*)\|rmUID=(?<rmuid>[^|]*)\|Dn=(?<dn>[^|]*)\|UserLabel=(?<userlabel>[^|]*)\|PVFlag=(?<pvflag>[^|]*)\|VMID=(?<vmid>[^|]*)
TimeStamp=2016-11-12T00:08:50|TimeZone=UTC+8|VendorName=Huawei|ElementType=GGSN|CmVersion=V4.0.1|ObjectType=EthernetPort|rmUID=1100HWWX1EPO3221565467|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1,EthernetPort=Ethernet69/0/2|UserLabel=Ethernet69/0/2|PVFlag=VNF|VNFInstanceID=hellovnf|VMID=hellovimid
TimeStamp=2016-11-11T00:08:50|TimeZone=UTC+8|VendorName=Huawei|ElementType=GGSN|CmVersion=V4.0.1|ObjectType=EthernetPort|rmUID=1100HWWX1EPO3221565467|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1,EthernetPort=Ethernet69/0/2|UserLabel=Ethernet69/0/2|PVFlag=VNF|VMID=hellovimid
TimeStamp=2016-11-10T00:08:50|TimeZone=UTC+8|VendorName=Huawei|ElementType=GGSN|CmVersion=V4.0.1|ObjectType=EthernetPort|rmUID=1100HWWX1EPO3221565467|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1,EthernetPort=Ethernet69/0/2|UserLabel=Ethernet69/0/2|PVFlag=VNF|VNFInstanceID=helloVNFInstanceID
TimeStamp=2016-11-09T00:08:50|TimeZone=UTC+8|VendorName=Huawei|ElementType=GGSN|CmVersion=V4.0.1|ObjectType=EthernetPort|rmUID=1100HWWX1EPO3221565467|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1,EthernetPort=Ethernet69/0/2|UserLabel=Ethernet69/0/2|PVFlag=hellopvflag
TimeStamp=2015-10-08T00:02:50|TimeZone=UTC+8|VendorName=Huawei||ElementType=GGSN|CmVersion=V4.0.1|ObjectType=EthernetPort|rmUID=1100HWWX1EPO3221565467|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1,EthernetPort=Ethernet69/0/2|UserLabel=Ethernet69/0/2|PVFlag=VNF|VMID=de2f2dc95c104bfa9eb5c8cd524315d9,6b3dd95b06c34d89bd1d76646d3cd1d3,5e5ad570965046e1871a03f9f470e08a,fb3bd5a0d6d94c3ab9a1d7806dcf8ffa,506e626b82df4f89bf64ddc839480cf3,0250f78fc4614672b5b6d35e14cb299f,a96e6179853b4faeb1ddb329cc4c5718,e99c717bc7a44c779aad2da442750a4e
TimeStamp=2015-10-09T00:02:50|TimeZone=UTC+8|VendorName=Huawei||ElementType=GGSN|CmVersion=V4.0.1|ObjectType=EthernetPort|rmUID=1100HWWX1EPO3221565467|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1,EthernetPort=Ethernet69/0/2|UserLabel=Ethernet69/0/2|PVFlag=VNF||VMID=de2f2dc95c104bfa9eb5c8cd524315d9,6b3dd95b06c34d89bd1d76646d3cd1d3,5e5ad570965046e1871a03f9f470e08a,fb3bd5a0d6d94c3ab9a1d7806dcf8ffa,506e626b82df4f89bf64ddc839480cf3,0250f78fc4614672b5b6d35e14cb299f,a96e6179853b4faeb1ddb329cc4c5718,e99c717bc7a44c779aad2da442750a4e
CM-OSS-VSDATACONTAINER-02-V3.2.0-20181102000000.csv
CM-OSS-VSDATACONTAINER-02-V3.2.0-201811.csv
%{TIMESTAMP_ISO8601:timestamp}\|(?<timezone>[^|]*)\|(?<vendorname>[^|]*)\|(?<elementtype>[^|]*)\|(?<cmversion>[^|]*)\|(?<objecttype>[^|]*)\|(?<rmuid>[^|]*)\|(?<dn>[^|]*)\|(?<userlabel>[^|]*)\|(?<pvflag>[^|]*)\|(?<vnfinstanceid>[^|]*)
%{TIMESTAMP_ISO8601:timestamp}\|(?<timezone>[^|]*)\|(?<vendorname>[^|]*)\|(?<elementtype>[^|]*)\|(?<cmversion>[^|]*)\|(?<objecttype>[^|]*)\|(?<rmuid>[^|]*)\|(?<dn>[^|]*)\|(?<userlabel>[^|]*)\|(?<pvflag>[^|]*)
TimeStamp=2014-01-02T00:17:01|TimeZone=UTC+8|VendorName=Huawei|ElementType=CSCF|CmVersion=V3.1.4|ObjectType=EpRpDynCxScscf|rmUID=3301HWWX1CXS1003227406336|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1924,ScscfFunction=0,EpRpDynCxScscf=21_59|UserLabel=HZVSCSCF91BHW_TO_HZHSS07BHW_01|PVFlag=HZVSCSCF91BHW_TO_HZHSS07BHW_01||VMID=hellovmid
TimeStamp=2014-01-03T00:17:01|TimeZone=UTC+8|VendorName=Huawei|ElementType=CSCF|CmVersion=V3.1.4|ObjectType=EpRpDynCxScscf|rmUID=3301HWWX1CXS1003227406336|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1924,ScscfFunction=0,EpRpDynCxScscf=21_59|UserLabel=HZVSCSCF91BHW_TO_HZHSS07BHW_01|PVFlag=HZVSCSCF91BHW_TO_HZHSS07BHW_01|-|-|VMID=hellovimd
TimeStamp|TimeZone|VendorName|ElementType|CmVersion|ObjectType|rmUID|Dn|UserLabel|PVFlag|VNFInstanceID|VMID
TimeStamp=2018-11-02T00:17:01|TimeZone=UTC+8|VendorName=Huawei|ElementType=CSCF|CmVersion=V3.1.4|ObjectType=EpRpDynCxScscf|rmUID=3301HWWX1CXS1003227406336|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1924,ScscfFunction=0,EpRpDynCxScscf=21_59|UserLabel=HZVSCSCF91BHW_TO_HZHSS07BHW_01|PVFlag=VNF|VNFInstanceID=kong|VMID=25d80372-acd6-4d67-b598-0f840c307767,233a9407-4c86-4319-8557-6d9f4bb098d8,aeab237a-9af4-4ae9-b3be-f09b342b9b3d,ada0920f-7deb-4ccc-a2d1-7cd69ed9d2e1,48269b3f-19aa-405f-be4c-939b04217bb2,1c749557-9d45-448e-83a9-5197bde05c88,059e1244-08b9-40d8-99b0-cfb3ac00d005,f936f218-f2e7-4ba6-a9bf-b8c05966ed78,c299d819-7ccc-44fb-aee4-63b1fa09ec71,30ffb959-da47-4f9e-899a-b6f0228da1e8
1,安裝mysql映象
docker pull mysql:5.7.21
執行映象
sudo docker run --name mysql -e MYSQL_ROOT_PASSWORD=root -d mysql:5.7.21
2、進入到mysql docker容器裡：
安裝vim
apt-get update
apt-get install -y vim
修改mysql配置檔案：vi /etc/mysql/mysql.conf.d/mysqld.cnf
將bind_address的值從127.0.0.1修改成0.0.0.0
bind_address=0.0.0.0
server-id = 123454
log-bin = /var/lib/mysql/mysql-bin
binlog_format = ROW
expire_logs_days = 7
/var/log/mysql/mariadb-bin.index
3、登入到mysql中，執行下面的授權命令：
GRANT ALL PRIVILEGES ON . TO 'root'@'%'IDENTIFIED BY 'root' WITH GRANT OPTION;
FLUSH PRIVILEGES;
4、退出資料庫，執行下面的命令重啟mysql:
service mysql restart
sudo docker restart mysql
5,檢視binlog日誌的狀態
sudo docker exec -it mysql bash
mysql -uroot -proot
show variables like '%log_bin%';
CREATE DATABASE IF NOT EXISTS RUNOOB DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
use RUNOOB;
CREATE TABLE IF NOT EXISTS runoob(runoob_id INT UNSIGNED AUTO_INCREMENT,runoob_title VARCHAR(100) NOT NULL,runoob_author VARCHAR(40) NOT NULL,submission_date DATE,submission_datetime DATETIME,submission_timestamp TIMESTAMP,timeint1 INT,timeint2 INT,
PRIMARY KEY ( runoob_id )
)ENGINE=InnoDB DEFAULT CHARSET=utf8;
select * from runoob;
INSERT INTO runoob (runoob_title, runoob_author, submission_date,submission_datetime,submission_timestamp,timeint1,timeint2) VALUES ("學習scala", "菜鳥教程01", "2018-11-01","2018-11-01 10:10:10","2018-11-01 10:10:10",1000,1000);
[[source]]
schema = "RUNOOB"
tables = ["runoob"]
[[rule]]
schema = "RUNOOB"
table = "runoob"
index = "gomysql"
type = "go1"
id = ["runoob_id"]
submission_date=",date"
submission_datetime=",date"
submission_timestamp=",date"
timeint1=",date"
INSERT INTO runoob_tbl (runoob_title, runoob_author, submission_date) VALUES ("學習 MySQL", "菜鳥教程", NOW());
INSERT INTO runoob_tbl (runoob_title, runoob_author, submission_date) VALUES ("huawei cloud", "RUNOOB.COM", '2018-10-10');
TimeStamp=2018-10-14T00:02:51|TimeZone=UTC+8|VendorName=Huawei|ElementType=PGW|CmVersion=V3.0.0|ObjectType=EthernetPort|rmUID=1100HWWX1EPO3221565467|Dn=DC=www.huawei.com,SubNetwork=1,ManagedElement=1,EthernetPort=Ethernet69/0/2|UserLabel=Ethernet69/0/2|PVFlag=VNF|-|-|VMID=de2f2dc95c104bfa9eb5c8cd524315d9,6b3dd95b06c34d89bd1d76646d3cd1d3,5e5ad570965046e1871a03f9f470e08a,fb3bd5a0d6d94c3ab9a1d7806dcf8ffa,506e626b82df4f89bf64ddc839480cf3,0250f78fc4614672b5b6d35e14cb299f,a96e6179853b4faeb1ddb329cc4c5718,e99c717bc7a44c779aad2da442750a4e
use nfvofcaps;
select * from FMALARM;
CREATE DATABASE IF NOT EXISTS nfvofcaps DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
CREATE TABLE `FMALARM` (
`ORIGIN` varchar(255) NOT NULL,
`ID` varchar(255) NOT NULL,
`ALARMID` varchar(255) NOT NULL,
`ALARMTITLE` varchar(255) NOT NULL,
`ALARMSTATUS` integer NOT NULL,
`ORIGSEVERITY` integer NOT NULL,
`ALARMTYPE` varchar(255) NOT NULL,
`EVENTTIME` varchar(255) NOT NULL,
`MSGSEQ` varchar(255) NOT NULL,
`CLEARTIME` varchar(255),
`CLEARFLAG` integer,
`CLEARMSGSEQ` varchar(255),
`SPECIFICPROBLEMID` varchar(255) NOT NULL,
`SPECIFICPROBLEM` varchar(255) NOT NULL,
`NEUID` varchar(255) NOT NULL,
`NENAME` varchar(255) NOT NULL,
`NETYPE` varchar(255) NOT NULL,
`OBJECTUID` varchar(255) NOT NULL,
`OBJECTNAME` varchar(255) NOT NULL,
`OBJECTTYPE` varchar(255) NOT NULL,
`LOCATIONINFO` varchar(255) NOT NULL,
`ADDINFO` varchar(255),
`PVFLAG` varchar(255) NOT NULL,
`CONFIRMFLAG` integer,
`CONFIRMTIME` varchar(255),
`REMARK` varchar(255),
`REMARKTIME` varchar(255),
PRIMARY KEY(ID,ALARMID)
);
mysql下的nfvofcaps的表FMALARM中的4個欄位REMARKTIME,CONFIRMTIME,CLEARTIME,ALARMTITLE
現在的型別是varchar(255),
用go-mysql-elasticsearch匯入ES中是text型別,沒辦法轉到ES的date型別,
原來用logstash的jdbc外掛可以轉換成ES的date型別
楊溢是按照date型別寫的介面,如果是ES的text型別,需要重新寫介面
經過測試,mysql的DATETIME,TIMESTAMP型別匯入後,是ES的date型別
詢問下,你這裡的時間型別改成DATETIME方便不方便,改動大不大?
這個需要和楊溢,我們們再一塊商量下
use nfvofcaps;
select * from FMALARM;
INSERT INTO VIMFMALARM (ORIGIN,ID,ALARMID,ALARMTITLE,ALARMSTATUS,ORIGSEVERITY,ALARMTYPE,EVENTTIME,MSGSEQ,CLEARTIME,CLEARFLAG,CLEARMSGSEQ,SPECIFICPROBLEMID,SPECIFICPROBLEM,NEUID,NENAME,NETYPE,OBJECTUID,OBJECTNAME,OBJECTTYPE,LOCATIONINFO,ADDINFO,PVFLAG,CONFIRMFLAG,CONFIRMTIME,REMARK,REMARKTIME) VALUES ("虛擬層","81f1d9d0-ca13-4eea-a4ce-9bd89a50c9c3","251","Disk Array Controller 1 Failed!",1,2,"da","2018-06-19 21:12:00",283,null,null,null,"3712.31","Hardware Error","129","DiskArray-120","diskArray","4350","Controller-0","Controller","","","vim",1,"2018-09-19 21:12:00","已確認告警","2018-09-30 21:12:00");
INSERT INTO FMALARM (ORIGIN,ID,ALARMID,ALARMTITLE,ALARMSTATUS,ORIGSEVERITY,ALARMTYPE,EVENTTIME,MSGSEQ,CLEARTIME,CLEARFLAG,CLEARMSGSEQ,SPECIFICPROBLEMID,SPECIFICPROBLEM,NEUID,NENAME,NETYPE,OBJECTUID,OBJECTNAME,OBJECTTYPE,LOCATIONINFO,ADDINFO,PVFLAG,CONFIRMFLAG,CONFIRMTIME,REMARK,REMARKTIME) VALUES ("虛擬層","120004","251","Disk Array Controller 1 Failed!",1,2,"da","2018-06-19 21:12:00",283,null,null,null,"3712.31","Hardware Error","129","DiskArray-120","diskArray","4350","Controller-0","Controller","","","vim",1,"2018-11-11 21:12:00","已確認告警","2018-11-12 21:12:00");
delete from FMALARM where id=1004;
update FMALARM set origin="試一試" where id=1002;
ORIGIN,ID,ALARMID,ALARMTITLE,ALARMSTATUS,ORIGSEVERITY,ALARMTYPE,EVENTTIME,MSGSEQ,CLEARTIME,CLEARFLAG,CLEARMSGSEQ,SPECIFICPROBLEMID,SPECIFICPROBLEM,NEUID,NENAME,NETYPE,OBJECTUID,OBJECTNAME,OBJECTTYPE,LOCATIONINFO,ADDINFO,PVFLAG,CONFIRMFLAG,CONFIRMTIME,REMARK,REMARKTIME
[error] sync.go:486 index index: nfvomysql140, type: go1, id: 1011:251, status: 400, error: {"type":"mapper_parsing_exception","reason":"failed to parse [eventtime]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: \"2018-06-19 21:12:00\" is malformed at \" 21:12:00\""}}
[2018/11/14 07:31:07] [info] master.go:54 save position (mysql-bin.000001, 12373)
[2018/11/14 07:32:30] [error] sync.go:486 index index: nfvomysql140, type: go1, id: 1012:251, status: 400, error: {"type":"mapper_parsing_exception","reason":"failed to parse [eventtime]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: \"2018-06-19 21:12:00\" is malformed at \" 21:12:00\""}}
[2018/11/14 07:32:30] [info] master.go:54 save position (mysql-bin.000001, 13468)
"eventtime": "2018-06-19 21:12:00",
"eventtime": "2018-06-19T21:12:00.000Z",
my_addr = "192.168.106.215:3306"
my_user = "root"
my_pass = "root"
my_charset = "utf8"
es_addr = "192.168.106.215:9200"
data_dir = "./var"
stat_addr = "127.0.0.1:12800"
server_id = 1234
flavor = "mysql"
bulk_size = 128
flush_bulk_time = "200ms"
skip_no_pk_table = false
[[source]]
schema = "nfvofcaps"
tables = ["FMALARM"]
[[rule]]
schema = "nfvofcaps"
table = "FMALARM"
index = "nfvomysql180"
type = "go1"
id = ["ID","ALARMID"]
[rule.field]
ORIGIN="origin"
ID="id"
ALARMID="alarmid"
ALARMTITLE="alarmtitle"
ALARMSTATUS="alarmstatus"
ORIGSEVERITY="origseverity"
ALARMTYPE="alarmtype"
EVENTTIME="eventtime"
MSGSEQ="msgseq"
CLEARTIME="cleartime"
CLEARFLAG="clearflag"
CLEARMSGSEQ="clearmsgseq"
SPECIFICPROBLEMID="specificproblemid"
SPECIFICPROBLEM="specificproblem"
NEUID="neuid"
NENAME="nename"
NETYPE="netype"
OBJECTUID="objectuid"
OBJECTNAME="objectname"
OBJECTTYPE="objecttype"
LOCATIONINFO="locationinfo"
ADDINFO="addinfo"
PVFLAG="pvflag"
CONFIRMFLAG="confirmflag"
CONFIRMTIME="confirmtime"
REMARK="remark"
REMARKTIME="remarktime"
[rule.field]
ORIGIN="origin"
ID="id"
ALARMID="alarmid"
ALARMTITLE="alarmtitle"
ALARMSTATUS="alarmstatus"
ORIGSEVERITY="origseverity"
ALARMTYPE="alarmtype"
EVENTTIME="eventtime"
MSGSEQ="msgseq"
CLEARTIME="cleartime"
CLEARFLAG="clearflag"
CLEARMSGSEQ="clearmsgseq"
SPECIFICPROBLEMID="specificproblemid"
SPECIFICPROBLEM="specificproblem"
NEUID="neuid"
NENAME="nename"
NETYPE="netype"
OBJECTUID="objectuid"
OBJECTNAME="objectname"
OBJECTTYPE="objecttype"
LOCATIONINFO="locationinfo"
ADDINFO="addinfo"
PVFLAG="pvflag"
CONFIRMFLAG="confirmflag"
CONFIRMTIME="confirmtime"
REMARK="remark"
REMARKTIME="remarktime"
PUT /_template/template_1
{
"template" : ["nfvomysql"],
"order" : 0,
"mappings": {
"go1": {
"properties": {
"remarktime": {
"type": "date",
"format": "YYYY-MM-dd HH:mm:ss"
},
"confirmtime": {
"type": "date",
"format": "YYYY-MM-dd HH:mm:ss"
},
"cleartime": {
"type": "date",
"format": "YYYY-MM-dd HH:mm:ss"
},
"eventtime": {
"type": "date",
"format": "YYYY-MM-dd HH:mm:ss"
},
"neuid": {
"type": "long"
}
}
}
}
}
curl -XPUT "http://10.12.25.56:9200/nfvomysql" -H 'Content-Type: application/json' -d'
{
"mappings": {
"mysqltable": {
"properties": {
"remarktime": {
"type": "date",
"format": "YYYY-MM-dd HH:mm:ss"
},
"confirmtime": {
"type": "date",
"format": "YYYY-MM-dd HH:mm:ss"
},
"cleartime": {
"type": "date",
"format": "YYYY-MM-dd HH:mm:ss"
},
"eventtime": {
"type": "date",
"format": "YYYY-MM-dd HH:mm:ss"
}
}
}
}
}'
{
"index_patterns": ["te*", "bar*"],
"settings": {
"number_of_shards": 1
},
}
{
"nfvomysql": {
"mappings": {
"go": {
"properties": {
"addinfo": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"alarmid": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"alarmstatus": {
"type": "long"
},
"alarmtitle": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"alarmtype": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"confirmflag": {
"type": "long"
},
"confirmtime": {
"type": "date"
},
"eventtime": {
"type": "date"
},
"id": {
"type": "long"
},
"locationinfo": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"msgseq": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"nename": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"netype": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"neuid": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"objectname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"objecttype": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"objectuid": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"origin": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"origseverity": {
"type": "long"
},
"pvflag": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"remark": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"remarktime": {
"type": "date"
},
"specificproblem": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"specificproblemid": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
{
"nfvocatalog": {
"mappings": {
"log": {
"properties": {
"asctime": {
"type": "date"
},
"funcname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"input_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"levelname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"mdc": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"msg": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"myid": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"source": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"thread": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
apt install -y mariadb-client-10.0
# mysql or mariadb
flavor = "mariadb"
mysqldump = "mysqldump"
# mysqldump execution path
# if not set or empty, ignore mysqldump.
mysqldump = "mysqldump"
相關文章
- go-mysql-es同步elasticsearchGoMySqlElasticsearch
- mysql 如何毫秒級同步資料到 elasticsearchMySqlElasticsearch
- MySQL級聯複製中資料同步MySql
- Logstash同步ES
- docker搭建Elasticsearch、Kibana、Logstash 同步mysql資料到ESDockerElasticsearchMySql
- ES6---非同步非同步
- ES升級
- 京東雲開發者｜mysql基於binlake同步ES積壓解決方案MySql
- MySQL級聯複製的同步問題（一）MySql
- 國內衛星授時鐘同步精度被推至亞納秒級
- 同步秒殺實現：Redis在秒殺功能的實踐Redis
- es6 用generator進行非同步方法同步非同步
- mysql通過kafka實現資料實時同步（三）——es叢集配置MySqlKafka
- ES同步Mysql資料庫（包括出現問題怎麼處理哦）MySql資料庫
- MYSQL 同步到ES 如何設計架構保持一致性MySql架構
- 跨機房ES同步實戰
- 【NTP】手工同步NTP保證RAC 節點主機之間秒級一致
- mysql 時間型別秒級資料，取分鐘資料方案MySql型別
- ES7之async/await 同步還是非同步AI非同步
- ES6非同步方式全面解析非同步
- 重讀 ES6 - async+await 同步/非同步方案AI非同步
- 炸！億級資料DB秒級平滑擴容！！！
- 【深入淺出ES6】非同步Promise非同步Promise
- [FFmpeg + OpenGL + OpenSL ES]音視訊同步- 8
- ES6 Promise非同步程式設計Promise非同步程式設計
- ES6 Generators的非同步應用非同步
- MySQL級聯複製中的資料同步（第二篇）MySql
- ES-MySQL-PHP的使用MySqlPHP
- Mysql同步到GreenplumMySql
- Mysql半同步配置MySql
- mysql主主同步MySql
- mysql主從同步MySql主從同步
- Mysql主主同步-配置資料同步MySql
- OGG表級同步
- 【高併發】秒殺系統架構解密，不是所有的秒殺都是秒殺（升級版）！！架構解密
- EasyExcel處理Mysql百萬資料的匯入匯出案例，秒級效率，拿來即用！ExcelMySql
- Logstash mysql匯入es注意MySql
- Es6初級入門（一）