Logstash7.6.2同步Mysql資料到ElasticSearch

表演給自己看的認真發表於2020-07-02
MySqlElasticsearch

1、準備工作：存在的mysql的資料庫表、建立好的elasticsearch索引

2、下載mysql-connector

下載地址：https://dev.mysql.com/downloads/connector/j/ 

選擇 PlatForm Independent

下載後解壓後得到connector的jar包 

3、新建配置檔案 sync_tablename.conf，

input {
    stdin{
    }
    jdbc {
      # 連線的資料庫地址和資料庫，指定編碼格式，禁用SSL協議，設定自動重連
      jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/userdb?characterEncoding=UTF-8&useSSL=false&autoReconnect=true"
      # 使用者名稱密碼
      jdbc_user => "user"
      jdbc_password => "password"
      # jar包的位置
      jdbc_driver_library => "/u02/tomcat/mysql-connector-java-8.0.20/mysql-connector-java-8.0.20.jar"
      # mysql的Driver
      jdbc_driver_class => "com.mysql.jdbc.Driver"
      jdbc_default_timezone => "Asia/Shanghai"
      jdbc_paging_enabled => "true"
      jdbc_page_size => "10"
      #statement_filepath => "config-mysql/test.sql"
      #注意這個sql不能出現type，這是es的保留欄位
      statement => "select * from user where name='張三' "
      schedule => "* * * * *"
    }
}
output {
    elasticsearch {
        hosts => "127.0.0.1:9200"
        # index名
        index => "user"
        # type名
        document_type => "_doc"
        # 需要關聯的資料庫中有有一個id欄位，對應索引的id號
        document_id => "%{id}"
    }
    stdout {
        codec => json_lines
    }
}

啟動報錯記錄1： [logstash.config.source.local.configpathloader] No config files found in path {:path=>"/u02/tomcat/logstash-7.6.2/bin/config/sync_tablename.conf"}                     ./logstash -f config/sync_tablename.conf

        報錯原因：啟動的時候找不到指定路徑

        解決方法：可以放在bin目錄下，啟動時不需要指定路徑；          ./logstash -f  sync_tablename.conf

啟動報錯記錄2：[FATAL][logstash.runner          ] The given configuration is invalid. Reason: Expected one of [ \t\r\n], "#", "{", [A-Za-z0-9_-], '"', "'", "}" at line 17, column 1 (byte 618) after input {

        報錯原因：配置檔案中17行有錯誤

        解決方法：檢查配置檔案的內容，是否合規

啟動報錯記錄3：

[ERROR][logstash.inputs.jdbc     ] Invalid setting for jdbc input plugin:

  input {
    jdbc {
      # This setting must be a number
      # Expected number, got "" (type )
      jdbc_page_size => ""
      ...
    }
  }

        報錯原因：jdbc_page_size必須是一個數字

        解決方法：jdbc_page_size => "10"

#驗證配置檔案命令：
./logstash -f sync_tablename.conf --config.test_and_exit
#顯示內容表示配置檔案正確
Configuration OK
[INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

 4、啟動命令進行資料同步   ./logstash -f sync_tablename.conf

啟動報錯記錄1：

Error: unable to load /home/u02/tomcat/logstash-7.6.2/bin/config-mysql/mysql-connector-java-8.0.20/mysql-connector-java-8.0.20.jar from :jdbc_driver_library, file not readable (please check user and group permissions for the path)
  Exception: LogStash::PluginLoadingError

        報錯原因：外掛載入失敗，mysql-connector-java外掛路徑配置錯誤，

        解決方法：去掉/home，改為    /u02/tomcat/logstash-7.6.2/bin/config-mysql/mysql-connector-java-8.0.20/mysql-connector-java-8.0.20.jar

啟動報錯記錄2：

[WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"%{id}", :_index=>"user", :routing=>nil, :_type=>"doc"}, #<LogStash::Event:0x4d04559>], :response=>{"index"=>{"_index"=>"user", "_type"=>"doc", "_id"=>"%{id}", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [user] as the final mapping would have more than 1 type: [_doc, doc]"}}}}
{"@timestamp":"2020-07-02T10:11:23.182Z","message":"","host":"127.0.0.1","@version":"1"}

        報錯原因：無法將事件索引到Elasticsearch，拒絕對[user]的對映更新，因為最終對映將有一個以上的型別：[_doc，doc]”

        解決方法：配置檔案中的type要和elasticSearch中的type保持一致

相關文章