Elasticsearch資料庫 | Elasticsearch-7.5.0應用搭建實戰

PivotalCloud發表於2020-11-06

Elasticsearch 是一個可用於分散式以及符合RESTful 風格的搜尋和資料分析引擎。—— Elastic Stack 官網

0FNnW8.png

搭建Elasticsearch的“那些事兒"

BW1nxS.jpg

有一天,在黃金梅麗號的甲板上,韋柏告訴薩博,需要在接下來的專案開發過程中,運用到Elasticsearch資料庫,主要用於黃金梅麗號上的各種裝置採集資料實時查詢,或許後期還會運用於分散式日誌系統的搭建運用等,讓薩博先做一次技術預研。於是,在薩博查詢大量的資料發現,關於Elasticsearch資料庫的搭建,網上的資料幾乎是千篇一律,或者多數都是沒有一個完整的流程。甚至,還發現,對於Elasticsearch資料庫實際應用方面,大多數海賊只侷限於ELK等這樣的情況,就像是大多數海賊提到Redis,就只覺得這個玩意兒,只能用於快取層面,但是實際上人家的功能強大到超出了海賊的想象空間。甚至於,薩博在一個阿里巴巴的地方,找到了關於Elasticsearch資料庫免費試用的資源,但是對於免費的午餐,也許那一天收費之後,那就只能用一首“浪浪”來祭奠這萬惡的黃金之惡。於是在薩博精心研究和分析發現,除了傳統部署Elasticsearch資料庫之外,有個叫Docker 的東西,除了能夠穩定執行Elasticsearch資料庫之外,還能節省不少的伺服器記憶體。於是呀,薩博就開始了探索在Docker上搭建Elasticsearch之之路......

Docker部署Elasticsearch的“那些事兒"

1.基礎先決條件

阿里雲或者虛擬機器VM主機一個,並且自行裝有Docker容器:
BWUuHU.png

ps[⚠️注意事項]:
對於Docker安裝請參考:Linux環境基於CentOS7 搭建部署Docker容器

2.最大檔案開啟數

[2.1]引數優化:ulimit -a |grep open

[root@centos-meteor ~]# ulimit -a |grep open
open files   (-n) 65535
[root@centos-meteor ~]# 

[2.2] 修改配置檔案: /etc/security/limits.conf

#在最下邊新增如下兩行
* soft nofile 65536
* hard nofile 65536
* soft nproc 2048
* hard nproc 4096

[2.3] 確認是否修改: ulimit -a |grep open

[root@centos-meteor ~]# ulimit -a |grep open
open files   (-n) 65535
[root@centos-meteor ~]# 

3.調整系統程式數

修改方法:vim /etc/security/limits.d/20-nproc.conf

#調整成以下配置
*          soft    nproc     4096
root       soft    nproc     unlimited

4.調整虛擬記憶體和最大併發連線數

虛擬記憶體(Virtual memory)->vm.max_map_count
Elasticsearch預設使用 hybrid mmapfs / niofs 目錄來儲存索引。預設作業系統對mmap計數的限制太低,可能引發記憶體不足的異常

[1]臨時解決方法:

sysctl –w vm.max_map_count=262144
sysctl –w vm.max_map_count=262144

如果不優化會出現啟動Elasticsearch的時候會出現如下提示:

max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144]

[2]永久生效方案:更新系統檔案/etc/sysctl.conf的vm.max_map_count欄位
執行命令:vim /etc/sysctl.conf

[root@turbo-master ~]# vim /etc/sysctl.conf
[root@turbo-master ~]#
#Settings Elasticsearch Virtual Memory—>655360
vm.max_map_count=655360
fs.file-max=655360
vm.overcommit_memory=1

載入到系統:sysctl -p --load /etc/sysctl.conf

[root@centos-meteor elasticsearch]# sysctl -p --load /etc/sysctl.conf
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.sysrq = 1
vm.max_map_count = 655360
fs.file-max = 655360
vm.overcommit_memory = 1
[root@centos-meteor elasticsearch]#

5.關閉elasticsearch自檢測

在elasticsearch.yml中新增配置項:

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

ps[⚠️注意事項]:
主要是避免以下問題:
[1]最大檔案開啟數: max file descriptors [1024] for elasticsearch process is too low, increase to at least [65536]
[2]調整虛擬記憶體和最大併發連線數: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[3]關閉elasticsearch自檢測: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[4]調整系統程式數: max number of threads [1024] for user [lish] likely too low, increase to at least [2048]

實戰部署ElasticSearch服務

Docker 部署應用服務的基本步驟:Search[查詢映象]->Pull[拉取映象]->Run[部署映象]

1.查詢Elasticsearch 映象:

docker search elasticsearch 

elasticsearch-search

ps[注意事項]:

  1. 一般拉取映象資源都是從Docker官方倉庫[docker-hub]拉取,或者自己構建的Docker雲倉庫aliyun-docker
  2. 本教程選取的ELK映象均是基於ELK官方Docker倉庫elastic-io

2.拉取Elasticsearch 映象:

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.3.1

elasticsearch-pull

ps[注意事項]:
1.本教程採用7.3.x版本,目前最新版本7.4.x[主要用7.3.x版本在阿里雲搭建過,避免入坑問題]
2.拉取的過程中可能會出現[net/http: TLS handshake timeout]問題,多嘗試幾次,主要是網路頻寬限制問題

3.修改映象名稱:

docker tag docker.elastic.co/elasticsearch/elasticsearch:7.3.1  elasticsearch:latest

elasticsearch-tag

ps[注意事項]:
1.名稱過長導致檢視些許不便,通過docker tag source-image[來源映象] target-image[目標映象],推薦統一採用[target-image:target-version]格式定義,且不佔用空間,相當於重新命名映象
2.對於拉取kibana[docker.elastic.co/kibana/kibana:7.3.1]和logstash[docker.elastic.co/logstash/logstash:7.3.1] 都建議修改。

4.部署映象服務:
部署命令:

docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server -e "discovery.type=single-node" -e ES_JAVA_OPTS="-Xms=512m -Xms=512m" elasticsearch:latest

預設配置目錄:

/usr/share/elasticsearch/config
/usr/share/elasticsearch/logs

檢視容器列表:

docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"
![docker ps]

(https://ftp.bmp.ovh/imgs/2019/11/d7e09058af3fadbd.png)
ps[注意事項]:

1.需要開放埠[9200和9300]->9200作為Http協議,主要用於外部通訊,9300作為Tcp協議,jar之間就是通過tcp協議通訊,通常部署叢集就是通過9300通訊。推薦[宿主機自定義埠:9200]
2.--restart=always :配置容器重啟策略,當宿主機重啟由於配置了開機自啟動,不用手動啟動
3.--privileged:配置容器操作許可權[true-root操作許可權,false-當前容器使用者操作許可權]
4.對於部署網路模式推薦預設橋接模式,也自定義可以host模式等

5.修改配置:
進入容器:docker exec -it container-id[容器id] or container-name[容器名稱] /bin/bash
例如:docker exec -it f2d2e97da375 /bin/bash #f2d2e97da375-> container-id
docker-exec

修改配置檔案:

[root@f2d2e97da375 elasticsearch]# ls 
LICENSE.txt  NOTICE.txt  README.textile  bin  config  data  jdk  lib  logs  modules  plugins
[root@f2d2e97da375 elasticsearch]# 
[root@f2d2e97da375 elasticsearch]# cd config  
[root@f2d2e97da375 config]# ls
elasticsearch.keystore  elasticsearch.yml  jvm.options  log4j2.properties  role_mapping.yml  roles.yml  users  users_roles
[root@f2d2e97da375 config]# vi elasticsearch.yml 

新增跨域配置:http.cors.enabled: true && http.cors.allow-origin: "*"

cluster.name: "docker-cluster"
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"

然後退出exit容器,在宿主機重啟容器:docker restart container-id[容器id] or container-name[容器名稱]
docker restart f2d2e97da375

[root@f2d2e97da375 config]# exit
exit
[root@centos-meteor ~]# docker restart f2d2e97da375
f2d2e97da375
[root@centos-meteor ~]# 

ps[注意事項]:
1.進入容器方式:包括使用 docker attach 命令或 docker exec 命令,
推薦使用 docker exec 命令。原因:

  • docker attach: 使用exit退出容器,會導致容器的停止
  • docker exec:使用exit退出容器,不會導致容器的停止
  • 參考docker進入容器的幾種方法部落格-docker進入容器的幾種方法
    2.如果Docker安裝了視覺化介面 Portainer,推薦採用這種方式進入容器:
    docker-portainer

搭建部署ElasticSearch-Head服務

ElasticSearch-Head:彈性搜尋叢集的Web前端介面,是使用Nodjs構建的,主要用於檢視ElasticSearch相關資訊

1.拉取Elasticsearch-Head 映象:docker pull mobz/elasticsearch-head:5

[root@centos-amber ~]# docker pull mobz/elasticsearch-head:5
5: Pulling from mobz/elasticsearch-head
75a822cd7888: Pull complete 
57de64c72267: Pull complete 
4306be1e8943: Pull complete 
871436ab7225: Pull complete 
0110c26a367a: Pull complete 
1f04fe713f1b: Pull complete 
723bac39028e: Pull complete 
7d8cb47f1c60: Pull complete 
7328dcf65c42: Pull complete 
b451f2ccfb9a: Pull complete 
304d5c28a4cf: Pull complete 
4cf804850db1: Pull complete 
Digest: sha256:55a3c82dd4ba776e304b09308411edd85de0dc9719f9d97a2f33baa320223f34
Status: Downloaded newer image for mobz/elasticsearch-head:5
docker.io/mobz/elasticsearch-head:5
[root@centos-amber ~]# 

2.修改Elasticsearch-Head 映象名稱:docker tag  mobz/elasticsearch-head:5   elasticsearch-head:latest

[root@centos-amber ~]# docker tag  mobz/elasticsearch-head:5        elasticsearch-head:latest
[root@centos-amber ~]# docker images
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
grafana/grafana                                 latest              05d1bcf30d16        7 days ago          207MB
nginx                                           latest              540a289bab6c        3 weeks ago         126MB
prom/prometheus                                 latest              2c8e464e47f4        3 weeks ago         129MB
moxm/sentinel-dashboard                         latest              0ccaac81584e        4 weeks ago         167MB
portainer                                       latest              4cda95efb0e4        4 weeks ago         80.6MB
portainer/portainer                             latest              4cda95efb0e4        4 weeks ago         80.6MB
apache/skywalking-ui                            latest              fa66ca9c9862        2 months ago        123MB
apache/skywalking-oap-server                    latest              376a37cdf65c        2 months ago        190MB
docker.elastic.co/kibana/kibana                 7.3.1               b54865ba6b0b        2 months ago        1.01GB
docker.elastic.co/elasticsearch/elasticsearch   7.3.1               3d3aa92f641f        2 months ago        807MB
elasticsearch                                   latest              3d3aa92f641f        2 months ago        807MB
prom/node-exporter                              latest              e5a616e4b9cf        5 months ago        22.9MB
google/cadvisor                                 latest              eb1210707573        12 months ago       69.6MB
elasticsearch-head                              latest              b19a5c98e43b        2 years ago         824MB
mobz/elasticsearch-head                         5                   b19a5c98e43b        2 years ago         824MB
tutum/influxdb                                  latest              c061e5808198        3 years ago         290MB
[root@centos-amber ~]# 

3.部署Elasticsearch-Head 容器:

docker run -itd --restart=always --privileged=true -p 9100:9100 --name elasticsearch-head-server elasticsearch-head:latest

檢視容器服務:

docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"

docker-elasticsearch-head

4.瀏覽器訪問:http://remote-ip:9100/
elasticsearch-head

實戰測試ElasticSearch-Head服務

1.準備資料:
BW0ZiF.png

{
    "companyId": "ebcb4f99e0cb4ad781278ae636c1031f",
    "classifyName": "激素檢測",
    "deviceType": "激素檢測",
    "data": {
    	"deviceId": "20000060100000002",
        "deviceNo": "QC00020000060100000002",
        "appid": "00000000",
        "sim": "89860404191792655118",
        "csq": "30",
        "electric": "98",
        "voltage": "13279",
        "softVer": "5143000500010601-01020006|5143000500010601-01010003",
        "hardVer": "5143000500010601-00000002|5143000500010601-00000002",
        "status": "0",
        "date": "2020-09-13 11:23:52",
        "elements": [
        	{
                "key": "20",
                "value": "10",
                "time": "2020-09-13 11:23:52"
            },
            {
                "key": "21",
                "value": "11",
                "time": "2020-09-13 11:23:52"
            },
            {
                "key": "22",
                "value": "12",
                "time": "2020-09-13 11:23:52"
            },
            {
                "key": "23",
                "value": "13",
                "time": "2020-09-13 11:23:52"
            }
        ]
    }
}

2.設定請求頭:application/json;charset=UTF-8
BW0KMR.png

3.傳送PUT請求:http://remote-ip/device/deviceData/QC00020000060100000001
BW0tRH.png

{
    "_index": "device",
    "_type": "deviceData",
    "_id": "QC00020000060100000002",
    "_version": 1,
    "result": "created",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    },
    "created": true
}

4.開啟瀏覽器檢視結果:
BW0gzj.png

實戰搭建ElasticSearch總結

1.部署基本命令:

docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server --network-alias elasticsearch-server --hostname elasticsearch-server -v /docker/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /docker/elasticsearch/data:/usr/share/elasticsearch/data -v /docker/elasticsearch/logs:/usr/share/elasticsearch/logs  -e "discovery.type=single-node" -e ES_JAVA_OPTS="-server -Xms512m -Xmx512m -Xmn256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:+AlwaysPreTouch -XX:-UseBiasedLocking " -e TZ="Asia/Shanghai"  elasticsearch:latest

2.配置檔案[elasticsearch.yml ]:

cluster.name: elasticsearch-cluster
node.name: elasticsearch-server
network.host: 0.0.0.0
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
transport.tcp.port: 9300
transport.tcp.compress: true
http.max_content_length: 128mb

3.分詞外掛安裝:

./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.5.0/elasticsearch-analysis-ik-7.5.0.zip

4.Docker 部署 ElasticSearch-Head:

docker run -itd --restart=always --privileged=true -p 9100:9100 --network-alias elasticsearch-head-server --name elasticsearch-head-server --hostname elasticsearch-head-server -v /docker/elasticsearch-head/app:/usr/src/app elasticsearch-head:latest

5.ElasticSearch-Head 使用406錯誤:
BWBnk8.png
ajaxSettings部分json配置:

    ajaxSettings: {
      url: ajaxLocation,
      isLocal: rlocalProtocol.test(ajaxLocParts[1]),
      global: true,
      type: "GET",
      //contentType: "application/x-www-form-urlencoded",
      contentType: "application/json;charset=UTF-8",
      processData: true,
      async: true,
      /*
		timeout: 0,
		data: null,
		dataType: null,
		username: null,
		password: null,
		cache: null,
		traditional: false,
		headers: {},
		*/

      accepts: {
        xml: "application/xml, text/xml",
        html: "text/html",
        text: "text/plain",
        json: "application/json, text/javascript",
        "*": "*/*",
      },

inspectData 部分json配置:

  // Detect, normalize options and install callbacks for jsonp requests
  jQuery.ajaxPrefilter("json jsonp", function (s, originalSettings, jqXHR) {
    //var inspectData = s.contentType === "application/x-www-form-urlencoded" &&
    var inspectData =
      (s.contentType === "application/x-www-form-urlencoded" &&
        typeof s.data === "string") ||
      (s.contentType === "application/json;charset=UTF-8" &&
        typeof s.data === "string");

版權宣告:本文為博主原創文章,遵循相關版權協議,如若轉載或者分享請附上原文出處連結和連結來源。

相關文章