用 Vagrant 一鍵部署開發環境 kafka 叢集

xueyubingsen發表於2020-02-24
開發環境Kafka

vagrant介紹

vagrant是一個很適合開發者的虛擬環境部署工具，本身整合了主流的虛擬器管理工具，支援vmvare和virtualbox。vagrant的精髓在一個Vagrantfile裡面，和docker的Dockerfile功能上一樣。我們只需要把需要安裝部署的步驟寫在Vagrantfile裡面，便可以實現輕鬆部署。vagrant還支援把當前系統做成一個.box字尾命名的映象，類似docker的image，可輕鬆實現環境的移植。

叢集環境預覽

VM Name Host Name IP Address
zookeeper1 vkc-zk1 10.30.3.2
zookeeper2 vkc-zk2 10.30.3.3
zookeeper3 vkc-zk3 10.30.3.4
broker1 vkc-br1 10.30.3.30
broker2 vkc-br2 10.30.3.20
broker3 vkc-br3 10.30.3.10

Hosts file entries:

10.30.3.2    vkc-zk1
10.30.3.3     vkc-zk2
10.30.3.4     vkc-zk3
10.30.3.30     vkc-br1
10.30.3.20     vkc-br2
10.30.3.10     vkc-br3

準備

軟體

下載並安裝如下的軟體：

  1. virtualbox 官網地址
  2. vagrant2.2.7 官網地址
  3. kafka_2.11-2.4.0.tgz下載地址
  4. 映象：vagrant-centos-7.2.box
    下載地址：
    連結：https://pan.baidu.com/s/15S2OZq37FcL9RWWST...
    提取碼：3xb0

空間：

15G的硬碟空間

vagrant新增映象

vagrant box add centos72 D:\\zjdata\\virtualbox\\centos72\\vagrant-centos-7.2.box

vagrant box add命令會在c盤使用者下面的.vagrant.d/boxes/目錄下面建立一個和映象大小差不多的檔案（700M左右）。

注意：以下的命令列都是在git-bash下面執行的。

cd /c/zjdata/vagrant-vm/
git clone https://github.com/xueyubingsen/vagrant-kafka.git
cd vagrant-kafka
mv Vagrantfile Vagrantfile.bak
vagrant init vagrant-kafka
  1. vagrant init 會生成一個Vagrantfile檔案和一些隱藏目錄，這個vagrant-kafka是隨便起的。
  2. 先備份一下從github上下載的Vagrantfile檔案，等init完之後，再替換回來。
mv Vagrantfile.bak Vagrantfile

修改Vagrantfile

將config.vm.box的值改為我們剛剛新增的box的名字
將KAFKA_VERSION的值改為2.4.0

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  config.vm.box = "centos72"
  config.ssh.forward_agent = true # So that boxes don't have to setup key-less ssh
  config.ssh.insert_key = false # To generate a new ssh key and don't use the default Vagrant one

  vars = {
     "KAFKA_VERSION" => "2.4.0",
     "KAFKA_NAME" => "kafka_2.11-$KAFKA_VERSION",
     "KAFKA_TARGET" => "/vagrant/tars/",
     "KAFKA_HOME" => "$HOME/$KAFKA_NAME"
  }

  # escape environment variables to be loaded to /etc/profile.d/
  as_str = vars.map{|k,str| ["export #{k}=#{str.gsub '$', '\$'}"] }.join("\n")

  # common provisioning for all
  config.vm.provision "shell", path: "scripts/hosts-file-setup.sh", env: vars
  config.vm.provision "shell", inline: "echo \"#{as_str}\" > /etc/profile.d/kafka_vagrant_env.sh", run: "always"
  config.vm.provision "shell", path: "scripts/init.sh", env: vars

  # configure zookeeper cluster
  (1..3).each do |i|
    config.vm.define "zookeeper#{i}" do |s|
      s.vm.hostname = "zookeeper#{i}"
      s.vm.network "private_network", ip: "10.30.3.#{i+1}"
      #s.vm.network "private_network", ip: "10.30.3.#{i+1}", netmask: "255.255.255.0", virtualbox__intnet: "my-network", drop_nat_interface_default_route: true
      s.vm.provision "shell", run: "always", path: "scripts/zookeeper.sh", args:"#{i}", privileged: false, env: vars
    end
  end

  # configure brokers
  (1..3).each do |i|
    config.vm.define "broker#{i}" do |s|
      s.vm.hostname = "broker#{i}"
      s.vm.network "private_network", ip: "10.30.3.#{4-i}0"
      #s.vm.network "private_network", ip: "10.30.3.#{4-i}0", netmask: "255.255.255.0", virtualbox__intnet: "my-network", drop_nat_interface_default_route: true
      s.vm.provision "shell", run: "always", path: "scripts/broker.sh", args:"#{i}", privileged: false, env: vars
    end
  end

  config.vm.provider "virtualbox" do |v|
    #  This setting controls how much cpu time a virtual CPU can use. A value of 50 implies a single virtual CPU can use up to 50% of a single host CPU.
    v.customize ["modifyvm", :id, "--cpuexecutioncap", "50"]
  end
end

拷貝kafka安裝包

把開始的時候下載的kafka_2.11-2.4.0.tgz安裝包拷貝到專案目錄的tars目錄下，tars目錄需要手工建立。

啟動叢集

只需要啟動叢集，vagrant就會按照Vagrantfile裡面配置的那樣，開始逐個安裝叢集。

vagrant up

安裝之後的樣子

$ vagrant global-status
id       name       provider   state   directory
---------------------------------------------------------------------------
436fb44  zookeeper1 virtualbox running C:/zjdata/vagrant-vm/vagrant-kafka
6292373  zookeeper2 virtualbox running C:/zjdata/vagrant-vm/vagrant-kafka
5325c42  zookeeper3 virtualbox running C:/zjdata/vagrant-vm/vagrant-kafka
6b594e2  broker1    virtualbox running C:/zjdata/vagrant-vm/vagrant-kafka
690398d  broker2    virtualbox running C:/zjdata/vagrant-vm/vagrant-kafka
a26ed4b  broker3    virtualbox running C:/zjdata/vagrant-vm/vagrant-kafka

The above shows information about all known Vagrant environments
on this machine. This data is cached and may not be completely
up-to-date (use "vagrant global-status --prune" to prune invalid
entries). To interact with any of the machines, you can go to that
directory and run Vagrant, or you can use the ID directly with
Vagrant commands from any directory. For example:
"vagrant destroy 1a2b3c4d"

注意：可以單獨針對某一個節點執行provision

vagrant provision zookeeper1

連線到zookeeper1節點

vagrant ssh zookeeper1
本作品採用《CC 協議》，轉載必須註明作者和本文連結

相關文章