運維除錯記錄:Ubuntu14.04下部署Opendaylight Nitrogen叢集

KiteRunner24發表於2018-03-21

建議參考官網教程:Setting Up Clustering


一、實驗環境

  • 主機節點系統版本: Ubuntu 14.04 (64bit)
odl@mpodl:~$ uname -a
Linux mpodl 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  • 主機節點硬體配置:單核CPU+4GB記憶體+50GB硬碟
odl@mpodl:~$ sudo lshw        
    description: Computer
    product: Standard PC (i440FX + PIIX, 1996) ()
    vendor: QEMU
    version: pc-i440fx-trusty
    width: 64 bits
    capabilities: smbios-2.4 dmi-2.4 vsyscall32
    configuration: boot=normal uuid=053D43B6-2E3C-CEA4-4C52-833DDD1749BE
  *-core
       description: Motherboard
       physical id: 0
     # CPU資訊 
     *-cpu
          description: CPU
          product: QEMU Virtual CPU version 2.0.0
          vendor: Intel Corp.
          physical id: 401
          bus info: cpu@0
          slot: CPU 1
          size: 2GHz
          capacity: 2GHz
          width: 64 bits
     # 記憶體資訊
     *-memory
          description: System Memory
          physical id: 1000
          size: 4GiB
     # 硬碟資訊
     *-disk
           description: ATA Disk
           product: QEMU HARDDISK
           physical id: 0.0.0
           bus info: scsi@0:0.0.0
           logical name: /dev/sda
           version: 0
           serial: QM00001
           size: 50GiB (53GB)
           *-volume:0
                description: EXT4 volume
                vendor: Linux
                physical id: 1
                bus info: scsi@0:0.0.0,1
                logical name: /dev/sda1
                logical name: /
                version: 1.0
                serial: 65d80188-ddfb-4f08-8018-a0c2e1da8af3
                size: 46GiB
                capacity: 46GiB
           *-volume:1
                description: Extended partition
                physical id: 2
                bus info: scsi@0:0.0.0,2
                logical name: /dev/sda2
                size: 4093MiB
                capacity: 4093MiB
  • 叢集環境:3臺主機節點
Cluster_Node1: Ubuntu 14.04 -- [IP_Addr]=192.168.1.124
Cluster_Node2: Ubuntu 14.04 -- [IP_Addr]=192.168.1.125
Cluster_Node3: Ubuntu 14.04 -- [IP_Addr]=192.168.1.104

二、部署方法

1. 下載Opendaylight Nitrogen

在每臺主機節點上執行如下命令,下載tar.gz格式的壓縮包到使用者目錄下的ODL_N子目錄:

odl@mpodl:~/ODL_N$ wget -P . https://nexus.opendaylight.org/content/repositories/public/org/opendaylight/integration/karaf/0.7.2/karaf-0.7.2.tar.gz

或在官網下載,然後通過xftp軟體從本地上傳到伺服器的Opendaylight執行環境中。

官網地址:https://www.opendaylight.org/technical-community/getting-started-for-developers/downloads-and-documentation

2. 安裝Opendaylight Nitrogen

在每臺主機節點上執行如下命令,解壓壓縮包到ODL_N子目錄:

odl@mpodl:~/ODL_N$ tar -zxvf karaf-0.7.2.tar.gz

解壓完成,即認為Opendaylight Nitrogen安裝成功。

3. 配置Opendaylight Nitrogen叢集

在每臺主機節點下執行如下命令,完成叢集指令碼配置工作:

命令格式:sudo bash ./karaf-0.7.2/bin/configure_cluster.sh [index] [seed_node_list]

其中,[index]為正整數,表示在[seed_node_list]中對應哪個主機節點上配置叢集指令碼。另外,[seed_node_list]是組成叢集的各個主機節點對應的IP地址列表,以空格或逗號隔開。

因此,具體執行命令如下:

# Cluster_Node1: IP=192.168.1.124
odl@mpodl:~/ODL_N$ sudo bash ./karaf-0.7.2/bin/configure_cluster.sh 1 192.168.1.124 192.168.1.125 192.168.1.104

# Cluster_Node2: IP=192.168.1.125
odl@mpodl:~/ODL_N$ sudo bash ./karaf-0.7.2/bin/configure_cluster.sh 2 192.168.1.124 192.168.1.125 192.168.1.104

# Cluster_Node3: IP=192.168.1.104
odl@mpodl:~/ODL_N$ sudo bash ./karaf-0.7.2/bin/configure_cluster.sh 3 192.168.1.124 192.168.1.125 192.168.1.104

當執行完之後,結果如下所示:

# Cluster_Node1: IP=192.168.1.124
odl@mpodl:~/ODL_N$ sudo bash ./karaf-0.7.2/bin/configure_cluster.sh 1 192.168.1.124 192.168.1.125 192.168.1.104
################################################
##             Configure Cluster              ##
################################################
Configuring unique name in akka.conf
Configuring hostname in akka.conf
Configuring data and rpc seed nodes in akka.conf
modules = [

    {
        name = "inventory"
        namespace = "urn:opendaylight:inventory"
        shard-strategy = "module"
    },
    {
        name = "topology"
        namespace = "urn:TBD:params:xml:ns:yang:network-topology"
        shard-strategy = "module"
    },
    {
        name = "toaster"
        namespace = "http://netconfcentral.org/ns/toaster"
        shard-strategy = "module"
    }
]
Configuring replication type in module-shards.conf
################################################
##   NOTE: Manually restart controller to     ##
##         apply configuration.               ##
################################################

備註:

(1)執行如上命令後,會在karaf-0.7.2/configuration目錄下生成initial子目錄,結果如下所示:

odl@mpodl:~/ODL_N$ ls ./karaf-0.7.2/configuration/
context.xml  factory  initial  logback.xml  tomcat-logging.properties  tomcat-server.xml

odl@mpodl:~/ODL_N$ ls ./karaf-0.7.2/configuration/initial/
akka.conf  modules.conf  module-shards.conf

可見,叢集配置指令碼在initial子目錄下生成了akka.confmodules.confmodule-shards.conf三個配置檔案。

(2)檢視第一臺Ubuntu主機節點的akka.conf檔案內容,具體如下所示:

odl@mpodl:~/ODL_N$ cat ./karaf-0.7.2/configuration/initial/akka.conf 

odl-cluster-data {
  akka {
    remote {
      artery {
        enabled = off
        canonical.hostname = "192.168.1.124"   # 本機IP地址
        canonical.port = 2550
      }
      netty.tcp {
        hostname = "192.168.1.124"             # 本機IP地址
        port = 2550
      }
      # when under load we might trip a false positive on the failure detector
      # transport-failure-detector {
        # heartbeat-interval = 4 s
        # acceptable-heartbeat-pause = 16s
      # }
    }

    cluster {
      # Remove ".tcp" when using artery.
      # 叢集節點列表
      seed-nodes = ["akka.tcp://opendaylight-cluster-data@192.168.1.124:2550",
                "akka.tcp://opendaylight-cluster-data@192.168.1.125:2550",
                "akka.tcp://opendaylight-cluster-data@192.168.1.104:2550"]

      roles = ["member-1"]

    }

    persistence {
      # By default the snapshots/journal directories live in KARAF_HOME. You can choose to put it somewhere else by
      # modifying the following two properties. The directory location specified may be a relative or absolute path. 
      # The relative path is always relative to KARAF_HOME.

      # snapshot-store.local.dir = "target/snapshots"
      # journal.leveldb.dir = "target/journal"

      journal {
        leveldb {
          # Set native = off to use a Java-only implementation of leveldb.
          # Note that the Java-only version is not currently considered by Akka to be production quality.

          # native = off
        }
      }
    }
  }
}

同樣地,第二臺和第三臺Ubuntu主機節點的akka.conf檔案內容相似,只是對應的IP地址不同。

(3)檢視第一臺Ubuntu主機節點的module-shards.conf檔案內容,具體如下所示:

odl@mpodl:~/ODL_N$ cat ./karaf-0.7.2/configuration/initial/module-shards.conf 
module-shards = [
    {
        name = "default"
        shards = [
            {
                name = "default"
                replicas = ["member-1",
                "member-2",
                "member-3"]
            }
        ]
    },
    {
        name = "inventory"
        shards = [
            {
                name="inventory"
                replicas = ["member-1",
                "member-2",
                "member-3"]
            }
        ]
    },
    {
        name = "topology"
        shards = [
            {
                name="topology"
                replicas = ["member-1",
                "member-2",
                "member-3"]
            }
        ]
    },
    {
        name = "toaster"
        shards = [
            {
                name="toaster"
                replicas = ["member-1",
                "member-2",
                "member-3"]
            }
        ]
    }
]

同樣地,第二臺和第三臺Ubuntu主機節點的module-shards.conf檔案內容完全相同。

4. 啟動Opendaylight Nitrogen叢集

在每臺主機節點下執行如下命令,完成Opendaylight節點啟動工作:

odl@mpodl:~/ODL_N$ ./karaf-0.7.2/bin/karaf 
karaf: JAVA_HOME not set; results may vary
Apache Karaf starting up. Press Enter to open the shell now...
100% [========================================================================]
Karaf started in 8s. Bundle stats: 208 active, 209 total

    ________                       ________                .__  .__       .__     __       
    \_____  \ ______   ____   ____ \______ \ _____  ___.__.|  | |__| ____ |  |___/  |_     
     /   |   \\____ \_/ __ \ /    \ |    |  \\__  \<   |  ||  | |  |/ ___\|  |  \   __\    
    /    |    \  |_> >  ___/|   |  \|    `   \/ __ \\___  ||  |_|  / /_/  >   Y  \  |      
    \_______  /   __/ \___  >___|  /_______  (____  / ____||____/__\___  /|___|  /__|      
            \/|__|        \/     \/        \/     \/\/            /_____/      \/          


Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown OpenDaylight.

opendaylight-user@root>

然後,執行命令feature:list -i檢查odl-mdsal-clustering是否處於已安裝狀態。如果沒有安裝,則執行命令feature:install odl-mdsal-clustering完成對應Feature的安裝。

opendaylight-user@root>feature:list -i
Name                            | Version | Required | State   | Repository                      | Description
----------------------------------------------------------------------------------------------------------------------------------------------------
odl-mdsal-broker                | 1.6.2   |          | Started | odl-mdsal-1.6.2                 | odl-mdsal-broker
odl-mdsal-clustering            | 1.6.2   | x        | Started | odl-mdsal-clustering            | odl-mdsal-clustering

5. 檢查Opendaylight Nitrogen叢集是否建立

在每臺主機節點下執行如下命令,獲取主機節點的角色資訊(Leader/Follower):

opendaylight-user@root> ld | grep clustering

於是,在主機節點一,輸出如下:

2018-03-21 13:22:45,688 | INFO  | d-dispatcher-125 | Shard                            | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-prefix-configuration-shard-config (Candidate): Starting new election term 21
2018-03-21 13:22:45,741 | INFO  | d-dispatcher-125 | Shard                            | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-prefix-configuration-shard-config (Candidate) :- Switching from behavior Candidate to Leader, election term: 21
2018-03-21 13:22:45,742 | INFO  | ult-dispatcher-5 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-1-shard-prefix-configuration-shard-config , received role change from Candidate to Leader
2018-03-21 13:22:45,821 | INFO  | d-dispatcher-121 | Shard                            | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-prefix-configuration-shard-operational (Candidate): Starting new election term 21
2018-03-21 13:22:45,858 | INFO  | d-dispatcher-125 | Shard                            | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-prefix-configuration-shard-operational (Candidate) :- Switching from behavior Candidate to Leader, election term: 21
2018-03-21 13:22:45,858 | INFO  | ult-dispatcher-5 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-1-shard-prefix-configuration-shard-operational , received role change from Candidate to Leader
2018-03-21 13:22:45,872 | INFO  | d-dispatcher-125 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-entity-ownership-operational (Candidate): Starting new election term 21
2018-03-21 13:22:45,912 | INFO  | d-dispatcher-145 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-entity-ownership-operational (Candidate) :- Switching from behavior Candidate to Leader, election term: 21
2018-03-21 13:22:45,921 | INFO  | lt-dispatcher-21 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-1-shard-entity-ownership-operational , received role change from Candidate to Leader

在主機節點二,輸出如下:

2018-03-21 13:22:36,286 | INFO  | ult-dispatcher-4 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-prefix-configuration-shard-config , received role change from null to Follower
2018-03-21 13:22:36,287 | INFO  | ult-dispatcher-4 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-prefix-configuration-shard-operational , received role change from null to Follower
2018-03-21 13:22:36,287 | INFO  | ult-dispatcher-4 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-prefix-configuration-shard-config , registered listener akka://opendaylight-cluster-data/user/shardmanager-config
2018-03-21 13:22:36,287 | INFO  | ult-dispatcher-4 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-prefix-configuration-shard-operational , registered listener akka://opendaylight-cluster-data/user/shardmanager-operational
2018-03-21 13:22:36,305 | INFO  | rd-dispatcher-32 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | Starting recovery for member-2-shard-entity-ownership-operational with journal batch size 1
2018-03-21 13:22:36,313 | INFO  | rd-dispatcher-38 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | Recovery completed - Switching actor to Follower - Persistence Id =  member-2-shard-entity-ownership-operational Last index in log = -1, snapshotIndex = -1, snapshotTerm = -1, journal-size = 0
2018-03-21 13:22:36,317 | INFO  | ult-dispatcher-2 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-entity-ownership-operational , received role change from null to Follower

在主機節點三,輸出如下:

2018-03-21 13:22:39,418 | INFO  | ult-dispatcher-6 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-prefix-configuration-shard-operational , received role change from null to Follower
2018-03-21 13:22:39,418 | INFO  | ult-dispatcher-6 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-prefix-configuration-shard-config , received role change from null to Follower
2018-03-21 13:22:39,418 | INFO  | ult-dispatcher-6 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-prefix-configuration-shard-operational , registered listener akka://opendaylight-cluster-data/user/shardmanager-operational
2018-03-21 13:22:39,418 | INFO  | ult-dispatcher-6 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-prefix-configuration-shard-config , registered listener akka://opendaylight-cluster-data/user/shardmanager-config
2018-03-21 13:22:39,466 | INFO  | rd-dispatcher-23 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | Starting recovery for member-3-shard-entity-ownership-operational with journal batch size 1
2018-03-21 13:22:39,470 | INFO  | rd-dispatcher-23 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | Recovery completed - Switching actor to Follower - Persistence Id =  member-3-shard-entity-ownership-operational Last index in log = -1, snapshotIndex = -1, snapshotTerm = -1, journal-size = 0
2018-03-21 13:22:39,473 | INFO  | lt-dispatcher-31 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-entity-ownership-operational , received role change from null to Follower

可以看出,第一臺Ubuntu主機節點成為Leader,其它兩臺Ubuntu主機節點成為 Follower ,叢集配置成功。但是,從日誌也可以看出,對於不同的Shard,存在不同的叢集關係。

三、小結

本文詳細介紹了Opendaylight Nitrogen叢集的搭建指南,後續碰到問題將繼續補充。

相關文章