運維除錯記錄：Ubuntu14.04下部署Opendaylight Nitrogen叢集

KiteRunner24發表於2018-03-21

一、實驗環境

主機節點系統版本： Ubuntu 14.04 (64bit)

odl@mpodl:~$ uname -a
Linux mpodl 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

主機節點硬體配置：單核CPU+4GB記憶體+50GB硬碟

odl@mpodl:~$ sudo lshw        
    description: Computer
    product: Standard PC (i440FX + PIIX, 1996) ()
    vendor: QEMU
    version: pc-i440fx-trusty
    width: 64 bits
    capabilities: smbios-2.4 dmi-2.4 vsyscall32
    configuration: boot=normal uuid=053D43B6-2E3C-CEA4-4C52-833DDD1749BE
  *-core
       description: Motherboard
       physical id: 0
     # CPU資訊 
     *-cpu
          description: CPU
          product: QEMU Virtual CPU version 2.0.0
          vendor: Intel Corp.
          physical id: 401
          bus info: cpu@0
          slot: CPU 1
          size: 2GHz
          capacity: 2GHz
          width: 64 bits
     # 記憶體資訊
     *-memory
          description: System Memory
          physical id: 1000
          size: 4GiB
     # 硬碟資訊
     *-disk
           description: ATA Disk
           product: QEMU HARDDISK
           physical id: 0.0.0
           bus info: scsi@0:0.0.0
           logical name: /dev/sda
           version: 0
           serial: QM00001
           size: 50GiB (53GB)
           *-volume:0
                description: EXT4 volume
                vendor: Linux
                physical id: 1
                bus info: scsi@0:0.0.0,1
                logical name: /dev/sda1
                logical name: /
                version: 1.0
                serial: 65d80188-ddfb-4f08-8018-a0c2e1da8af3
                size: 46GiB
                capacity: 46GiB
           *-volume:1
                description: Extended partition
                physical id: 2
                bus info: scsi@0:0.0.0,2
                logical name: /dev/sda2
                size: 4093MiB
                capacity: 4093MiB

叢集環境：3臺主機節點

Cluster_Node1: Ubuntu 14.04 -- [IP_Addr]=192.168.1.124
Cluster_Node2: Ubuntu 14.04 -- [IP_Addr]=192.168.1.125
Cluster_Node3: Ubuntu 14.04 -- [IP_Addr]=192.168.1.104

所有主機節點都連線到同一臺OpenvSwitch交換機上，且能互相PING通！！！
每臺主機節點都安裝配置有JDK 1.8，安裝方法見：Ubuntu下通過PPA方式安裝Java 8並自動配置環境變數

二、部署方法

1. 下載Opendaylight Nitrogen

在每臺主機節點上執行如下命令，下載tar.gz格式的壓縮包到使用者目錄下的ODL_N子目錄：

odl@mpodl:~/ODL_N$ wget -P . https://nexus.opendaylight.org/content/repositories/public/org/opendaylight/integration/karaf/0.7.2/karaf-0.7.2.tar.gz

或在官網下載，然後通過xftp軟體從本地上傳到伺服器的Opendaylight執行環境中。

官網地址：https://www.opendaylight.org/technical-community/getting-started-for-developers/downloads-and-documentation

2. 安裝Opendaylight Nitrogen

在每臺主機節點上執行如下命令，解壓壓縮包到ODL_N子目錄：

odl@mpodl:~/ODL_N$ tar -zxvf karaf-0.7.2.tar.gz

解壓完成，即認為Opendaylight Nitrogen安裝成功。

3. 配置Opendaylight Nitrogen叢集

在每臺主機節點下執行如下命令，完成叢集指令碼配置工作：

命令格式：sudo bash ./karaf-0.7.2/bin/configure_cluster.sh [index] [seed_node_list]

其中，[index]為正整數，表示在[seed_node_list]中對應哪個主機節點上配置叢集指令碼。另外，[seed_node_list]是組成叢集的各個主機節點對應的IP地址列表，以空格或逗號隔開。

因此，具體執行命令如下：

# Cluster_Node1: IP=192.168.1.124
odl@mpodl:~/ODL_N$ sudo bash ./karaf-0.7.2/bin/configure_cluster.sh 1 192.168.1.124 192.168.1.125 192.168.1.104

# Cluster_Node2: IP=192.168.1.125
odl@mpodl:~/ODL_N$ sudo bash ./karaf-0.7.2/bin/configure_cluster.sh 2 192.168.1.124 192.168.1.125 192.168.1.104

# Cluster_Node3: IP=192.168.1.104
odl@mpodl:~/ODL_N$ sudo bash ./karaf-0.7.2/bin/configure_cluster.sh 3 192.168.1.124 192.168.1.125 192.168.1.104

當執行完之後，結果如下所示：

# Cluster_Node1: IP=192.168.1.124
odl@mpodl:~/ODL_N$ sudo bash ./karaf-0.7.2/bin/configure_cluster.sh 1 192.168.1.124 192.168.1.125 192.168.1.104
################################################
##             Configure Cluster              ##
################################################
Configuring unique name in akka.conf
Configuring hostname in akka.conf
Configuring data and rpc seed nodes in akka.conf
modules = [

    {
        name = "inventory"
        namespace = "urn:opendaylight:inventory"
        shard-strategy = "module"
    },
    {
        name = "topology"
        namespace = "urn:TBD:params:xml:ns:yang:network-topology"
        shard-strategy = "module"
    },
    {
        name = "toaster"
        namespace = "http://netconfcentral.org/ns/toaster"
        shard-strategy = "module"
    }
]
Configuring replication type in module-shards.conf
################################################
##   NOTE: Manually restart controller to     ##
##         apply configuration.               ##
################################################

備註：

（1）執行如上命令後，會在karaf-0.7.2/configuration目錄下生成initial子目錄，結果如下所示：

odl@mpodl:~/ODL_N$ ls ./karaf-0.7.2/configuration/
context.xml  factory  initial  logback.xml  tomcat-logging.properties  tomcat-server.xml

odl@mpodl:~/ODL_N$ ls ./karaf-0.7.2/configuration/initial/
akka.conf  modules.conf  module-shards.conf

可見，叢集配置指令碼在initial子目錄下生成了akka.conf、modules.conf和module-shards.conf三個配置檔案。

（2）檢視第一臺Ubuntu主機節點的akka.conf檔案內容，具體如下所示：

odl@mpodl:~/ODL_N$ cat ./karaf-0.7.2/configuration/initial/akka.conf 

odl-cluster-data {
  akka {
    remote {
      artery {
        enabled = off
        canonical.hostname = "192.168.1.124"   # 本機IP地址
        canonical.port = 2550
      }
      netty.tcp {
        hostname = "192.168.1.124"             # 本機IP地址
        port = 2550
      }
      # when under load we might trip a false positive on the failure detector
      # transport-failure-detector {
        # heartbeat-interval = 4 s
        # acceptable-heartbeat-pause = 16s
      # }
    }

    cluster {
      # Remove ".tcp" when using artery.
      # 叢集節點列表
      seed-nodes = ["akka.tcp://opendaylight-cluster-data@192.168.1.124:2550",
                "akka.tcp://opendaylight-cluster-data@192.168.1.125:2550",
                "akka.tcp://opendaylight-cluster-data@192.168.1.104:2550"]

      roles = ["member-1"]

    }

    persistence {
      # By default the snapshots/journal directories live in KARAF_HOME. You can choose to put it somewhere else by
      # modifying the following two properties. The directory location specified may be a relative or absolute path. 
      # The relative path is always relative to KARAF_HOME.

      # snapshot-store.local.dir = "target/snapshots"
      # journal.leveldb.dir = "target/journal"

      journal {
        leveldb {
          # Set native = off to use a Java-only implementation of leveldb.
          # Note that the Java-only version is not currently considered by Akka to be production quality.

          # native = off
        }
      }
    }
  }
}

同樣地，第二臺和第三臺Ubuntu主機節點的akka.conf檔案內容相似，只是對應的IP地址不同。

（3）檢視第一臺Ubuntu主機節點的module-shards.conf檔案內容，具體如下所示：

odl@mpodl:~/ODL_N$ cat ./karaf-0.7.2/configuration/initial/module-shards.conf 
module-shards = [
    {
        name = "default"
        shards = [
            {
                name = "default"
                replicas = ["member-1",
                "member-2",
                "member-3"]
            }
        ]
    },
    {
        name = "inventory"
        shards = [
            {
                name="inventory"
                replicas = ["member-1",
                "member-2",
                "member-3"]
            }
        ]
    },
    {
        name = "topology"
        shards = [
            {
                name="topology"
                replicas = ["member-1",
                "member-2",
                "member-3"]
            }
        ]
    },
    {
        name = "toaster"
        shards = [
            {
                name="toaster"
                replicas = ["member-1",
                "member-2",
                "member-3"]
            }
        ]
    }
]

同樣地，第二臺和第三臺Ubuntu主機節點的module-shards.conf檔案內容完全相同。

4. 啟動Opendaylight Nitrogen叢集

在每臺主機節點下執行如下命令，完成Opendaylight節點啟動工作：

odl@mpodl:~/ODL_N$ ./karaf-0.7.2/bin/karaf 
karaf: JAVA_HOME not set; results may vary
Apache Karaf starting up. Press Enter to open the shell now...
100% [========================================================================]
Karaf started in 8s. Bundle stats: 208 active, 209 total

    ________                       ________                .__  .__       .__     __       
    \_____  \ ______   ____   ____ \______ \ _____  ___.__.|  | |__| ____ |  |___/  |_     
     /   |   \\____ \_/ __ \ /    \ |    |  \\__  \<   |  ||  | |  |/ ___\|  |  \   __\    
    /    |    \  |_> >  ___/|   |  \|    `   \/ __ \\___  ||  |_|  / /_/  >   Y  \  |      
    \_______  /   __/ \___  >___|  /_______  (____  / ____||____/__\___  /|___|  /__|      
            \/|__|        \/     \/        \/     \/\/            /_____/      \/          


Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown OpenDaylight.

opendaylight-user@root>

然後，執行命令feature:list -i檢查odl-mdsal-clustering是否處於已安裝狀態。如果沒有安裝，則執行命令feature:install odl-mdsal-clustering完成對應Feature的安裝。

opendaylight-user@root>feature:list -i
Name                            | Version | Required | State   | Repository                      | Description
----------------------------------------------------------------------------------------------------------------------------------------------------
odl-mdsal-broker                | 1.6.2   |          | Started | odl-mdsal-1.6.2                 | odl-mdsal-broker
odl-mdsal-clustering            | 1.6.2   | x        | Started | odl-mdsal-clustering            | odl-mdsal-clustering

5. 檢查Opendaylight Nitrogen叢集是否建立

在每臺主機節點下執行如下命令，獲取主機節點的角色資訊（Leader/Follower）：

opendaylight-user@root> ld | grep clustering

於是，在主機節點一，輸出如下：

2018-03-21 13:22:45,688 | INFO  | d-dispatcher-125 | Shard                            | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-prefix-configuration-shard-config (Candidate): Starting new election term 21
2018-03-21 13:22:45,741 | INFO  | d-dispatcher-125 | Shard                            | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-prefix-configuration-shard-config (Candidate) :- Switching from behavior Candidate to Leader, election term: 21
2018-03-21 13:22:45,742 | INFO  | ult-dispatcher-5 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-1-shard-prefix-configuration-shard-config , received role change from Candidate to Leader
2018-03-21 13:22:45,821 | INFO  | d-dispatcher-121 | Shard                            | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-prefix-configuration-shard-operational (Candidate): Starting new election term 21
2018-03-21 13:22:45,858 | INFO  | d-dispatcher-125 | Shard                            | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-prefix-configuration-shard-operational (Candidate) :- Switching from behavior Candidate to Leader, election term: 21
2018-03-21 13:22:45,858 | INFO  | ult-dispatcher-5 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-1-shard-prefix-configuration-shard-operational , received role change from Candidate to Leader
2018-03-21 13:22:45,872 | INFO  | d-dispatcher-125 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-entity-ownership-operational (Candidate): Starting new election term 21
2018-03-21 13:22:45,912 | INFO  | d-dispatcher-145 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-entity-ownership-operational (Candidate) :- Switching from behavior Candidate to Leader, election term: 21
2018-03-21 13:22:45,921 | INFO  | lt-dispatcher-21 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-1-shard-entity-ownership-operational , received role change from Candidate to Leader

在主機節點二，輸出如下：

2018-03-21 13:22:36,286 | INFO  | ult-dispatcher-4 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-prefix-configuration-shard-config , received role change from null to Follower
2018-03-21 13:22:36,287 | INFO  | ult-dispatcher-4 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-prefix-configuration-shard-operational , received role change from null to Follower
2018-03-21 13:22:36,287 | INFO  | ult-dispatcher-4 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-prefix-configuration-shard-config , registered listener akka://opendaylight-cluster-data/user/shardmanager-config
2018-03-21 13:22:36,287 | INFO  | ult-dispatcher-4 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-prefix-configuration-shard-operational , registered listener akka://opendaylight-cluster-data/user/shardmanager-operational
2018-03-21 13:22:36,305 | INFO  | rd-dispatcher-32 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | Starting recovery for member-2-shard-entity-ownership-operational with journal batch size 1
2018-03-21 13:22:36,313 | INFO  | rd-dispatcher-38 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | Recovery completed - Switching actor to Follower - Persistence Id =  member-2-shard-entity-ownership-operational Last index in log = -1, snapshotIndex = -1, snapshotTerm = -1, journal-size = 0
2018-03-21 13:22:36,317 | INFO  | ult-dispatcher-2 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-entity-ownership-operational , received role change from null to Follower

在主機節點三，輸出如下：

2018-03-21 13:22:39,418 | INFO  | ult-dispatcher-6 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-prefix-configuration-shard-operational , received role change from null to Follower
2018-03-21 13:22:39,418 | INFO  | ult-dispatcher-6 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-prefix-configuration-shard-config , received role change from null to Follower
2018-03-21 13:22:39,418 | INFO  | ult-dispatcher-6 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-prefix-configuration-shard-operational , registered listener akka://opendaylight-cluster-data/user/shardmanager-operational
2018-03-21 13:22:39,418 | INFO  | ult-dispatcher-6 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-prefix-configuration-shard-config , registered listener akka://opendaylight-cluster-data/user/shardmanager-config
2018-03-21 13:22:39,466 | INFO  | rd-dispatcher-23 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | Starting recovery for member-3-shard-entity-ownership-operational with journal batch size 1
2018-03-21 13:22:39,470 | INFO  | rd-dispatcher-23 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | Recovery completed - Switching actor to Follower - Persistence Id =  member-3-shard-entity-ownership-operational Last index in log = -1, snapshotIndex = -1, snapshotTerm = -1, journal-size = 0
2018-03-21 13:22:39,473 | INFO  | lt-dispatcher-31 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-entity-ownership-operational , received role change from null to Follower

可以看出，第一臺Ubuntu主機節點成為Leader，其它兩臺Ubuntu主機節點成為 Follower ，叢集配置成功。但是，從日誌也可以看出，對於不同的Shard，存在不同的叢集關係。

三、小結

本文詳細介紹了Opendaylight Nitrogen叢集的搭建指南，後續碰到問題將繼續補充。

運維除錯記錄：Opendaylight鈹版本開發環境搭建流程
2017-06-22
運維除錯開發環境
運維利器-ClusterShell叢集管理操作記錄
2016-11-24
運維
Docker叢集管理工具 - Kubernetes 部署記錄 (運維小結)
2016-06-11
Docker運維
MongoDB叢集運維筆記
2018-01-02
MongoDB運維筆記
ElasticSearch 叢集的規劃部署與運維
2021-03-03
Elasticsearch運維
Linux下效能除錯工具運維筆記
2018-01-04
Linux除錯運維筆記
ProxySQL Cluster 高可用叢集環境部署記錄
2019-02-21
SQL
Redis+TwemProxy(nutcracker)叢集方案部署記錄
2016-07-19
Redis
Mongodb副本集＋分片叢集環境部署記錄
2016-07-19
MongoDB
Centos7下GlusterFS分散式儲存叢集環境部署記錄
2018-04-08
CentOS分散式
RabbitMQ叢集運維實踐
2024-03-19
MQ運維
CentOS7下Elasticsearch叢集部署記錄
2017-10-19
CentOSElasticsearch
Centos6.9下RabbitMQ叢集部署記錄
2017-12-09
CentOSMQ
docker筆記41-ceph叢集的日常運維
2018-10-29
Docker筆記運維
mongos分片叢集管理和運維
2018-05-23
Go運維
kafka 基礎知識梳理及叢集環境部署記錄
2018-05-10
Kafka
ELK批量刪除索引及叢集相關操作記錄
2018-11-21
索引
徹底搞懂 etcd 系列文章（三）：etcd 叢集運維部署
2020-06-06
運維
Elasticsearch叢集運維相關知識
2018-11-21
Elasticsearch運維
Centos7下ELK+Redis日誌分析平臺的叢集環境部署記錄
2018-05-29
CentOSRedis
「實戰篇」開源專案docker化運維部署-搭建mysql叢集（四）
2019-03-28
Docker運維MySql
400+節點的 Elasticsearch 叢集運維
2019-03-26
Elasticsearch運維
400+ 節點的 Elasticsearch 叢集運維
2019-04-25
Elasticsearch運維
Tidb 運維--叢集檢視的使用
2022-05-01
TiDB運維
GBase XDM（單機/分片叢集）資料庫刪除記錄
2021-09-08
資料庫
GDB除錯使用記錄
2020-11-02
除錯
如何運維多叢集資料庫？58 同城 NebulaGraph Database 運維實踐
2023-02-16
運維資料庫Database
MongoDB副本集（一主一備+仲裁）環境部署-運維操作記錄
2017-11-22
MongoDB運維
京東雲開發者｜IoT運維 - 如何部署一套高可用K8S叢集
2022-11-03
運維K8S
阿里雲註冊叢集+Prometheus 解決多雲容器叢集運維痛點
2022-10-31
阿里Prometheus運維
Oracle RAC日常運維-NetworkManager導致叢集故障
2019-06-08
Oracle運維
阿里超大規模 Flink 叢集運維實踐
2022-06-07
阿里運維
docker部署mysql叢集
2021-07-30
DockerMySql
Docker部署ElasticSearch叢集
2021-01-18
DockerElasticsearch
GitHub學習除錯記錄
2020-11-09
Github除錯
Liunx常用運維命令整理記錄
2020-06-04
運維
插曲：Kafka的叢集部署實踐及運維相關
2019-11-21
Kafka運維
opendaylight筆記3.opendaylight
2013-09-22
筆記