IB_Switch交換機SB7890配置及Band網路卡

IB_Switch交換機SB7890配置及Band網路卡

1.建立軟體存放目錄

[root@localhost ~]#  mkdir -p /soft

2.上傳OS映象，本案例以RHEL7.7為例

mount -o loop /soft/rhel-server-7.7-x86_64-dvd.iso /mnt/

3.配置yum源

[root@localhost ~]# vi /etc/yum.conf 

[rheldvd]

name=rhel7.7

baseurl=file:///mnt

enabled=1

gpgcheck=0

[main]

………………………………

4.IB交接機驅動下載或上傳已有驅動包至soft目錄

驅動包：

MLNX_OFED_LINUX-5.1-2.3.7.1-rhel7.7-x86_64.tgz

5.解壓驅動並安裝

cd /soft

tar -xvzf MLNX_OFED_LINUX-5.1-2.3.7.1-rhel7.7-x86_64.tgz

6.安裝驅動

yum install pciutils perl gtk2 atk cairo gcc-gfortran tcsh openssl-libs libusbx lsof tcl fuse-libs tk -y

cd /soft/MLNX_OFED_LINUX-5.1-2.3.7.1-rhel7.7-x86_64

./mlnxofedinstall 

7.Load HCA驅動

/etc/init.d/openibd restart

8.啟動子網管理並設定開機啟動

/etc/init.d/opensmd start   ——對於當前SB7890交換不自子網管理，因此只能在OS端開啟子網管理服務，若多個節點建議開1——2個節點即可，不建議全部開啟；

/sbin/chkconfig opensmd on

/sbin/chkconfig opensmd --list

9.重啟OS

reboot

10.檢視IB是否正常識別

ifconfig -a

正常識別如下：

ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2044

Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).

        infiniband 00:00:10:87:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  txqueuelen 256  (InfiniBand)

        RX packets 2  bytes 232 (232.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 0  bytes 0 (0.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ib1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 4092

Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).

        infiniband 00:00:18:87:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  txqueuelen 256  (InfiniBand)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 0  bytes 0 (0.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

        

11.Band IB配置網路

01. Create bond0 Interface on the server. Update the following file:

touch /etc/sysconfig/network-scripts/ifcfg-bond0

Note: the only meaningful bonding policy in IPoIB is High-Availability (bonding mode number 1,

or "active-backup").

To learn more about the Linux bonding options click here .

Here is an example:

DEVICE=bond0

IPADDR=11.11.11.1

NETWORK=11.11.11.0

NETMASK=255.255.255.0

BROADCAST=11.11.11.255

USERCTL=no

BOOTPROTO=none

NM_CONTROLLED=yes

BONDING_OPTS="mode=active-backup primary=ib0 miimon=100 updelay=100 downdelay=100"

MTU=2044

02.Update/Create ib0 Interface on the server. Update the following file:

touch /etc/sysconfig/network-scripts/ifcfg-ib0

Here is an example:

DEVICE=ib0

TYPE=InfiniBand

NM_CONTROLLED=yes

USERCTL=no

MASTER=bond0

SLAVE=yes

BOOTPROTO=none

PRIMARY=yes

03.Update/Create ib1 Interface on the server. Update the following file:

touch /etc/sysconfig/network-scripts/ifcfg-ib1

Here is an example:

DEVICE=ib1

TYPE=InfiniBand

NM_CONTROLLED=yes

USERCTL=no

MASTER=bond0

SLAVE=yes

BOOTPROTO=none

PRIMARY=no

04.Create/Update the bond.conf file.

touch /etc/modprobe.d/bond.conf

Configure this file as follows:

alias bond0 bonding # Adapter bonding driver

options bond0 max_bonds=2 miimon=100 mode=1 # 100ms fail-over timer. Mode 1 = Active/Backup

05. Restart the network services and the driver.

# /etc/init.d/network restart

# /etc/init.d/openibd restart

06.Verify the bond configuration:

# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: fault-tolerance (active-backup) (fail_over_mac active)

Primary Slave: ib0 (primary_reselect always)

Currently Active Slave: ib0

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 100

Down Delay (ms): 100

Slave Interface: ib1

MII Status: up

Speed: 40000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: a0:04:03:00:fe:80

Slave queue ID: 0

Slave Interface: ib0

MII Status: up

Speed: 40000 Mbps

Duplex: full

Link Failure Count: 254

Permanent HW addr: a0:04:02:20:fe:80

Slave queue ID: 0

07.Fail Over MAC

Since rdma_cm does not support MAC enslavement, fail_over_mac=1 (or active) should be added

to the bond interface configuration in case of Ethernet interface.

The value active(1) for fail_over_mac policy indicates that the MAC address of the bond should

always be the MAC address of the currently active slave. The MAC address of the slaves is not

changed; instead, the MAC address of the bond changes during a failover.

Changing to fail_over_mac=1 solves this problem as if this not active, the bond MAC jumps from

one interface to another according to the active interface.

Add the bonding options (BONDING_OPT)? "fail_over_mac=1" to /etc/sysconfig/network- scripts/ifcfg-bond0. Note: the default of this parameter is disabled (0).

Here is an example:

DEVICE=bond0

IPADDR=11.11.11.1

NETWORK=11.11.11.0

NETMASK=255.255.255.0

BROADCAST=11.11.11.255

USERCTL=no

BOOTPROTO=none

NM_CONTROLLED=yes

BONDING_OPTS="mode=active-backup primary=ib0 miimon=100 updelay=100 downdelay=100 fail_over_mac=1"

MTU=2044