APPLIES TO:

Oracle Database - Enterprise Edition - Version 10.2.0.1 to 11.2.0.3 [Release 10.2 to 11.2]
Information in this document applies to any platform.

PURPOSE

Network information in Oracle Clusterware MUST be consistent with the setting on OS level. This note explains the basics of IPv4 subnet and Oracle Clusterware (CRS or Grid Infrastructure)

DETAILS

A. What's IPv4 Subnet

The IP subnet is a logical separation of IP addresses bounded by the subnet mask. The first address is used as the Subnet Identification (subnet ID or subnet number or route definition) while the last is the broadcast address.

For example, subnet ID 10.1.0.0 and netmask 255.255.255.128 produces an IP range of 10.1.0.0 - 10.1.0.127. In this range, 10.1.0.0 is the subnet ID and 10.1.0.127 is the broadcast address. Changing the netmask to 255.255.254.0, the IP range will be 10.1.0.0 - 10.1.1.255 where the broadcast address is now 10.1.1.255.

B. Calculate Subnet ID online with ifconfig output

Some websites can be used to calculate subnet ID if IP address and netmask is known, search for "subnet calculator" to find them.

Assuming the following as ifconfig output of a network adapter:

eth7 Link encap:Ethernet HWaddr 00:16:3E:11:22:88
..
inet 10.220.13.108 netmask 0xfffff800 broadcast 10.220.15.255

For this subnet, netmask is 255.255.248.0 (converted from hex 0xfffff800), broadcast is 10.220.15.255 and subnet ID is 10.220.8.0

C. Find out Subnet ID from oifcfg

oifcfg from Oracle Clusterware (CRS or Grid Infrastructure) can also be used to find out subnet ID and netmask:

$CLUSTERWARE_HOME/bin/oifcfg iflist -p -n

eth0  10.208.4.0  PRIVATE  255.255.255.0

eth1  10.1.0.128  PRIVATE  255.255.255.128

eth1  169.254.0.0  UNKNOWN  255.255.0.0

Note: 

The first column is the network adapter name

The second column is the subnet ID

The third column indicates whether it's private, public or unknown according to RFC standard, it has NOTHING to do whether it's used as private or public network in Oracle Clusterware

The last column is the netmask

D. Subnet Info in Oracle Clusterware - OCR

Both public and private network information are stored in OCR.

For pre-11.2, the private network information in OCR is used by ASM and database instances; for 11.2+, the information is used by Oracle Clusterware(GI), ASM and databases instances.

To find out what's in OCR:

$CLUSTERWARE_HOME/bin/oifcfg getif

eth1  10.1.0.0  global  cluster_interconnect

eth3  120.0.0.0  global  public

Note: 

The first column is the network adapter name

The second column is the subnet ID

The third column is always "global" and should not be changed

The last column indicates whether it's public or cluster_interconnect(private) in Oracle Clusterware

To modify:

Network info in OCR can not be modified. To change, as Oracle Clusterware user, remove the incorrect network info and add it back.

For example, if "eth3" has wrong subnet number in OCR, to fix it, as Oracle Clusterware user:

$CLUSTERWARE_HOME/bin/oifcfg delif -global eth3

$CLUSTERWARE_HOME/bin/oifcfg setif -global eth3/120.0.0.0:public

Note: the same network adapter is not allowed to be in OCR for more than once, otherwise the command will fails with error: PRIF-50: duplicate interface is given in the input

Refer to Document 283684.1 for details to change both public and private network in OCR.

E. Subnet Info in Oracle Clusterware - Nodeapps(network resource)

To find out nodeapps information

For pre-11.2 CRS:

$CLUSTERWARE_HOME/bin/srvctl config nodeapps -n <nodename> -a

VIP exists.: /racnode1v/10.1.0.44/255.255.255.128/eth3

Note: 

The first column is the node VIP name - in this example "racnode1v"

The second column is the node VIP IP address

The third column is the public network netmask for this VIP

The forth column is the network adapter name

For 11.2 GI

$CLUSTERWARE_HOME/bin/srvctl config network

Network exists: 1/10.10.0.0/255.255.255.128/eth3, type static

Network exists: 2/10.20.1.0/255.255.255.128/eth4, type static

$CLUSTERWARE_HOME/bin/srvctl config nodeapps

Network exists: 1/10.10.0.0/255.255.255.128/eth3, type static

VIP exists: /wirac1fv/10.10.0.52/10.10.0.0/255.255.255.128/eth3, hosting node wirac1f

GSD exists

ONS exists: Local port 6100, remote port 6200, EM port 2016

Note:

The VIP part is the same as in 10.2

For Network part:

The first column is the network number, default network will have number 1

The second column is the subnet ID, in this example 10.10.10.0 for the first network

The third column is the netmask, in this example both network have 255.255.255.128

The forth column is the network adapter name

"type static" means this is static network instead of a dynamic network (DHCP etc)

Error number/messages associated with mismatch of public network info

PRCR-1013 : Failed to start resource ora.net1.network
PRCR-1064 : Failed to start resource ora.net1.network on node racnode1
CRS-2674: Start of 'ora.net1.network' on 'racnode1' failed
PRCR-1079 : Failed to start resource ora.racnode1.vip
CRS-2632: There are no more servers to try to place resource 'ora.racnode1.vip' on that would satisfy its placement policy

To modify nodeapps (network resource)

As root execute "srvctl modify network -h" for 11.2.0.2 and above or "srvctl modify nodeapps -h" for pre-11.2.0.2. Refer to note 276434.1 for details.

The Basics of IPv4 Subnet and Oracle Clusterware（Doc ID 1386709.1）