In this Document

Applies to:

Purpose

The purpose of this note is to supplement the HAIP information available in Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip Note 1210883.1

Questions and Answers

1. What does HAIP stand for?

Highly Available IP (address)

2. What clusterware version was HAIP introduced in?

Oracle Grid Infrastructure 11.2.0.2

3. What is the purpose of HAIP?  

Haip has two primary functions:

HAIP allows for redundant cluster interconnect NICs on cluster nodes without requiring any OS level bonding, teaming or aggregating of the NICs.
HAIP provides a layer of transparency between multiple Network Interface Cards (NICs) on a given node that are used for the cluster interconnect and Oracle components such as the RDBMS and ASM instances.

4. How does HAIP create a layer of transparency between the cluster interconnect NICs on a given node and the RDBMS/ASM instances?

When clusterware on a given node is started, a virtual ip address (HAIP)  is dynamically allocated and linked (link-local) to each NIC configured for use as a cluster interconnect.  Each NIC defined as an interconnect (using oifcfg setif) will be assigned a unique HAIP during clusterware startup.  The HAIP from each interconnect NIC is then used by the the ASM and RDBMS instances and ACFS.  The static ip address associated with the physical NIC is not used by the ASM and RDBMS instances. The NIC name is not used by the ASM or RDBMS instance but is used by other clusterware components.

Each NIC defined as a cluster interconnect (using oifcfg setif) on a given node will have a static ip address assigned to it and each cluster interconnect NIC on a given node must be on a unique subnet. Clusterware requires that each subnet used by the cluster interconnect have a functioning NIC on each active cluster node. If any one of the cluster interconnect NICs is down on a node, then the subnet associated with the down NIC is considered not usable by any node of the cluster.

5. How does HAIP obtain an IP address for use in the virtual link?

HAIP takes advantage of linked-local ip addresses as defined in the Internet Engineering Task Force RFC-3927 .  This RFC reserves the IPv4 prefix 169.254/16 specifically for linked local use.

6. How can I check if the HAIP is in use?

Use the OS command "ifconfig -a". Look in the ASM and RDBMS instance alert logs. During instance startup the interfaces and associated HAIP will be shown.

7. Which Oracle clusterware log shows the failover or failback of a HAIP?

The ohasd orarootagent_root logs under /log//agent/ohasd/orarootagent_root/orarootagent_root.log.

8. What happens to the HAIP if a physical NIC fails or is disabled?

The HAIP associated with the failed NIC will failover and be linked-local to one of the remaining working physical NICs designated for the cluster interconnect. So the HAIP will now be on a different subnet, one that is working on all nodes.
An ARP broadcast will be made using the new MAC address for the NIC that the HAIP has attached to.  Since the ASM and RDBMS instances are using the HAIP, they transparently continue to work using the NIC that HAIP failed over to. An important point to note is that that HAIP will perform the same failover on EVERY active cluster node. The reason this is done is to ensure that the HAIP is attached to an NIC on the same subnet across all nodes. 

9. What happens to the HAIP if the failed or disabled physical NIC is brought back up?

The HAIP will failback to the physical NIC it was linked to before the failover. The HAIP will be unlinked from the redundant NIC it was temporarily on, and locally-linked to the interconnect NIC that was brought back online. As with a failover this also will happen on all nodes of the cluster to ensure the HAIP is on the same subnet on all nodes. An ARP broadcast is made to publish the changed MAC address of the NIC that each HAIP that failed back to.

10. Does HAIP require NIC bonding or teaming?

No. HAIP allows you to provide redundant NICS for the cluster interconnect without using OS level NIC bonding or teaming or aggregation. Each physical interface is presented to the clusterware and registered with the clusterware via the Oracle clusterware command "oifcfg setif".

11. Can OS bonding or teaming still be used with HAIP?

Yes, however Oracle's best practice is to use unbonded, unteamed NICs, without any additional layers of virtualization.

12 . Is the HAIP active even if there is only one NIC per node for the cluster interconnect?

Yes.  This cannot be disabled manually (with the exception on Solaris where, when IPMP is in use, HAIP will not be enabled during installation).

13. Is there any limit to how many redundant interconnect NICs that can be used by clusterware?

Yes. Clusterware will use up to four NICs, even if more are available.

14. With HAIP, are all the interconnect NICs active concurrently?

The default is for all interconnect NICs to be active.  The Oracle best practice to have all NICs active.

15. Can I manually failover or failback HAIP?

No. There is no command that allows you to directly control failover and failback of HAIP.  Clusterware does this automatically.

16. Is HAIP supported with pre-11.2.0.2 databases?

The default behaviour for pre-11.2.0.2 databases is to use the fixed IP address associated with NICs and not use the HAIP.

The recommendation is to use OS level NIC bonding/teaming for pre 11.2.0.2 databases.

17. What is the recommendation regarding using OS level NIC virtualization with HAIP?

Oracle recommends using just HAIP, with no additional OS level NIC virtualization.

18. Can HAIP feature be disabled manually?

No, It's NOT supported to disable HAIP while the cluster is up and running. There are certain situations where HAIP feature is disabled during installation. Please refer to Note 1210883.1 Section Miscellaneous. For IPMP implementation on Solaris case, cluster_interconnects is required to set to IPMP IP. For other platforms, OS bonding technology should be used if more than 1 private network is used.

References