Using iSCSI on clustered storage systems

About clustered storage systems

A clustered storage system consists of two storage systems that are both connected to the same set of disks. If one storage system fails, its partner storage system can take over for the failed system and continue to make its data available.

About CFO

The takeover process is called cluster failover (CFO). During CFO, the surviving storage system responds to iSCSI hosts as if it were the original storage system. Specifically, the partner assumes the IP addresses, iSCSI target identities, LUNs, igroups, CHAP settings, and other settings of the failed system.

From the host's perspective, during CFO the target stops responding to the initiator and the iSCSI session is lost. Then the target (now running on the partner storage system) resumes responding to the initiator, and a new iSCSI session is established. If the initiator had outstanding SCSI commands, the initiator resends those commands and the new target processes them. The host is not aware that the CFO took place, only that the target did not respond for a short period of time and that the iSCSI session had to be reestablished.

Requirements for clustered iSCSI systems

For CFO to work correctly, the two storage systems must be configured correctly, and the TCP/IP network must be configured correctly. Of special concern are the target portal group tags configured on the two storage systems.

The best practice is to configure the two partners of the cluster identically:

Use the same network cards in the same slots.
Create the same networking configuration with the matching pairs of ports connected to the same subnets.
Put the matching pairs of interfaces into the matching target portal groups and assign the same tag values to both groups.

Simple configuration

Consider the following simplified example. Storage system A has a two-port Ethernet card in slot 9. Interface e9a has the IP address 10.1.2.5, and interface e9b has the IP address 10.1.3.5. The two interfaces belong to a user-defined target portal group with tag value 2.

Storage system B has the same Ethernet card in slot 9. Interface e9a is assigned 10.1.2.6, and e9b is assigned 10.1.3.6. Again, the two interfaces are in a user-defined target portal group with tag value 2.

In the cluster configuration, interface e9a on storage system A is the partner of e9a on storage system B. Likewise, e9b on system A is the partner of e9b on system B. For more information on configuring interfaces for a cluster, see the Data ONTAP Cluster Installation and Administration Guide.

Now assume that storage system B fails and its iSCSI sessions are dropped. Storage system A assumes the identity of storage system B. Interface e9a now has two IP addresses: its original address of 10.1.2.5, and the 10.1.2.6 address from storage system B. The iSCSI host that was using storage system B reestablishes its iSCSI session with the target on storage system A.

If the e9a interface on storage system A was in a target portal group with a different tag value than the interface on storage system B, the host might not be able to continue its iSCSI session from storage system B. This behavior varies depending on the specific host and initiator.

To ensure correct CFO behavior, both the IP address and the tag value must be the same as on the failed system. And because the target portal group tag is a property of the interface and not the IP address, the surviving interface cannot change the tag value during a CFO.

Complex configurations

If your cluster has a more complex networking configuration, including virtual interfaces (vifs) and VLANs, follow the same best practice of making the configurations the same. For example, if you have a vif on storage system A, create the same vif on storage system B. Make sure the target portal group tag assigned to each vif is the same. The name of the target portal group does not have to be the same, only the tag value matters.