Recovering from a disaster

What to do

After determining that the failure to the MetroCluster configuration is a disaster, you should do the following:

Ensure that the surviving node is isolated from its partner. See Restricting access to the disaster site cluster node.
Force the surviving node to take over the functions of it partner. See Forcing a node into takeover mode.
Recover access to the failed partner's data by completing one of the following tasks:
- If you are using file-access protocols, remount the failed partner's volumes. See Remounting volumes of the failed node.
- If you are using iSCSI, bring the failed partner's LUNs online.
Fix the cluster at the disaster site. See Fixing failures caused by the disaster.
Re-create the MetroCluster configuration. See Reestablishing the MetroCluster configuration.

Restricting access to the disaster site cluster node

You must restrict access to the disaster site cluster node to prevent the cluster node from resuming service. If you do not restrict access, you risk the possibility of data corruption.

You can restrict access to the disaster site cluster node in the following ways:

Turn off power to the disaster site cluster node.
Use manual fencing.

Turning off power to the disaster site cluster node

To turn off power to the disaster site cluster node, complete the following step.

Step	Action
1	Switch off the power at the back of the storage system.

This is the preferred method for restricting access to the disaster site cluster node. You can perform this task at the disaster site or remotely, if you have that capability.

Using manual fencing

You can use manual fencing as an alternative to turning off power to the disaster site cluster node. The manual fencing method restricts access using software and physical means.

To manually fence off the disaster site cluster node, complete the following steps.

Step	Action
1	Disconnect the cluster interconnect and Fibre Channel adapter cables of the cluster node at the surviving site.
2	If you are using...	Then fencing is achieved by...
	Application failover	Using any application-specified method that either prevents the application from restarting at the disaster site or prevents the application clients from accessing the application servers at the disaster site. Methods can include turning off the application server, removing an application server from the network, or any other method that prevents the application server from running applications.
	IP failover	Using network management procedures to ensure that the storage systems at the disaster site are isolated from the external public network.

Forcing a node into takeover mode

To force the surviving node into takeover mode, complete the following step.

Step	Action
1	Enter the following command on the surviving cluster node: cf forcetakeover -d Result Data ONTAP causes the following to occur: The surviving cluster node takes over the functions of the failed partner. The mirrored relationships between the two plexes of mirrored aggregates are broken, thereby creating two unmirrored aggregates. This is called splitting the mirrored aggregates. The overall result of using the `cf forcetakeover -d` command is that a cluster node at the surviving site is running in takeover mode with all the data in unmirrored aggregates.

Step

Action

Enter the following command on the surviving cluster node:

cf forcetakeover -d

Result

Data ONTAP causes the following to occur:

The surviving cluster node takes over the functions of the failed partner.
The mirrored relationships between the two plexes of mirrored aggregates are broken, thereby creating two unmirrored aggregates. This is called splitting the mirrored aggregates.

The overall result of using the cf forcetakeover -d command is that a cluster node at the surviving site is running in takeover mode with all the data in unmirrored aggregates.

Remounting volumes of the failed node

You must remount the volumes of the failed node because the volumes are accessed through the surviving node. For more information about mounting volumes, see the File Access and Protocols Management Guide.

To remount the volumes, complete the following steps.

Step	Action
1	On an NFS client at the surviving site, create a directory to act as a mount point. Example mkdir /n/toaster/home
2	Mount the volume. Example mount toaster:/vol/vol0/home /n/toaster/home

Step

Action

On an NFS client at the surviving site, create a directory to act as a mount point.

Example

mkdir /n/toaster/home

Mount the volume.

Example

mount toaster:/vol/vol0/home /n/toaster/home

Recovering LUNs of the failed node

If you have a MetroCluster configuration with iSCSI-attached hosts, you must actively track the state of LUNs (track whether they are online or offline) on the node at each site. If there is a failure to a MetroCluster configuration that qualifies as a disaster and the node at one site is inaccessible, all LUNs in the aggregates that were mirrored at the surviving site are offline. There is no way to distinguish the LUNs that were offline before the disaster from the LUNs that were online before the disaster unless you have been tracking their status.

When you recover access to the failed node's LUNs, it is important to bring back online only the LUNs that were online before the disaster. To avoid igroup mapping conflicts, do not bring a LUN online if it was offline before the disaster. For example, suppose you have two LUNs with IDs of 0 mapped to the same igroup, but one of these LUNs was offline before the disaster. If you bring the previously offline LUN online first, you cannot bring the second LUN online because you cannot have two LUNs with the same ID mapped to the same host.

To recover access to the failed node's LUNs, complete the following steps.

Step	Action
1	Identify the LUNs that were online before the disaster occurred.
2	Make sure that the LUNs are mapped to an igroup that contains the hosts attached to the surviving cluster node. For more information about mapping LUNs to igroups, see your Block Access Management Guide.
3	On the surviving cluster node, enter the following command: *lun online lun-path* ...** lun-path is the path to the LUN you want to bring online. You can specify more than one path to bring multiple LUNs online. Example 1 `lun online /vol/vol1/lun0` Example 2 lun online /vol/vol1/lun0 /vol/vol1/lun1 Note After you bring LUNs back online, you might have to perform some application or host-side recovery procedures. For more information, see the documentation for your application and for your host operating system.

Step

Action

Identify the LUNs that were online before the disaster occurred.

Make sure that the LUNs are mapped to an igroup that contains the hosts attached to the surviving cluster node.

For more information about mapping LUNs to igroups, see your Block Access Management Guide.

On the surviving cluster node, enter the following command:

lun online lun-path ...

lun-path is the path to the LUN you want to bring online. You can specify more than one path to bring multiple LUNs online.

Example 1

lun online /vol/vol1/lun0

Example 2

lun online /vol/vol1/lun0 /vol/vol1/lun1

Note
After you bring LUNs back online, you might have to perform some application or host-side recovery procedures. For more information, see the documentation for your application and for your host operating system.

Fixing failures caused by the disaster

You need to fix the failures caused by the disaster, if possible. For example, if a prolonged power outage to one of the MetroCluster sites caused the failure, restoring the power fixes the failure.

To fix failures, complete the following steps.

Step	Action
1	Ensure that the storage system and disk shelves have power.
2	Ensure that all Fibre Channel adapters, VI cluster adapters, cables, and switch ports are connected and working.

You cannot fix failures if the disaster causes a site to be destroyed. For example, a fire or an earthquake could destroy one of the MetroCluster sites. In this case, you fix the failure by creating a new MetroCluster configured partner at a different site.

After the node at the surviving site can see the disk shelves at the disaster site, Data ONTAP renames the volumes in the mirrored aggregates that were split by adding a number in parenthesis to the volume name. For example, if the volume name was vol1 before the disaster and the split, the renamed volume name could be vol1(1).

Reestablishing the MetroCluster configuration

Depending on the state of a mirrored aggregate before you forced the surviving cluster to take over its partner, you use one of two procedures to reestablish the MetroCluster configuration:

If the mirrored aggregate was in a normal state before the forced takeover, you can rejoin the two aggregates to reestablish the MetroCluster configuration. This is the most typical case.
If the mirrored aggregate was in an initial resynchronization state (level-0) before the forced takeover, you cannot rejoin the two aggregates. You must re-create the synchronous mirror to reestablish the MetroCluster configuration.

Rejoining the two aggregates

To return the cluster to normal operation by rejoining the two aggregates, complete the following steps.

Step	Action
1	Validate that you can access the remote storage by entering the following command: `aggr status -r`
2	Turn on power to the cluster node at the disaster site. Result After the cluster node at the disaster site boots, it displays the following message: Waiting for Giveback...
3	Determine which aggregates are at the surviving site and which aggregates are at the disaster site by entering the following command: aggr status Aggregates at the disaster site show plexes that are in a failed state with an out-of-date status. Aggregates at the surviving site show plexes as online.
4	If aggregates at the disaster site are online, take them offline by entering the following command for each online aggregate: aggr offline disaster_aggr disaster_aggr is the name of the aggregate at the disaster site. Note An error message appears if the aggregate is already offline.
5	Re-create the mirrored aggregates by entering the following command for each aggregate that was split: aggr mirror aggr_name -v disaster_aggr aggr_name is the aggregate on the surviving site's cluster node. disaster_aggr is the aggregate on the disaster site's cluster node. Result The aggr_name aggregate rejoins the disaster_aggr aggregate to reestablish the MetroCluster configuration.
6	Enter the following command at the partner cluster node: cf giveback Result The cluster node at the disaster site reboots.

Example of rejoining aggregates

The following example shows the commands and status output when you rejoin aggregates to reestablish the MetroCluster configuration.

First, the aggregate status of the disaster site's storage after reestablishing access to the cluster partner at the surviving site is shown.

filer1> aggr status -r

Aggregate mir (online, normal) (zoned checksums)

  Plex /mir/plex5 (online, normal, active)

    RAID group /filer1/plex5/rg0 (normal)

RAID Disk Device HA  SHELF BAY CHAN  Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ----- -------------- -------------

parity   8a.2   8a   0     2   FC:B  34500/70656000 35003/71687368

data     8a.8   8a   1     0   FC:B  34500/70656000 35003/71687368

Aggregate mir(1) (failed, out-of-date) (zoned checksums)

  Plex /mir(1)/plex1 (offline, normal, out-of-date)

    RAID group /mir(1)/plex1/rg0 (normal)

RAID Disk Device HA  SHELF BAY CHAN  Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ----- -------------- -------------

parity   6a.0   6a   0     0   FC:B  34500/70656000 35003/71687368

data     6a.1   6a   0     1   FC:B  34500/70656000 35003/71687368

  Plex /mir(1)/plex5 (offline, failed, out-of-date)

Next, the mirror is reestablished using the aggr mirror -v command.

Note
The node at the surviving site is called filer1; the node at the disaster site is called filer2.

filer1> aggr mirror mir -v mir(1)

This will destroy the contents of mir(1).  Are you sure? y

Mon Nov 18 15:36:59 GMT [filer1: 
raid.mirror.resync.snapcrtok:info]: mir: created mirror 
resynchronization snapshot mirror_resync.1118153658(filer2)

Mon Nov 18 15:36:59 GMT [filer1: raid.rg.resync.start:notice]: 
/mir/plex6/rg0: start resynchronization (level 1)

Mon Nov 18 15:36:59 GMT [filer1: raid.mirror.resync.start:notice]: 
/mir: start resynchronize to target /mir/plex6

After the aggregates rejoin, the synchronous mirrors of the MetroCluster configuration are reestablished.

filer1> aggr status -r mir

Aggregate mir (online, mirrored) (zoned checksums)

  Plex /mir/plex5 (online, normal, active)

    RAID group /mir/plex5/rg0 (normal)

RAID Disk Device HA  SHELF BAY CHAN  Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ----- -------------- -------------

parity   8a.2   8a   0     2   FC:B  34500/70656000 35003/71687368

data     8a.8   8a   1     0   FC:B  34500/70656000 35003/71687368

  Plex /mir/plex6 (online, normal, active)

    RAID group /mir/plex6/rg0 (normal)

RAID Disk Device HA  SHELF BAY CHAN  Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ----- -------------- -------------

parity   6a.0   6a   0     0   FC:B  34500/70656000 35003/71687368

data 6a.1 6a 0 1 FC:B 34500/70656000 35003/71687368

Re-creating mirrored aggregates

To return the cluster to normal operation by re-creating the MetroCluster mirror, complete the following steps.


aggr status -r
Waiting for Giveback...
aggr offline disaster_aggr
aggr destroy plex_name
aggr mirror plex_name
cf giveback

Step	Action
1	Validate that you can access the remote storage by entering the following command: aggr status -r Note A `(level-0 resync in progress)` message indicates that a plex cannot be rejoined.
2	Turn on the power to the cluster node at the disaster site. Result After the cluster node at the disaster site boots up, it displays the following: Waiting for Giveback...
3	If the aggregates at the disaster site are online, take them offline by entering the following command for each aggregate that was split: aggr offline disaster_aggr disaster_aggr is the name of the aggregate at the disaster site. Note An error message appears if the aggregate is already offline.
4	Destroy every target plex that is in a level-0 resync state by entering the following command: aggr destroy plex_name For more information about the SyncMirror feature, see the Data ONTAP Data Protection Online Backup and Recovery Guide.
5	Re-create the mirrored aggregates by entering the following command for each aggregate that was split: aggr mirror plex_name
6	Enter the following command at the partner cluster node: cf giveback Result The cluster node at the disaster site reboots.

Example of re-creating a mirrored aggregate

The following example shows the commands and status output when re-creating aggregates to reestablish the MetroCluster configuration.

First, the aggregate status of the disaster site's storage after reestablishing access to the cluster partner at the surviving site is shown.

filer1>aggr status -r

Aggregate mir1 (online, normal) (zoned checksums)

  Plex /mir1/plex0 (online, normal, active)

  RAID group /mir1/plex0/rg0 (normal)

RAID Disk Device HA  SHELF BAY CHAN  Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ----- -------------- -------------

parity   8a.3   8a   0     3   FC:B  34500/70656000 35003/71687368

data     8a.4   8a   0     4   FC:B  34500/70656000 35003/71687368

data     8a.6   8a   0     6   FC:B  34500/70656000 35003/71687368

data     8a.5   8a   0     5   FC:B  34500/70656000 35003/71687368

Aggregate mir1(1) (failed, partial) (zoned checksums)

  Plex /mir1(1)/plex0 (offline, failed, inactive)

  Plex /mir1(1)/plex6 (online, normal, resyncing)

    RAID group /mir1(1)/plex6/rg0 (level-0 resync in progress)

RAID Disk Device HA  SHELF BAY CHAN  Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ----- -------------- -------------

parity    6a.6   6a  0     6   FC:B  34500/70656000 35003/71687368

data      6a.2   6a  0     2   FC:B  34500/70656000 35003/71687368

data      6a.3   6a  0     3   FC:B  34500/70656000 35003/71687368

data      6a.5   6a  0     5   FC:B  34500/70656000 35003/71687368

The mir1(1)/plex6 plex shows that a level-0 resynchronization was in progress; therefore, an attempt to rejoin the plexes fails, as shown in the following output:

filer1> aggr mirror mir1 -v mir1(1)

aggr mirror: Illegal mirror state for aggregate 'mir1(1)'

Because the mir1(1)/plex6 plex had a level-0 resynchronization in progress, the mir1(1) aggregate must be destroyed and the mir aggregate remirrored to reestablish a synchronous mirror, as shown in the following output:

filer1> aggr mirror mir1 -v mir1(1)

aggr mirror: Illegal mirror state for aggregate 'mir1(1)'

filer1> aggr destroy mir1(1)

Are you sure you want to destroy this aggregate? y

Aggregate 'mir1(1)' destroyed.

filer1> aggr mirror mir1

Creation of a mirror plex with 4 disks has been initiated. The 
disks need to be zeroed before addition to the aggregate. The 
process has been initiated and you will be notified via the system 
log as disks are added.