Recovering from a disaster

What to do

After determining that the failure to the MetroCluster configuration is a disaster, you should do the following:

Ensure that the surviving filer is isolated from its partner. See Restricting access to the disaster site cluster node.
Force the surviving filer to take over the functions of it partner. See Forcing a node into takeover mode.
Remount the failed partner's volumes. See Remounting volumes of the failed filer.
Fix the cluster at the disaster site. See Fixing failures caused by the disaster.
Re-create the MetroCluster configuration. See Reestablishing the MetroCluster configuration.

Restricting access to the disaster site cluster node

You must restrict access to the disaster site cluster node to prevent the cluster node from resuming service. If you do not restrict access, you risk the possibility of data corruption.

You can restrict access to the disaster site cluster node in the following ways:

Turn off power to the disaster site cluster node.
Use manual fencing.

Turning off power to the disaster site cluster node

To turn off power to the disaster site cluster node, complete the following step.

Step	Action
1	Switch off the power at the back of the filer.

This is the best method for restricting access to the disaster site cluster node. You can perform this task at the disaster site or remotely, if you have that capability. Network Appliance recommends this method.

Using manual fencing

You can use manual fencing as an alternative to turning off power to the disaster site cluster node. The manual fencing method restricts access using software and physical means.

To manually fence off the disaster site cluster node, complete the following steps.

Step	Action
1	Disconnect the cluster interconnect and Fibre Channel adapter cables of the cluster node at the surviving site.
2	If you are using...	Then fencing is achieved by...
	Application failover	Using any application-specified method that either prevents the application from restarting at the disaster site or prevents the application clients from accessing the application servers at the disaster site. Methods can include turning off the application server, removing an application server from the network, or any other method that prevents the application server from running applications.
	IP failover	Using network management procedures to ensure that the filers at the disaster site are isolated from the external public network.

Forcing a node into takeover mode

To force the surviving cluster node into takeover mode, complete the following step.

Step	Action
1	Enter the following command on the surviving cluster node: cf forcetakeover -d Result Data ONTAP causes the following to occur: The surviving cluster node takes over the functions of the failed partner. The mirrored relationships between the two plexes of mirrored volumes are broken, thereby creating two unmirrored volumes. This is called splitting the mirrored volumes. The overall result of using the `cf forcetakeover -d` command is that a cluster node at the surviving site is running in takeover mode with all the data in unmirrored volumes.

Step

Action

Enter the following command on the surviving cluster node:

cf forcetakeover -d

Result

Data ONTAP causes the following to occur:

The surviving cluster node takes over the functions of the failed partner.
The mirrored relationships between the two plexes of mirrored volumes are broken, thereby creating two unmirrored volumes. This is called splitting the mirrored volumes.

The overall result of using the cf forcetakeover -d command is that a cluster node at the surviving site is running in takeover mode with all the data in unmirrored volumes.

Remounting volumes of the failed filer

You must remount the volumes of the failed filer because the volumes will be accessed through the surviving filer. To remount the volumes, complete the following steps.

Step	Action
1	On an NFS client at the surviving site, create a directory to act as a mount point. Example mkdir /n/toaster/home
2	Mount the volume. Example toaster:/vol/vol0/home /n/toaster/home

Step

Action

On an NFS client at the surviving site, create a directory to act as a mount point.

Example

mkdir /n/toaster/home

Mount the volume.

Example

toaster:/vol/vol0/home /n/toaster/home

See the Software Setup Guide for details about mounting volumes.

Fixing failures caused by the disaster

Fix the failures caused by the disaster, if possible. For example, if a prolonged power outage to one of the MetroCluster sites caused the failure, the disaster is over after you restore power.

To fix failures, complete the following steps.

Step	Action
1	Ensure that the filer and disk shelves have power.
2	Ensure that all Fibre Channel adapters, VI cluster adapters, cables, and switch ports are connected and working.

You cannot fix failures if the disaster causes a site to be destroyed. For example, a fire or an earthquake could destroy one of the MetroCluster sites. In this case, you fix the failure by creating a new MetroCluster configured partner at a different site.

Once the filer at the surviving site can see the disk shelves at the disaster site, Data ONTAP renames the mirrored volumes that were split by adding a number in parenthesis to the volume name. For example, if the volume name was vol1 before the disaster and the split, the renamed volume name could be vol1(1).

Reestablishing the MetroCluster configuration

Depending on the state of a mirrored volume before you forced the surviving cluster to take over its partner, you use one of two procedures to reestablish the MetroCluster configuration:

If the mirrored volume was in a normal state before the forced takeover, you can rejoin the two volumes to reestablish the MetroCluster configuration. This is the most typical case.
If the mirrored volume was in an initial resynchronization state (level-0) before the forced takeover, you cannot rejoin the two volumes. You must re-create the synchronous mirror to reestablish the MetroCluster configuration.

Rejoining the two volumes

To return the cluster to normal operation by rejoining the two volumes, complete the following steps.


sysconfig -v
Waiting for Giveback...
cf giveback
vol status -r
vol offline disaster_vol
vol mirror volume_name -v disaster_vol

Step	Action
1	Validate that you can access the remote storage by entering one of the following commands: sysconfig -v or vol status -r
2	Turn on power to the cluster node at the disaster site. Result After the cluster node at the disaster site boots up, it displays the following: Waiting for Giveback...
3	Enter the following command at the partner cluster node: cf giveback Result The cluster node at the disaster site reboots.
4	Determine which volumes are at the surviving site and which volumes are at the disaster site by entering the following command: vol status -r
5	Take the disaster site's volumes offline by entering the following command for each volume that was split: vol offline disaster_vol disaster_vol is the name of the volume at the disaster site. Note An error message appears if the volume is already offline.
6	Re-create the mirrored volumes by entering the following command for each volume that was split: vol mirror volume_name -v disaster_vol volume_name is the volume on the surviving site's cluster node. disaster_vol is the volume on the disaster site's cluster node. Result The volume_name volume rejoins the disaster_vol volume to reestablish the MetroCluster configuration.

For an example of rejoining volumes, see Example of rejoining volumes.

Re-creating the synchronous mirror

To return the cluster to normal operation by re-creating the synchronous mirror, complete the following steps.


sysconfig -v
Waiting for Giveback...
cf giveback
vol destroy plex_name
vol mirror volume_name

Step	Action
1	Validate that you can access the remote storage by entering one of the following commands: sysconfig -v or vol status -r Note A `(level-0 resync in progress)` message indicates that a plex cannot be rejoined.
2	Turn on the power to the cluster node at the disaster site. Result After the cluster node at the disaster site boots up, it displays the following: Waiting for Giveback...
3	Enter the following command at the partner cluster node: cf giveback Result The cluster node at the disaster site reboots.
4	Destroy every target plex that is in a level-0 resync state by entering the following command: vol destroy plex_name For details about synchronous mirroring, see the Data Protection Guide.
5	Re-create the mirrored volumes by entering the following command for each volume that was split: vol mirror volume_name For details about synchronous mirroring, see the Data Protection Guide.

For an example of rejoining volumes, see Example of re-creating the synchronous mirror.

Example of rejoining volumes

The following example shows the commands and status output when rejoining volumes to reestablish the MetroCluster configuration.

First, the volume status of the disaster site's storage after reestablishing access to the cluster partner at the surviving site is shown.

filer1> vol status -r

Volume mir (online, normal) (zoned checksums)

  Plex /mir/plex5 (online, normal, active)

    RAID group /filer1/plex5/rg0 (normal)

RAID Disk Device HA  SHELF BAY CHAN  Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ----- -------------- -------------

parity   8a.2   8a   0     2   FC:B  34500/70656000 35003/71687368

data     8a.8   8a   1     0   FC:B  34500/70656000 35003/71687368

Volume mir(1) (failed, out-of-date) (zoned checksums)

  Plex /mir(1)/plex1 (offline, normal, out-of-date)

    RAID group /mir(1)/plex1/rg0 (normal)

RAID Disk Device HA  SHELF BAY CHAN  Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ----- -------------- -------------

parity   6a.0   6a   0     0   FC:B  34500/70656000 35003/71687368

data     6a.1   6a   0     1   FC:B  34500/70656000 35003/71687368

  Plex /mir(1)/plex5 (offline, failed, out-of-date)

Next, the mirror is reestablished using the vol mirror -v command.

Note
The filer at the surviving site is called filer1; the filer at the disaster site is called filer2.

filer1> vol mirror mir -v mir(1)

This will destroy the contents of mir(1).  Are you sure? y

Mon Nov 18 15:36:59 GMT [filer1: 
raid.mirror.resync.snapcrtok:info]: mir: created mirror 
resynchronization snapshot mirror_resync.1118153658(filer2)

Mon Nov 18 15:36:59 GMT [filer1: raid.rg.resync.start:notice]: 
/mir/plex6/rg0: start resynchronization (level 1)

Mon Nov 18 15:36:59 GMT [filer1: raid.mirror.resync.start:notice]: 
/mir: start resynchronize to target /mir/plex6

After the volumes rejoin, the synchronous mirrors of the MetroCluster configuration are reestablished.

filer1> vol status -r mir

Volume mir (online, mirrored) (zoned checksums)

  Plex /mir/plex5 (online, normal, active)

    RAID group /mir/plex5/rg0 (normal)

RAID Disk Device HA  SHELF BAY CHAN  Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ----- -------------- -------------

parity   8a.2   8a   0     2   FC:B  34500/70656000 35003/71687368

data     8a.8   8a   1     0   FC:B  34500/70656000 35003/71687368

  Plex /mir/plex6 (online, normal, active)

    RAID group /mir/plex6/rg0 (normal)

RAID Disk Device HA  SHELF BAY CHAN  Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ----- -------------- -------------

parity   6a.0   6a   0     0   FC:B  34500/70656000 35003/71687368

data     6a.1   6a   0     1   FC:B  34500/70656000 35003/71687368

Example of re-creating the synchronous mirror

The following example shows the commands and status output when re-creating volumes to reestablish the MetroCluster configuration.

The volume status of the disaster site's storage after reestablishing access to the cluster partner at the surviving site is shown.

filer1>vol status -r

Volume mir1 (online, normal) (zoned checksums)

  Plex /mir1/plex0 (online, normal, active)

  RAID group /mir1/plex0/rg0 (normal)

RAID Disk Device HA  SHELF BAY CHAN  Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ----- -------------- -------------

parity   8a.3   8a   0     3   FC:B  34500/70656000 35003/71687368

data     8a.4   8a   0     4   FC:B  34500/70656000 35003/71687368

data     8a.6   8a   0     6   FC:B  34500/70656000 35003/71687368

data     8a.5   8a   0     5   FC:B  34500/70656000 35003/71687368

Volume mir1(1) (failed, partial) (zoned checksums)

  Plex /mir1(1)/plex0 (offline, failed, inactive)

  Plex /mir1(1)/plex6 (online, normal, resyncing)

    RAID group /mir1(1)/plex6/rg0 (level-0 resync in progress)

RAID Disk Device HA  SHELF BAY CHAN  Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ----- -------------- -------------

parity    6a.6   6a  0     6   FC:B  34500/70656000 35003/71687368

data      6a.2   6a  0     2   FC:B  34500/70656000 35003/71687368

data      6a.3   6a  0     3   FC:B  34500/70656000 35003/71687368

data      6a.5   6a  0     5   FC:B  34500/70656000 35003/71687368

The mir1(1)/plex6 plex shows that a level-0 resynchronization was in progress; therefore, an attempt to rejoin the plexes fails, as shown in the following.

filer1> vol mirror mir1 -v mir1(1)

vol mirror: Illegal mirror state for volume 'mir1(1)'

Because the mir1(1)/plex6 plex had a level-0 resynchronization in progress, the mir1(1) volume must be destroyed and the mir volume remirrored to reestablish a synchronous mirror, as shown in the following.

filer1> vol mirror mir1 -v mir1(1)

vol mirror: Illegal mirror state for volume 'mir1(1)'

filer1> vol destroy mir1(1)

Are you sure you want to destroy this volume? y

Volume 'mir1(1)' destroyed.

filer1> vol mirror mir1

Creation of a mirror plex with 4 disks has been initiated.  The 
disks need to be zeroed before addition to the volume.  The process 
has been initiated and you will be notified via the system log as 
disks are added.