Managing a cluster in normal mode

Normal mode management tasks

You manage a cluster in normal mode by performing the following actions:

Monitoring cluster status
Viewing information about the cluster
- Displaying the partner's name
- Displaying disk information
Enabling and disabling takeover
Enabling and disabling immediate takeover of a panicked partner
Halting a node without takeover
Performing a takeover

Monitoring cluster status

To determine whether the cluster feature is enabled and whether the other node in the cluster is up, complete the following step.

Step	Action
1	Enter the following command: cf status Example filer1> cf status Cluster enabled, filer2 is up. This verifies the link between the clustered nodes and tells you that both filer1 and filer2 are functioning and available for takeover. Result If Data ONTAP returns "Cluster disabled," you disabled the cluster feature. You can reenable clustering using the `cf enable` command. Note Data ONTAP can disable cluster failover if a software or hardware problem exists that prevents a successful takeover. In this case, the message returned from the `cf status` command describes the reason why clustering is disabled.

Step

Action

Enter the following command:

cf status

Example

filer1> cf status

Cluster enabled, filer2 is up.

This verifies the link between the clustered nodes and tells you that both filer1 and filer2 are functioning and available for takeover.

Result

If Data ONTAP returns "Cluster disabled," you disabled the cluster feature. You can reenable clustering using the cf enable command.

Note
Data ONTAP can disable cluster failover if a software or hardware problem exists that prevents a successful takeover. In this case, the message returned from the cf status command describes the reason why clustering is disabled.

Description of cluster status messages

The following table shows some of the messages that the cf status command can display.

Message	Meaning
Cluster enabled, partner_name is up.	The cluster is operating normally.
partner_name_1 has taken over partner_name_2.	One node in the cluster took over the other node in the cluster.
Interconnect not present.	The system does not recognize the existence of a cluster interconnect adapter.
Interconnect is down.	The cluster interconnect adapter cannot access the partner. This might be due to cabling problems or the partner might be down.
Interconnect is up.	The cluster interconnect adapter is active and can transmit data to the partner.
partner_name_1 has detected a mailbox disk error, take over of partner_name_2 disabled.	One node in the cluster cannot access multiple mailbox disks. Check access to both the local and partner root volumes and mirrors, if they exist. Also check for disk or FC-AL problems or offline storage adapters.
partner_name_2 may be down and has disabled take over by partner_name_1.	One node in the cluster might be down.
Version mismatch	The partner node has an incompatible version of Data ONTAP.
partner_name_1 is attempting takeover of partner_name_2. takeover is in module n of N modules.	A takeover is being attempted (includes information about how far the takeover has progressed).
partner_name_1 has taken over partner_name_2, giveback in progress. giveback is in module n of N modules.	A giveback is being attempted (includes information about how far the giveback has progressed).
partner_name_1 has taken over partner_name_2, partner_name_2 is ready for giveback.	The takeover node received information that the failed node is ready for giveback.
partner_name_1 has taken over partner_name_2, partner_name_2 is ready for giveback. Automatic giveback is disabled due to exceeding retry count.	The takeover node received information that the failed node is ready for giveback, but giveback cannot take place because the number of retries exceeded the limit.

Displaying the partner's name

To determine the name of the other node in the cluster, complete the following step.

Step	Action
1	Enter the following command: cf partner Note If the node does not yet know the name of its partner, because the cluster is new, this command returns "partner".

Displaying disk information on a cluster

In a cluster, the sysconfig and aggr status command outputs are special in that they display information about both nodes.

For each node, the sysconfig command output displays disks on both FC-AL loop A and FC-AL loop B:

The information about disks on FC-AL loop A is the same as for storage systems not in a cluster.
The information about disks on FC-AL loop B is for hardware only; the sysconfig command displays information about the adapters supporting the disks. The command does not show whether a disk on FC-AL loop B is a file system disk, spare disk, or parity disk.

To display disk information for both nodes in a cluster, complete the following step.

Step	Action
1	Enter one of the following commands: sysconfig -r or aggr status -r

Enabling and disabling takeover

You might want to disable takeover if you are doing maintenance that typically causes a takeover. You can reenable takeover after you finish maintenance.

To enable or disable takeover on a cluster, complete the following step.

Step	Action
1	Enter the following command: cf enable\|disable Use `cf enable` to enable takeover or `cf disable` to disable takeover. Note You can enable or disable takeover from either node in the cluster.

Enabling and disabling immediate takeover of a panicked partner

You can enable a clustered node to take over immediately when its partner panics. This shortens the time between the initial failure and when service is fully restored, because a cluster can fail over in less time than it takes for a storage system to recover from a panic. However, the subsequent giveback causes another brief outage, so if service disruption is an issue (for example, if you have CIFS clients), you might not want to incur the initial takeover.

Note
If the panicked node needs to do a core dump and a spare disk is available, the core dump is directed to the spare disk and the partner takes over immediately (within 60 seconds) without waiting until the core dump is completed.

If the panicked node needs to do a core dump and no spare disk is available, the node does not perform a core dump; instead, it prints a message that a spare disk is needed. When the node reboots, it logs that no core dump took place.

To enable or disable immediate takeover of a panicked partner, complete the following steps.

Step	Action
1	Ensure that you enabled cluster takeover with the `cf enable` command.
2	Enter the following command: options cf.takeover.on_panic [on\|off] Use `on` to enable immediate takeover of a failed partner or `off` to disable immediate takeover. If this option is disabled, normal takeover procedures apply. Note If you enter this command on one node in a cluster, the value applies to both nodes in the cluster. The setting of this option is persistent across reboots.

Step

Action

Ensure that you enabled cluster takeover with the cf enable command.

Enter the following command:

options cf.takeover.on_panic [on|off]

Use on to enable immediate takeover of a failed partner or off to disable immediate takeover.

If this option is disabled, normal takeover procedures apply.

Note
If you enter this command on one node in a cluster, the value applies to both nodes in the cluster.

The setting of this option is persistent across reboots.

Halting a node without takeover

You can halt the node and prevent its partner from taking over. For example, you might need to perform maintenance on both the storage system and its disks and want to avoid an attempt by the partner node to write to those disks.

To halt a node in a cluster without causing a takeover, complete the following step.

Step	Action
1	Enter the following command: halt -f

Commands for performing a takeover

You can initiate a takeover on a node in a cluster to perform maintenance on that node while still serving the data on its disks to users. The following table lists and describes the commands you can use when initiating a takeover.

Command	Description
cf takeover	Initiates a takeover of the partner of the local node. Takeover is aborted if a core dump is in progress on the partner (if the `cf.takeover.on_panic` option is set to `off`). The takeover starts either after the partner halts successfully or after a timeout.
cf takeover -f	Initiates an immediate takeover of the partner of the local node regardless of whether the other node is dumping its core. The partner node is not allowed to halt gracefully.
cf forcetakeover	Tells the cluster monitor to ignore some configuration problems that would otherwise prevent a takeover, such as unsynchronized NVRAM due to a faulty cluster interconnect connection. It then initiates a takeover of the partner of the local node. Attention Use `cf forcetakeover` only in certain controlled circumstances, such as when you cannot get `cf takeover` to succeed. You can lose up to 10 seconds worth of data that was committed to NVRAM but not to disk when you use this command.
cf forcetakeover -d	Initiates a takeover of the local partner even in the absence of a quorum of partner mailbox disks. The `cf forcetakeover -d` command is valid only if the cluster_remote license is enabled. Attention Use the `-d` option only after you verify that the partner is down. Note The `-d` option is used in conjunction with RAID mirroring to recover from disasters in which one partner in a cluster is not available. For more information, see the Data Protection Online Backup and Recovery Guide.
cf forcetakeover -n	Initiates a takeover for a nondisruptive upgrade. For more information, see the Upgrade Guide.