Managing a cluster

Managing a cluster in normal mode

Normal mode management tasks

You manage a cluster in normal mode by performing the following actions:

Monitoring the status of the cluster as a whole
Viewing information about the cluster
Enabling and disabling takeover on a cluster to perform software upgrades or other maintenance
Halting a filer in a cluster without causing a takeover
Performing license operations on the cluster feature
Performing a takeover on the partner filer
Enabling and disabling the negotiated failover feature on a cluster

Monitoring cluster status

To find out whether the cluster feature is enabled and whether the other filer in the cluster is up, complete the following step.

Step	Action
1	Enter the following command: cf status [-t] `-t` displays the status of the node as time master or slave. Example filer1> cf status Cluster enabled, filer2 is up. This verifies the link between the clustered filers and tells you that both filer1 and filer2 are functioning and available for takeover.

Step

Action

Enter the following command:

cf status [-t]

-t displays the status of the node as time master or slave.

Example

filer1> cf status

Cluster enabled, filer2 is up.

This verifies the link between the clustered filers and tells you that both filer1 and filer2 are functioning and available for takeover.

If you see a "Cluster disabled" message, it might mean that

You disabled the cluster feature. To reenable the cluster feature, use the cf enable command.
The system disabled clustering because the interconnect failed.
Data ONTAP cannot access the cluster mailbox disks.

Description of cluster status messages

The following table shows the messages that the cf status command can display.

Message	Meaning
Cluster enabled, partner_name is up.	The cluster is operating normally.
partner_name_1 has taken over partner_name_2.	One filer in the cluster has taken over the other filer in the cluster.
Interconnect not present.	The system does not recognize the existence of a cluster interconnect adapter.
Interconnect is down.	The cluster interconnect adapter cannot access the partner. This might be due to cabling problems or the partner might be down.
Interconnect is up.	The cluster interconnect adapter is active and can transmit data to the partner.
X link is down; Y link is up.	The ServerNet X link (upper cable) is currently down. This condition might occur briefly when a cluster node is booting. If this condition persists, check the cable seating.
X link is up; Y link is down.	The ServerNet Y link (lower cable) is currently down. This condition might occur briefly when a cluster node is booting. If this condition persists, check the cable seating.
partner_name_1 has detected a mailbox disk error, take over of partner_name_2 disabled.	One filer in the cluster is unable to access multiple mailbox disks. Check access to both the local and partner root volumes and mirrors, if they exist. Also check for disk or FC-AL problems or offline storage adapters.
partner_name_2 may be down and has disabled take over by partner_name_1.	One filer in the cluster may be down. Check its state and reboot if necessary.
Version mismatch	The partner node has an incompatible version of Data ONTAP.
partner_name_1 is attempting takeover of partner_name_2. giveback is in module n of N modules.	A takeover is being attempted, with information on how far the takeover has progressed.
partner_name_1 has taken over partner_name_2	A takeover has occurred.
partner_name_1 has taken over partner_name_2, giveback in progress. giveback is in module n of N modules.	A giveback is being attempted, with information on how far the giveback has progressed.
partner_name_1 has taken over partner_name_2, partner_name_2 is ready for giveback.	The takeover filer has received information that the failed filer is ready for giveback.
partner_name_1 has taken over partner_name_2, partner_name_2 is ready for giveback. Automatic giveback is disabled due to exceeding retry count.	The takeover filer has received information that the failed filer is ready for giveback, but giveback cannot take place because the number of retries has exceeded the limit.

Note
Even if one of the ServerNet interconnect links (X or Y) is down, cluster communication continues over the link that is up.

Displaying the partner's name

To find out the name of the other filer in the cluster, complete the following step.

Step	Action
1	Enter the following command: cf partner

Cluster configuration information

In a cluster, the sysconfig and vol status command outputs are special in that they display information about both filers.

For each filer, the sysconfig command output displays disks on both FC-AL loop A and FC-AL loop B:

The information about disks on FC-AL loop A is the same as for filers not in a cluster.
The information about disks on FC-AL loop B is for hardware only; the sysconfig command displays information about the adapters supporting the disks. The command does not show whether a disk on FC-AL loop B is a file system disk, spare disk, or parity disk.

Displaying disk information on a cluster

To display disk information for both filers in a cluster, complete the following step.

Step	Action
1	Enter one of the following commands: sysconfig -r or vol status -r

Enabling and disabling takeover

You might want to disable takeover if you are doing maintenance that typically causes a takeover, then reenable takeover when you have finished maintenance.

To enable or disable takeover on a cluster, complete the following step.

Step	Action
1	Enter the following command: cf enable\|disable Use `cf enable` to enable takeover, and `cf disable` to disable takeover. You can enable or disable takeover using the command from either filer in the cluster.

Enabling and disabling immediate takeover of a panicked partner

You can enable a clustered filer to take over immediately when its partner panics. If the panicked filer needs to do a core dump, and a spare disk is available, the core dump is directed to the spare disk and the partner will take over immediately (within 60 seconds) without having to wait until the core dump is completed.

Note
Until the core dump to the spare disk is complete, the spare disk will not be allocated for use as a file system disk.

If the panicked filer needs to do a core dump and no spare disk is available, the filer does not perform a core dump and prints out a message that a spare disk is needed. When the filer reboots, it logs the fact that no core dump took place.

To enable or disable immediate takeover of a panicked partner, complete the following step.

Note
For immediate takeover to occur, you must enable cluster takeover using the cf enable command.

Step	Action
1	Enter the following command: options cf.takeover.on_panic [ on \| off ] Use `on` to enable immediate takeover of a failed partner or `off` to disable immediate takeover. If this option is disabled, normal takeover procedures will occur. Note By default this option is turned off, except when the iSCSI and FCP licenses have been previously enabled. If this command is entered on one filer in a cluster, the value applies to both filers in the cluster. The setting of this option is persistent across reboots.

Step

Action

Enter the following command:

options cf.takeover.on_panic [ on | off ]

Use on to enable immediate takeover of a failed partner or off to disable immediate takeover.

If this option is disabled, normal takeover procedures will occur.

Note
By default this option is turned off, except when the iSCSI and FCP licenses have been previously enabled. If this command is entered on one filer in a cluster, the value applies to both filers in the cluster. The setting of this option is persistent across reboots.

Halting a filer without takeover

You can halt the filer and prevent its partner from taking over; for example, if you need to perform maintenance on both the filer and its disks.

To halt a filer in a cluster without causing a takeover, complete the following step.

Step	Action
1	Enter the following command: halt -f

Licensing and unlicensing the cluster feature

You can license or unlicense the cluster feature only in normal mode with takeover disabled.

To unlicense the cluster feature, complete the following steps.

Step	Action
1	If the filer has...	Then...
	Taken over its partner	Enter the following command to perform a giveback: cf giveback
	Not taken over its partner	Go to Step 2.
2	Enter the following command to disable clustering: cf disable
3	Enter the following command to unlicense clustering: license delete cluster
4	Enter the following command to reboot the filer and make the change take effect: reboot

Takeover commands and descriptions

You can initiate a takeover on a filer in a cluster to perform maintenance on that filer while still serving the data on its disks to users. The following table lists the commands you can use when initiating a takeover and a description of those commands.

Command	Description
cf takeover	Initiates a takeover of the partner of the local filer. Takeover is aborted if a core dump is in progress on the partner (only if the `cf.takeover.on_panic` option is set to false).
cf takeover -f	Initiates a takeover of the partner of the local filer regardless of whether the other filer is dumping its core (only if the `cf.takeover.on_panic` option is set to false).
cf forcetakeover	Tells the cluster monitor to ignore some configuration problems that otherwise prevent a takeover, such as unsynchronized NVRAM due to a faulty cluster interconnect connection. It then initiates a takeover of the partner of the local filer. Caution Use `cf forcetakeover` only in certain controlled circumstances, such as when you cannot get `cf takeover` to succeed. You can lose uncommitted data with this command.
cf forcetakeover -d	Initiates a takeover of the local partner even in the absence of a quorum of partner mailbox disks. The `cf forcetakeover -d` command is valid only if the cluster_remote license has been enabled. Caution Use the `-d` option only after you have verified that the partner is down. Note The `-d` option can be used in conjunction with RAID mirroring to recover from disasters where one partner in a cluster is not available. For more information, see the Data Protection Guide.