Cluster messages

Kinds of cluster messages

This section lists a console message, LCD messages, and system messages that indicate conditions that you can take steps to address.

What to do when you see a message that indicates trouble

If a filer or cluster ceases to function properly, save or record as many messages as you can. If the message is not listed here, contact Network Appliance Technical Support, as described in "Getting technical assistance."

Where cluster LCD messages appear

LCD messages appear on two lines on the filer's LCD. Cluster messages, if they appear at all, appear on the first line of the LCD; other messages appear on the second line.

Cluster LCD messages that indicate trouble are accompanied by an amber LED indicator.

Cluster LCD messages that you can address

The following messages pertaining to clusters can appear on the LCD of a takeover filer. These are messages that you can take steps to address


Message	Explanation	What you do
cf disabled	Takeover has been disabled.	Follow these steps: Unless you want takeover to be disabled, issue the `cf enable` command. Check the interconnect cables to make sure that they are seated properly. Issue the `cf status` command to verify the link.
Iconnect down!	The interconnect link is broken.	Follow these steps: Check the cluster interconnect cables to make sure that they are seated properly. Issue the `cf status` command to verify the link.
Iconnect error	One of the interconnect links is broken.	Follow these steps: Check the interconnect to make sure that it is functioning properly. Issue the `cf status` command to verify the link.
L: vol volume n% rebuilt	The filer has taken over its partner and one of its own volumes is reconstructing itself.	Wait until the message disappears.
P: vol volume n% rebuilt	The filer has taken over its partner and one of its partner's volumes is reconstructing itself.	Wait until the message disappears.
Partner down	The partner is down and takeover is disabled.	If you want to take over the partner, issue the `cf enable` command, then the `cf takeover` command.
Takeover	The filer has taken over its partner.	Perform a giveback.
Takeover in nnn	The filer is counting down seconds until it takes over its partner.	Unless you want takeover to occur, issue the `cf disable` command.

The following message pertaining to a cluster can appear on the LCD of a failed filer.


Message	Explanation	What you do
taken over	The filer has been taken over.	Perform maintenance on the filer, then perform a giveback from the takeover filer.

Where syslog messages appear

Syslog messages appear by default in the /etc/messages file. For additional information about syslog messages, see the Message logging section of Chapter 2 of the System Administrator's Guide.

Syslog messages that you can address

The following syslog messages are messages that you can take steps to address. By default they appear on the system console.


Message	Explanation	What you do
cf.timed.enable: timed requires java which is not enabled on this system	Essential Java files are missing from the filer.	Follow these steps: Install the Java .zip files from the distribution CD-ROM onto the /etc/java directory of your filers. Reboot.
Interconnect is down	The cluster interconnect is not working properly. Takeover capability is disabled.	Follow these steps: Check the cluster interconnect cables to make sure that they are seated properly. If the connections are secure, and the message returns, check for Cluster Interconnect adapter messages in the /etc/messages file, then contact Network Appliance Technical Support, as described in "Getting technical assistance."
No disks were detected on Fibre Channel port B; this filer will be unable to takeover correctly	The B-port loop has not been connected to the local filer, which means that the local filer cannot take over its partner.	Follow these steps: Check the B-port loop connections to the partner's disk shelves and make sure that they are securely connected. Verify that the LRC cards are properly seated.
Root volume volume, RAID group n is missing n disks and is unusable.	A disk shelf or adapter failed, or the root volume's disks might be spread between A-port and B-port loops.	Check the disk shelves and make sure that they are wired properly.
Volume volume is missing a data disk and has dirty parity.	In this situation, the filer is refusing to perform a giveback because a volume is in degraded mode with dirty parity.	Caution: You can lose data if you perform a giveback with a volume that is missing a data disk and has dirty parity. Follow these steps: Wait until one if these events happen: Parity is clean. The data disk is reconstructed. Verify that one of the events in Step 1 has happened using the `sysconfig -r` command. Perform a giveback.
Volume volume is missing n disks from RAID group n and is being taken offline for this boot.	The volume's disks might be spread between A-port and B-port loops.	Follow these steps: Verify that the disk shelves are wired properly. Reboot the filer to clear this condition.
WARNING cluster monitor fast timeout was blocked for n secs.	The Data ONTAP kernel is supposed to schedule the cluster monitor's fast timeout thread at regular intervals to ensure that mailbox data is read and written within specific time frames. If a filer does not receive the mailbox data because the fast timeout thread is blocked, it might assume that its partner is down and attempt to perform a takeover.	If this message happens frequently, gather system utilization data (`sysstat`, profiling data, and so on), then contact Network Appliance Technical Support, as described in "Getting technical assistance" so that they can characterize the system.
X link is down	The Cluster Interconnect has detected a failure in one of its two communication channels. This circumstance does not disable takeover but deserves investigation. Note: This circumstance can also occur temporarily during the booting of a clustered filer.	Check the X (top) cable connections to make sure that the cables are seated properly.
Y link is down	The Cluster interconnect has detected a failure in one of its two communication channels. This circumstance does not disable takeover but deserves investigation. Note: This circumstance can also occur temporarily during the booting of a cluster filer.	Check the Y (bottom) cable connections to make sure that the cables are seated properly.

About console messages

Some messages appear on the console but are not written to the /etc/messages file. Most of them are self-explanatory.

Console message

The following table explains the console message "The cluster partner appears not to be operational."

Message Explanation What you do

The cluster partner appears not to be operational. You will be asked whether you want to continue. If you answer "yes", the existing cluster monitor disk state will be overwritten and this node will be rebooted. Answering "no" will halt this node with no modification to the cluster monitor disk state.

WARNING: Answering "yes" while the cluster partner is operational will have unexpected and potentially catastrophic results:

YOUR FILESYSTEMS MAY BE DESTROYED

Do you wish to continue [y/n]?
The local filer has been taken over but it appears its partner is not operational. This message is usually preceded by the following message:
This node was previously declared dead. Pausing to check cluster

partner status ... partner not responding.

Thu Aug 13 16:58:37 GMT [kbd_login_mgr]: Login from console
Follow these steps:

Verify that the partner is operational or repair it.
Reboot it.
Perform a giveback from the partner. Giveback is the only method to ensure that data is consistent after a takeover.
If you cannot repair the partner, perform one of the following actions:

Contact Network Appliance Technical Support, as described in "Getting technical assistance."

Caution: If you enter y and override the takeover, client applications might lose data that was written before the reboot.

Enter y to continue and override the takeover and allow the local filer to reboot.

Message	Explanation	What you do
The cluster partner appears not to be operational. You will be asked whether you want to continue. If you answer "yes", the existing cluster monitor disk state will be overwritten and this node will be rebooted. Answering "no" will halt this node with no modification to the cluster monitor disk state. WARNING: Answering "yes" while the cluster partner is operational will have unexpected and potentially catastrophic results: YOUR FILESYSTEMS MAY BE DESTROYED Do you wish to continue [y/n]?	The local filer has been taken over but it appears its partner is not operational. This message is usually preceded by the following message: This node was previously declared dead. Pausing to check cluster partner status ... partner not responding. Thu Aug 13 16:58:37 GMT [kbd_login_mgr]: Login from console	Follow these steps: Verify that the partner is operational or repair it. Reboot it. Perform a giveback from the partner. Giveback is the only method to ensure that data is consistent after a takeover. If you cannot repair the partner, perform one of the following actions: Contact Network Appliance Technical Support, as described in "Getting technical assistance." Caution: If you enter y and override the takeover, client applications might lose data that was written before the reboot. Enter y to continue and override the takeover and allow the local filer to reboot.

Part No. 210-02334
File: trbl22.htm
Last updated: 02/24/99 12:17:24