Library Table of Contents Index Previous Next Comments

Disk and FC-AL messages from the Storage Health Monitor

About Storage Health Monitor messages  Top

The Storage Health Monitor examines FC-AL, disk shelf, and disk events and provides two kinds of syslog messages about the events: urgent and non-urgent. Each message has a corresponding message that is sent through autosupport and is available through SNMP. One set of message strings is provided for each event. There is also an informational message.

Responding to a predicted failure event message  Top

A syslog message about a predicted failure event is an urgent message and should be acted on immediately.

Syslog message
shm: disk has reported a predicted failure (PFA) 
event: disk XX, serial_number XXXX 
Autosupport and SNMP message

DISK_FAILURE_PREDICTED!!!

Autosupport priority is alert.

Description of the problem

The disk has reported a Predicted Failure (PFA) Event. The disk's internal error processing and logging algorithm computation results are exceeding an internally set threshold. The disk is bound to fail in a matter of hours.

Action

If you experience a predicted failure event, complete the following step.

Step

Action

1

Replace the disk, as described in Chapter 5, Disk Management.

Responding to a FC-AL link failure message  Top

A syslog message about an FC-AL link failure is an urgent message and should be acted on immediately.

Syslog message
shm: link failure detected, upstream from disk: id 
XX, serial_number XXXXX
Autosupport and SNMP message

FC-AL LINK_FAILURE

Autosupport priority is err.

Description of the problem

A Fibre Channel disk or cable, if disks are in different disk shelves, might be malfunctioning and might have caused an open loop condition. This results in a loss of sync over 100 milliseconds for a downstream disk that reported it as a link failure.

Action

If you experience a FC-AL link failure, complete the following steps.

Step

Action

Note
The following steps can take several hours to complete.

1

Use the halt command to halt the filer.

2

Check the upstream neighbor device of this disk and the cable between the two if they are on different disk shelves.

3

Take out two adjacent disks and any connecting cable.

4

Replace the cable, if any, to complete the loop.

5

Bring the filer up.

6

Use the disk scrub command.

Result:

If this link is bad, the syslog message reappears.

7

Replace the cable.

8

Put in one disk.

9

Use the disk scrub command.

Result:

If this disk is bad, the syslog message reappears.

10

Put the other disk back in.

11

Use the disk scrub command.

Result:

If this disk is bad, the syslog message reappears.

12

Replace the malfunctioning disk.

Responding to a FC-AL unreliable message  Top

A FC-AL unreliable message is a non-urgent message and does not require immediate action. However, you should take the action listed in the Action column of the following table. If you need assistance in performing the actions, contact Network Appliance Technical Support.

Syslog message
shm: possible link errors on disk: id XX, serial 
number XXXXX. 
Autosupport and SNMP message

FC-AL UNRELIABLE

Autosupport priority is warning.

Description of the problem

One of a group of four Fibre Channel disks in a disk shelf or any connecting cable might be malfunctioning.

This results in a large number of invalid_CRC frames and data underruns on the loop. The invalid_CRC and underrun count has crossed its set threshold several times.

Action

If you receive a FC-AL unreliable message, complete the following steps.

Step

Action

Note
The following steps can take several hours to complete.

1

Use the halt command to halt the filer.

2

Take out the four disks in the group and any connecting cables as reported in the syslog messages.

3

Replace the cable, if any, to complete the loop.

4

Bring the filer up.

5

Use the disk scrub command.

Result:

If this link is bad, the syslog message reappears.

6

Put in one disk.

7

Use the disk scrub command.

Result:

If this disk is bad, the syslog message reappears.

8

Put the disks and cables back in, one at a time, performing Steps 3 through 7 to find the malfunctioning item.

9

Replace the malfunctioning item.

Responding to a disk degraded message  Top

A disk degraded message is a non-urgent message and does not require immediate action. However, you should take the action listed in the Action column of the following table. If you need assistance in performing the actions, contact Network Appliance Technical Support.

Syslog message
shm: disk I/O Completion times too long: disk XX, 
serial number XXXXX. 
Autosupport and SNMP message

DISK_DEGRADED-I/O

Autosupport priority is warning.

Description of the problem

Either the disk is old and slow, or it is internally recovering errors, thereby taking too long to complete an I/O. The message also indicates that there are too many I/O timeouts and retries on a disk. The disk might also be returning Command Aborted status quite often. All these can result in low data throughput rate for that disk, and lowered overall system performance.

Action

If you receive a disk degraded message, complete the following step.

Step

Action

1

The disk is failure-prone; you should replace it as described in Chapter 5, Disk Management.

Responding to a disk recovered errors message  Top

A disk recovered errors message is a non-urgent message and does not require immediate action. However, you should take the action listed in the Action column of the following table. If you need assistance in performing the actions, contact Network Appliance Technical Support.

Syslog message
shm: disk returns excessive recovered errors: 
disk XX, serial number XXXXX. 
Autosupport and SNMP message

DISK_RECOVERED_ERRORS

Autosupport priority is warning.

Description of the problem

Either the disk has found medium or hardware errors (unrecovered errors), or it has internally recovered large numbers of errors. The disk might also be returning Command Aborted status. The errors returned exceeded many times the disk vendor-specified bit error rate.

Action

If you receive a disk degraded message, complete the following step.

Step

Action

1

The disk is failure-prone; you should replace it as described in Chapter 5, Disk Management.

Responding to a loop intermittent instability message  Top

A loop intermittent instability message is an informational message and does not require any action.

Syslog message
shm: intermittent instability on the loop that is 
attached to Fibre channel adapter: id XXX, name XXXXX. NOTE: no 
immediate action needed.
Autosupport and SNMP message

None

Description of the problem

A Fibre Channel adapter, attached disk shelf, disk, cable, or connectors might cause instability on the FC-AL loop. It results in I/O completion rates that are below a set threshold.

Library Table of Contents Index Previous Next Comments