Handling disk failure

About disk failure

The effects of a disk failure on the appliance operation depend on whether or not the appliance is equipped with a hot spare. In either case, during a disk failure situation, an appliance will run in a degraded mode until the cause is remedied or shut down occurs.

Running in degraded mode

RAID-4 support enables the appliance to continue running in certain failure situations but in a degraded mode.

The appliance will enter degraded mode if either of the following conditions exist:

Low NVRAM battery--NVRAM support fails. This condition occurs when the battery that provides power to the NVRAM card becomes low.
Degraded RAID group--Single-disk failure. This condition occurs when a single disk, either data or parity, fails in a multi-disk system.

While running in degraded mode, the appliance will continue to run without losing data, but with the following reductions in performance:

Reduced risk protection due to the failed component
Calculation of parity for READS only
Entering proxy-only mode after 24 hours
Shutting down after 120 hours

When the appliance is rebooted, the countdown starts again. If the degraded mode condition is alleviated, the appliance will return to a fully functioning state and cease proxy-only mode (if the appliance had entered this mode).

Note
The appliance will panic if it enters double-degraded mode; that is, the appliance experiences both low-battery and degraded RAID group conditions. A double disk failure will also panic the appliance.

For more information about disk failures, see Chapter 16, Troubleshooting.

Disk failure with a hot spare

If the appliance has a hot spare installed, the system enters degraded mode after a single disk failure. In this state, the RAID feature enables the appliance to continue to run without losing data, although the appliance's performance is affected.

Note
NetCache appliance single-disk and internal disk systems do not support hot spare disks or the hot-swapping feature described in this section.

The appliance logs this activity in the Messages log and does not automatically shut down. If you power off the appliance while it is in degraded mode, it stops data reconstruction. After you power on the appliance again, it restarts the data reconstruction process from the beginning.

Except for a slight loss in performance while data is rebuilt on the spare disk, the failure of a single disk is transparent to the user. The appliance exits degraded mode and returns to normal operation after it finishes reconstructing data.

Network Appliance recommends that you replace the failed disk with a new hot spare after the appliance has finished reconstructing data. This way, the appliance continues to have a hot spare that can be used in case another disk fails.

The Disk - Management page (Maintenance tab > Disk > Management) in the Appliance Manager displays which disk is reserved as the hot spare.

In addition to logging disk failure and hot spare replacement activity, the Messages log on the Log Files - Messages page (Data tab > Log Files > Messages) logs any failure in the periodic check of the hot spare.

Disk failure without a hot spare

If the appliance is not equipped with a hot spare, when a disk fails, the appliance enters degraded mode. In this state, the RAID feature allows the appliance to continue to run without losing data, although the appliance's performance is affected.

Note
The disk failure process described in this section does not apply to C1100 or C1105 appliances, for which disk replacement is not an option.

In consideration that many appliance installations are in remote locations and that failures in such locations might not be easily corrected, the system continues to run in degraded mode, but shuts down every 120 hours of operation. This applies to all NetCache appliance platforms.

Caution
Replace the failed disk as soon as possible. A second disk failure will cause the entire cache of objects to be lost, and you will have to reinstall the file system.

When a disk fails, the appliance logs one of the following warning messages in the Messages log:

Parity disk in x, RAID group y, is broken. Halting in m hours.

Data disk in volume x, RAID group y, is broken. Halting in m hours.

(In the actual message, x is the volume number, y is the RAID group number, and m is the number of hours before the system halts.)

In addition, every hour thereafter, a message similar to the following is sent to the Messages log and the system console:

Wed Aug 28 16:00:00 PDT 
[monitor.shutdown.brokenDisk.pending:warning]: data disk in RAID 
group /vol0/plex0/rg0 is broken. Halting system in 120 hours.

What happens to the data on the failed disk

If no hot spare disk is available, the failed disk data will not be reconstructed but will be inaccessible until you replace the failed disk. The failed disk data is reconstructed on the new disk.

What happens with a double disk failure

If a second disk fails while the appliance is operating in degraded mode, or if two disks fail at once, the file system is damaged and cannot be repaired. You must then reinstall the appliance software and restore the configuration from a backup. For more information, see Chapter 16, Troubleshooting, and the topics on backing up and restoring a system configuration file in the Appliance Manager online Help.

What happens when the parity disk fails

When the parity disk fails, the appliance operates in degraded mode until you replace the failed disk. After you have installed a replacement disk, the appliance uses the information on the data disks to rebuild the parity information onto the replacement disk.

Recovering from disk failure

When the degraded mode condition is alleviated, the appliance will return to a fully functioning state and cease proxy-only mode (if the appliance entered this mode).