Handling disk failure

About disk failure

The effects of a disk failure on the appliance operation depend on whether the appliance is equipped with a hot spare.

Disk failure with a hot spare

If the appliance has a hot spare installed, the system enters a state called "degraded mode" after a single disk failure. In this state, the RAID feature allows the appliance to continue to run without losing data (although the appliance's performance is affected). In addition, while in degraded mode, parity calculations are performed for READs, and the appliance rebuilds the missing data on the hot spare in the background with minimal interruption to service.

The appliance logs this activity in the Messages log and does not automatically shut down. If you power off the appliance while it is in degraded mode, it stops data reconstruction. After you power on the appliance again, it restarts the data reconstruction process from the beginning.

Except for a slight loss in performance while data is rebuilt on the spare disk, the failure of a single disk is transparent to the user. The appliance exits degraded mode and returns to normal operation after it finishes reconstructing data.

Network Appliance recommends that you replace the failed disk with a new hot spare after the appliance has finished reconstructing data. This way, the appliance continues to have a hot spare that can be used in case another disk fails.

The Disk - Management page (Maintenance tab > Disk > Management) in the Appliance Manager displays which disk is reserved as the hot spare.

In addition to logging disk failure and hot spare replacement activity, the Messages log on the Log Files - Messages page (Data tab > Log Files > Messages) logs any failure in the periodic check of the hot spare.

Disk failures without a hot spare

If the appliance is not equipped with a hot spare, when a disk fails, the appliance enters a degraded mode. In this state, the RAID feature allows the appliance to continue to run without losing data, although the appliance's performance is affected.

The system shuts down after 24 hours to encourage you to replace the disk.

Caution
Replace the failed disk as soon as possible. A second disk failure will cause the entire cache of objects to be lost, and you will have to reinstall the file system.

When a disk fails, the appliance logs one of the following warning messages in the Messages log and to the system console every hour:

Parity disk in x, RAID group y, is broken. Halting in m hours.

Data disk in volume x, RAID group y, is broken. Halting in m hours.

(In the actual message, x is the volume number, y is the RAID group number, and m is the number of hours before the system halts.)

In addition, immediately before the system halts, a message similar to the following is sent to the Messages log and the system console:

Sat Oct 29 13:26:42 PDT [statd]: When a disk is broken, the 
system shuts down automatically every 24 hours to encourage you 
to replace the disk. If you reboot the system it will run for 
another 24 hours before shutting down.

The shutdown ensures that you notice the disk failure. You can restart the appliance without fixing the disk, but the appliance continues to shut itself off every 24 hours until the disk is replaced.

What happens to the data on the failed disk

If no hot spare disk is available, the failed disk data will not be reconstructed but will be inaccessible until you replace the failed disk. The failed disk data is reconstructed on the new disk.

What happens with a double disk failure

If a second disk fails while the appliance is operating in degraded mode, or if two disks fail at once, the file system is damaged and cannot be repaired. You must then reinstall the appliance software and restore the configuration from a backup. For more information, see Backing up and restoring the appliance configuration.

What happens when the parity disk fails

When the parity disk fails, the appliance operates in degraded mode until you replace the failed disk. After you have installed a replacement disk, the appliance uses the information on the data disks to rebuild the parity information onto the replacement disk.