Cluster operation modes

Takeover operation mode

Takeover process

The takeover process is shown in the following table.

Stage	Description
1	One of these four things happen: An operator issues the `halt` command without the `-f` option. One filer in a cluster undergoes a hardware or power failure and cannot reboot. That filer is the failed filer, and the partner filer is the takeover filer. An operator issues the `cf takeover` command from a filer. The filer from which the command was issued becomes the takeover filer; the other filer becomes the failed filer. One filer in a cluster experiences a disk shelf failure and the negotiated failover function determines that the partner filer should take over for the filer with the failed shelf. Note Negotiated failover occurs only once after one or both of the filers in the cluster are booted, establishing communication between the two filers. The two filers exchange disk shelf information and determine whether negotiated failover should occur. The next time a negotiated failover decision can be made is when one or both filers are rebooted, causing the filers to establish communication again.
2	The takeover filer reserves all disk drives in the cluster, so that they do not receive commands from the failed filer.
3	The takeover filer creates an emulated filer that takes over the resources and identity of the failed filer, and executes the equivalent of a reboot of the emulated filer. For example, it uses network interface information from the failed filer configuration files to duplicate the failed filer network services.
4	The emulated filer goes through a process similar to the reboot process: All filer information preserved across filer reboot is preserved in a takeover. Filer information that is lost in a reboot, such as CIFS connection state, is lost in a takeover.
5	Users of the failed filer experience the effects of a reboot of the failed filer, and then client service continues.

Messages indicating a successful takeover

During a typical takeover, the cluster monitor issues messages on the console of the takeover filer. In the following example, the cf takeover command was issued:

filer1> Tue Jul 21 23:02:02 GMT [filer1: 
cf.misc.operatorTakeover:warning]: Cluster monitor: takeover 
initiated by operator

Tue Jul 21 23:02:11 GMT [filer1: 
cf.fsm.nfo.acceptTakeoverReq:warning]: Negotiated failover: 
accepting takeover request by partner, reason: Operator initiated 
cf takeover. Asking partner to shutdown gracefully; will takeover 
in at most 180 seconds.

Tue Jul 21 23:02:20 GMT [filer1: 
cf.fsm.partnerNotResponding:notice]: Cluster monitor: partner not 
responding

Then the following messages appear:

Tue Jul 21 23:03:06 GMT [filer1: 
cf.fsm.nfo.partnerShutdown:warning]: Negotiated failover: partner 
has shutdown

Tue Jul 21 23:03:06 GMT [filer1: cf.fsm.stateTransit:warning]: 
Cluster monitor: UP --> TAKEOVER

Tue Jul 21 23:03:06 GMT [filer1: cf.fm.takeoverStarted:warning]: 
Cluster monitor: takeover started

Tue Jul 21 23:03:10 GMT [filer1: disk_admin:warning]: Resetting all 
devices on ISP2100 in slot 7.

Reboot messages follow as the emulated filer boots up. Then the following message appears and indicates a successful takeover:

Tue Jul 21 23:04:22 GMT [filer1 (takeover): 
cf.fm.takeoverComplete:warning]: Cluster monitor: takeover 
completed

filer1(takeover)>

Note
Negotiated failover messages are not printed when you issue a cf takeover -f command, a cf forcetakeover command, or a cf forcetakeover -f command.

Messages indicating a successful negotiated takeover

During negotiated takeover, the cluster monitor issues messages on the takeover filer similar to the following:

filer2> Thu May  4 01:41:15 GMT [ha4: 
cf.fsm.nfo.acceptTakeoverReq:warning]: Negotiated failover: 
accepting takeover request by partner, reason: partner sees fewer 
shelves on its A loop (4) than we do (5).  Asking partner to 
shutdown gracefully; will takeover in at most 180 seconds.

Additional messages indicate that the takeover filer is waiting for the failed filer to respond. When the failed filer shuts down, the partner filer begins the takeover process:

Thu May  4 01:41:32 GMT [filer2: cf.fsm.stateTransit:warning]: 
Cluster monitor: UP --> TAKEOVER

Thu May  4 01:41:32 GMT [filer2: cf.fm.takeoverStarted:warning]: 
Cluster monitor: takeover started

Thu May  4 01:41:36 GMT [filer2: disk_admin:error]: Resetting all 
devices on ISP2100 in slot 7.

...

The following message indicates a successful negotiated takeover:

Thu May  4 01:41:47 GMT [filer2 (takeover): 
cf.fm.takeoverComplete:warning]: Cluster monitor: takeover 
completed

filer2(takeover)>

Takeover result diagram

The following figure shows the state of a cluster after a takeover of Filer 1 by Filer 2.