Displaying storage system and RLM information

About displaying information

The RLM provides several ways to display information about the storage system and the RLM itself. You can display real-time information using the commands in the admin or advanced mode, or you can display information stored in the RLM's System Event Log (SEL) or Console log. You can also view the information displayed in the additional AutoSupport messages generated by the RLM. Most of the information is stored in the SEL or in captured console messages.

Displaying real-time information

RLM CLI in admin mode

Using the RLM CLI commands in admin mode, you can view information about:

The storage system power status (system power status)
The status of the RLM (rlm status)
The version of the RLM (version)

The following examples show how information is displayed using commands at the RLM admin mode prompt:

RLM toaster> system power status

Power supply1 status:

    Present: yes

    Turned on by Agent: yes

    Output power: yes

    Input power: yes

    Fault: no

Power supply 2 status:

    Present: yes

    Turned on by Agent: yes

    Output power: yes

    Input power: yes

    Fault: no

RLM toaster> rlm status

eth0  Link encap:Ethernet HWaddr 00:A0:98:01:9C:4B

      inet addr:10.41.42.73.231 Bcast:10.255.255.255

         Mask:255.255.255.0

      UP BROADCAST RUNNING MULTICAST   MTU:1500   Metric:1

      RX packets:8972  errors:0 dropped:0  overruns:0   frame:0

      TX  packets:72  errors:0  dropped:0  overruns:0  carrier:0

      collisions:0  txqueuelen:100

      RX bytes:622724  (608.1 kb)  TX bytes:8765 (8.5 kb)

      Interrupt:15

RLM toaster> version

serial#=123456

part#110-00030

rev#12

Agent revision: 12

Primary-

RLM_version=x.y (date)



Backup-

RLM_version=x.y (date)



Booted primary image

The RLM CLI in advanced mode

Using the RLM CLI commands in advanced mode, you can view information about:

Internal RLM statistics (rlm status -v)
The RLM environmental sensor (rlm sensors)

The following examples show how information is displayed using commands at the RLM admin mode prompt (note that the characters "..." indicate details have been omitted):

RLM toaster*> rlm status -v

eth0  Link encap:Ethernet HWaddr 00:A0:98:01:9C:4B

      inet addr:10.41.42.73.231 Bcast:10.255.255.255

         Mask:255.255.255.0

      UP BROADCAST RUNNING MULTICAST   MTU:1500   Metric:1

      RX packets:8972  errors:0 dropped:0  overruns:0   frame:0

      TX  packets:72  errors:0  dropped:0  overruns:0  carrier:0

      collisions:0  txqueuelen:100

      RX bytes:622724  (608.1 kb)  TX bytes:8765 (8.5 kb)

      Interrupt:15

packet reader daemon

----------------------------------

restarts                         1

port config errors               0

...

packet writer daemon

----------------------------------

restarts                         0

port config errors               0

...

console logger daemon

----------------------------------

logger restarts                  0

logger input packets             0

...

downbeat daemon

----------------------------------

Downbeat restarts                0

Downbeat packets                 0

...

upbeat daemon

----------------------------------

Upbeat restarts                  1

Upbeat packets                  93



ECC memory 

----------------------------------

total corrections                0

totat uncorrectable errors       0

...

Watcher daemon

----------------------------------

watcher restarts                 0

agentd restarts                  0

...

RLM toaster*> rlm sensors

Sensor     Sensor  Sensor   Critical  Warning   Warning   Critical

Name       State   Reading  Low       Low       High      High

========   =====   ======   ======    ======    ======    =====

Temperature Normal  19C       N/A        0C         45C        60C

System Event Log

The RLM has a non-volatile memory buffer that stores up to 4,000 records of system events in a System Event Log (SEL). When the buffer is full, the oldest records are overwritten by the newest records.

The records contain the following data:

Hardware events detected by the RLM--for example, system sensor status about power supplies, voltage, or other components
Errors (generated by the storage system or the RLM) detected by the RLM--for example, a communication error, a fan failure, a memory or CPU error, or a "boot image not found" message
Critical software events sent to the RLM by the storage system--for example, a system panic, a communication failure, an unexpected CFE prompt, a CFE boot failure, or a user-triggered "down system" as a result of issuing the system reset or system power cycle command
Note
The SEL uses the RLM's clock to time-stamp events. RLM begins synchronizing its clock with the system clock as soon as the storage system boots up. However, synchronizing takes a few seconds. If events occur during these few seconds, they are time-stamped January 1, 1970.

Recent records from the SEL are attached to the AutoSupport messages sent by the RLM.

You access the SEL by using the RLM events command, as shown in the following examples.

RLM toaster> events info

SEL version: 4

Maximum number of records: 4000



Number of records: 180

Record size: 64 Bytes

Oldest record's id: 1

Latest records's id: 180

RLM toaster> events oldest 1

Record 1: Wed Mar 30 20:17:38 2005 [RLM CLI.notice]: "log in"

RLM toaster> events all

(displays all events)

RLM toaster> events newest 2

Record 179: Wed Mar 30 10:32:49 2005 [Agent Event.warning]: FIFO

    0x001A - Agent XYZ, PS2_PRSNT easserted.

Record 180: Wed Mar 30 10:32:50 2005 [Agent Event.warning]: FIFO

    0x8017 - Agent XYZ, FAN_ALERT2 easserted

RLM toaster> events search WD

Record 5: Tue Mar 29 07:39:40 2005 [Agent Event.warning]: FIFO 
0x8FFF - Agent XYZ, L1_WD_TIMEOUT asserted.

Record 6: Tue Mar 29 07:39:42 2005 [Agent Event.critical]: FIFO 
0x8FFE - Agent XYZ, L2_WD_TIMEOUT asserted

Console log

The RLM monitors the storage system console regardless of whether any administrators are logged in or connected to the console. When storage system messages are sent to the console, the RLM stores them in the console log, which resides in a 96-KB buffer in its main memory. The console log can store approximately 2,000 lines of system console messages. When the buffer is full, the oldest messages are overwritten by the newest messages.

The console log persists as long as the RLM has power from either of the storage system's power supplies. Since the RLM operates with standby power, it remains available even when the storage system is power-cycled or turned off.

If the autosupport.content option is set to complete, and a "down filer," a system hang, or a reboot loop condition occurs, the console logs are attached to the AutoSupport messages sent by the RLM.

You display the contents of the console log with the system log command, as shown in the following example.

RLM toaster> system log

toaster> Wed Mar 30 10:32:49 GMT
    [monitor.fan.critical:EMERGENCY] : SYS 2 FAN 2 is spinning
    below the tolerable speed. Replace it immediately to avoid
    overheating.
toaster> Wed Mar 30 10:33:10 GMT
    [monitor.fan.critical:EMERGENCY] : current speed is 82 RPM,
    critical low is 400 RPM

AutoSupport messages

For storage systems with RLM, there are two additional types of AutoSupport messages:

RLM-generated AutoSupport messages about the storage system
Storage system-generated AutoSupport messages about the RLM

For information about how to interpret AutoSupport messages, see Contents of AutoSupport messages.

RLM-generated AutoSupport messages

These messages include the following information:

In the subject line--A system notification from the RLM of the storage system, listing the system condition or event that caused the AutoSupport message, and the log level (for information about log level, see Event and log-level descriptions)
In the message body--The RLM configuration and version information, the storage system ID, serial number, model and host name
In the gzipped attachments--the System Event Logs, the system sensor state as determined by the RLM, and console logs. (Console logs can be omitted by setting the autosupport.content option to minimal)

Typical RLM-generated AutoSupport messages occur in the following conditions:

The storage system reboots unexpectedly.
The storage system stops communicating with the RLM.
A Watchdog reset occurs.
The storage system is power-cycled.
Firmware POST errors occur.
A user-initiated AutoSupport message occurs.

Storage system-generated AutoSupport messages

include the following information:

In the subject line--A system notification from the name of the storage system with the RLM, a description of the RLM condition or event that caused the AutoSupport message, and the log level (for information about log level, see Event and log-level descriptions)
In the message body--A time stamp, the system software version and storage system ID, host name, and output from the sysconfig -a command
In the gzipped attachments--messages from EMS, rc, exports, hosts, resolv_conf, nsswitch_conf, and cm_stats

Typical storage system AutoSupport messages about the RLM occur in the following conditions:

The RLM stops communication with the storage system.
The RLM software fails.
The RLM hardware fails.