fcstat fcal_stats [ adapter number ]
fcstat device_map [ adapter number ]
link failure counts
The drive will note a link failure event, if it cannot synchronize
it's receiver PLL for a time greater than R_T_TOV, usually on the
order of milliseconds. A link failure results in the drive initiating
a loop initialization procedure.
loss of sync counts
The drive will note a loss of sync event, if it losses PLL synchronization
for a time no greater than a small number of tens of
microseconds, but thereafter manages to resynchronize. The number of
link failures are included in this count.
invalid CRC count
Every frame a drive receives contains a checksum which spans the whole
frame. If upon receiving the frame, the checksum doesn't match, the
invalid CRC counter is incremented, and the frame "dropped".
All three events will results, to some degree, in loop disruption. A link failure is considered the most serious since it may indicate a serious, long lived, transmitter problem upstream of the drive. All three events will typically result in in-flight frames being dropped, and may result in data underruns, or SCSI command timeouts. All three events indicate an upstream loop signal integrity problem.
Note that loop disruptions of this type, even though potentially resulting in data underruns and/or SCSI command timeouts, will not result in data corruption. The host adapter driver will detect such events and will retry the associated commands. The worst case effect is a drop in performance.
In general, all counts are persistent across filer reboots and drive resets, and can only be cleared by power-cycling the drives. conditions, exception conditions, and handler code paths executed. In general, interpretation of the fields requires understanding of the internal workings of the driver. However, some of the counts kept on a per drive basis, (e.g. device_underrun_cnt, device_overrun_cnt, device_timeout_cnt) may be helpful in identifying potentially problematic drives.
Counts are not persistent across filer reboots.
The relative physical position of drives on a loop is not necessarily directly related to their loop IDs (which are in turn determined by the drive shelf IDs). The device_map sub-command is helpful therefore in determining relative physical position.
Two pieces of information are displayed, (a) the physical relative position on the loop, as if the loop was one flat space, and (b) the mapping of devices to shelves, to aid in quick correlation of disk ID with shelf tenancy.
Sun Apr 18 06:02:11 GMT [isp2100_main]: Disk 0a.43(0xfffffc0000dfba48): op
0x2a:008cbf38:0010 sector 9223976 aborted command - Parity/CRC error (b 47, 0)
Sun Feb 28 01:46:06 GMT [isp2100_main]: Disk 0a.62: Data underrun (0xfffffc0000c16dd0,0x28,0)
toaster> fcstat link_stats 0a
Device Link Loss of Loss of Invalid Frame In Frame Out
Failure Sync Signal CRC count count
count count count count
0a.0 22 688 0 0 85700 105514
0a.1 0 215 0 0 88467 108223
0a.2 0 30 0 0 88229 107859
0a.3 0 75 0 0 85981 105647
0a.4 1 30 0 0 86958 105326
0a.5 0 29 0 0 86635 107651
0a.6 0 29 0 0 130159 156811
0a.8 0 30 0 0 193487 189231
0a.9 0 31 0 0 201382 191864
0a.10 0 30 0 0 34 42
0a.11 0 32 0 0 231161 218013
0a.12 1 43 0 0 36513 19815
0a.13 0 30 0 0 182970 183862
0a.14 0 29 0 0 183365 185091
0a.16 0 30 0 0 87770 107862
0a.17 0 30 0 0 86321 105501
0a.18 0 31 0 0 88892 108832
0a.19 0 30 0 0 88183 107521
0a.20 1 31 0 0 87375 109211
0a.21 0 30 0 0 87136 106172
0a.22 0 29 0 0 104973 125573
0a.24 0 30 0 0 112242 137036
0a.25 0 30 0 0 106090 130400
0a.26 0 30 0 0 105671 129123
0a.27 0 30 0 0 105690 126090
0a.28 0 29 0 0 105613 126339
0a.29 0 29 0 0 105125 127903
0a.30 0 29 0 0 110796 132422
0a.32 1 65387 0 0 103633 124969
0a.33 0 414 0 0 88639 109109
0a.34 0 142 0 0 88372 107148
0a.35 0 129 0 0 85495 105253
0a.36 2 934 0 0 96899 120639
0a.37 0 29 0 0 351265 335597
0a.38 0 29 0 0 351008 334888
0a.40 0 30 0 0 87951 106965
0a.41 0 31 0 0 86146 105930
0a.42 0 30 0 0 88140 107450
0a.43 0 30 0 0 96994 120934
0a.44 1 30 0 0 394056 372306
0a.45 0 29 0 0 87067 105831
0a.46 0 30 0 0 89177 107585
0a.48 0 30 0 0 87592 106658
0a.49 0 30 0 0 91692 112696
0a.50 0 30 0 0 85998 104912
0a.51 0 32 0 0 86694 106126
0a.52 1 186 0 0 85942 105626
0a.53 0 29 0 0 87278 105760
0a.54 0 30 0 0 103682 124354
0a.56 0 30 0 0 30517 16449
0a.57 0 31 0 0 29266 17378
0a.58 0 29 0 0 28906 17334
0a.59 0 30 0 0 29234 16880
0a.60 0 29 0 0 41727 14465
0a.61 0 29 0 0 360425 241067
0a.62 0 30 0 0 32684 19860
0a.ha 0 1 1 0 6839257 6339541
We see that the loss of sync count is paricularly high on three drives,
0a.0 (count = 688), 0a.32 (count = 65387) and 0a.36 (count = 934). We next
examine the relative physical position using the device_map sub-command.
toaster> fcstat device_map 0a
Loop Map for adapter 0a:
Raw LIRP Frame: Port Count 57
17 ef e8 e4 e2 d9 d6 d5 d4 cd cc cb ca c3 bc ba
b9 b2 b1 ae ad a7 a6 a5 a3 98 97 90 8f 80 7c 7a
79 76 75 74 88 84 82 9f 9e 9d ac ab aa b6 b5 b4
c9 c7 c6 d3 d2 d1 e1 e0 dc
Translated Map: Port Count 57
119 0 1 2 3 8 9 10 11 16 17 18 19 24 25 26
27 32 33 34 35 40 41 42 43 48 49 50 51 56 57 58
59 60 61 62 52 53 54 44 45 46 36 37 38 28 29 30
20 21 22 12 13 14 4 5 6
Eurologic shelf mapping:
Shelf 0: -- 6 5 4 3 2 1 0
Shelf 1: -- 14 13 12 11 10 9 8
Shelf 2: -- 22 21 20 19 18 17 16
Shelf 3: -- 30 29 28 27 26 25 24
Shelf 4: -- 38 37 36 35 34 33 32
Shelf 5: -- 46 45 44 43 42 41 40
Shelf 6: -- 54 53 52 51 50 49 48
Shelf 7: -- 62 61 60 59 58 57 56
Looking at the translated map, we see that drive number 0 is the first on
the loop immediately downstream of the host adapter. Note, that the host
adapter port (119) will always appear first on the position map. Since
every filer reboot involves reinitialization of the host adapters, we
expect the first drive on the loop to see loop breakages. We can therefore
ignore the high loss of sync count for this drive.
Next we note that drive 32 is immediately downstream of drive 27, and is in fact the first drive in shelf 4. We further note that the link statistics of drive 27 are fairly clean. One can therefore, with reasonably high probability, draw the conclusion that a component upstream of drive 32 and downstream of 27 is the culprit, i.e. the cable between shelves 3 and 4, or possibly drive 32 itself or the shelf it's in.
In this case, the plan of action would involve (a) replacing the cable between shelves 3 and 4, (b) reseating drives 32, and (c) replacing shelf 4.