logo

Manual Pages


Table of Contents

NAME

na_nfsstat - display NFS statistics

SYNOPSIS

nfsstat interval

nfsstat -h [ ip_address | host_name ]

nfsstat -l

nfsstat -z

nfsstat -d

nfsstat -C

nfsstat [ -c ] [ -t ]

DESCRIPTION

nfsstat displays statistical information about NFS (Network File System) and RPC (Remote Procedure Call) for the filer. That includes statistics for NFS versions 2, 3 and 4. It can also be used to reinitialize this information. If no arguments are given, nfsstat displays statistical information since last zeroed with the -z option (or since reboot if statistics have not been zeroed).

If the interval argument is specified, nfsstat continuously displays the summary information for the following NFS requests: getattr, lookup, readlink, read, write, create, remove, and readdir/readdirplus. The first line of data displayed, and every 20th line thereafter, contains cumulative statistics. Each subsequent line shows incremental statistics for the interval (in seconds) since the last display.

Per-client statistics can also be collected and displayed by enabling the nfs.per_client_stats.enable options (using the options command - see na_options(1)) and invoking nfsstat with the -h or the -l options. Per-client statistics are collected for up to the first 256 NFS clients that have mounted the filesystem on the given filer.

OPTIONS

-h
Displays per-client statistics since last zeroed with the -z option (or since reboot if statistics have not been zeroed). The statistics are displayed on a per-client basis, with the IP address and host name (where available) of each client being displayed at the head of each client's block of statistics.

If an optional IP address or host name is specified with the -h option, only the statistics associated with this client are displayed.

Statistics for read and write sizes are also displayed as number of read or write requests made by each client grouped by size of request. Without this option, the cumulative statistics for all clients are displayed.

-l
Displays a list of the clients whose statistics have been collected on a per-client basis, along with the total NFS calls for that client since last reboot, or last zeroed with the -z option, the count being displayed both as the actual count and as a percentage of calls from all clients.

-z
Zeroes (reinitializes) the current cumulative and per client statistics. (However, statistics since boot are also retained.) The per clients statistics also get reinitialized when nfs.per_client_stats.enable option is turned off.

-c
Includes reply cache statistics in the data displayed. The reply cache is used to cache replies to non-idempotent nfs requests, so that in case of a retransmit, the correct reply can be sent back to the client. Non-idempotent operations do not return same results if repeated e.g. remove, create. The various fields displayed are:

In Progress
Number of requests which were being processed at the time retransmit requests ware received for them.

Delay Hits
Number of requests which were dropped because they came in within nfs_rc_delay of sending out the reply. This field is deprecated. A zero value will always be printed for this field. This is to maintain backward compatibility with previous ONTAP releases.

Misses
Number of requests that were not found in the reply cache. These also include requests coming in for the first time.

Idempotent
Number of idempotent requests saved in the reply cache for the purpose of improving performance.

Non-idempotent Number of non-idempotent requests replied to from the reply cache.

-t
Displays the statistics since boot time, rather than since the last time they were zeroed.

-C
Displays the number and type of NFS version 2 and 3 requests received by all FlexCache volumes. In addition we display the number of each type that resulted in a cache hit.

-d
In addition to reply cache statistics, also includes statistics about incoming messages and allocated mbufs. The various fields displayed are:

nfs cache size
NFS reply cache size. The cache size depends on the Filer memory.

hash size
NFS reply cache hash size.

num msg
Number of incoming requests (over UDP) dropped because there are no free entries in the NFS operation table.

too many mbufs
Number of requests (over UDP) dropped because they require more mbufs than are available.

rpcErr
Total number of rpc errors encountered while generating the rpc reply header.

svrErr
Total number of RPC errors encountered while processing the request.

msg queued
Total number of NFS connections (over TCP) that have been deferred waiting for NFS resources to become available. The NFS requests queued on these connections will be processed once these resources become available. Connections can get queued up when the filesystem is busy doing back to back CPs.

no msg paused
This is always set to zero.

no msg dropped
Number of incoming requests (over UDP) dropped because there are no free entries available in the NFS operation table. Its the same as num msg. Requests can be dropped when the filesystem is busy doing back to back CPs.

no msg unqueued
Number of connections unqueued from the deferred list.

no msg discarded
Number of deferred connections, waiting for NFS resources to become available, dropped because of a timeout or shutdown of the socket.

no msg dropped from vol offline Number of requests dropped because the volume was offline.

no deferred msg processed
Number of NFS requests unqueued and processed from NFS over TCP connections that had been put on a queue waiting for NFS resources to become available. This is not the same as number of NFS connections queued up waiting for resources. Note that one connection can contain more than one NFS requests waiting to be processed.

sbfull queued
Number of requests queued because the send buffer is full.

sbfull unqueued
Number of requests unqueued when space available in the send buffer.

sbfull discarded
Number of requests waiting on a full send buffer dropped because of a forced shutdown of the socket.

no mbuf queued
Number of requests (over TCP) waiting for mbufs to send a reply.

no mbuf dropped
Number of requests (over UDP) dropped because of unavailability of enough mbufs to send a reply.

no mbuf unqueued
Number of requests (over TCP) processed from the wait queue when enough mbufs become available.

no mbuf discarded Number of requests (over TCP) waiting for mbufs dropped because nothing can be sent over the socket. This can happen if the socket is closing.

(cumulative) active Two values are printed here. The maximum (cumulative) number of entries available in the NFS operation table and the total number of entries processing requests at that time.

req mbufs
Number of mbufs allocated for an incoming NFS request over UDP.

tcp input flowcontrol receive
Number of times the Filer went into receive flow control.

tcp input flowcontrol xmit
Number of times the Filer went into transmit flow control.

sockets zapped nfs Number of nfs sockets disconnected because the send upcall argument in the underlying tcp socket had been zeroed.

sockets zapped tcp Number of tcp sockets disconnected because both the send and receive upcall arguments in that tcp socket had been zeroed.

no delegation
Number of times no delegation was handed out due to some error.

read delegation
Number of read delegations handed out to NFS version 4 clients.

write delegation
Number of write delegations handed out to NFS version 4 clients.

nfs msgs counts
The tot is the total number of available NFS messages. The unallocated is the number of messages which have yet to be allocated from the system. The free is the number of allocated messages which are not in use. The used is the number of allocated messages which are currently in use. The "VM cb heard" is how many times virtual memory pressure has asked for messages to be released back to the system. The "VM cb done" is how many messages were released back to the system.

nfs reply cache counts
The tot is the total number of available reply cache entries. The unallocated is the number of entries which have yet to be allocated from the system. The free is the number of allocated entries which are not in use. The used is the number of allocated entries which are currently in use. The "VM cb heard" is how many times virtual memory pressure has asked for entries to be released back to the system. The "VM cb done" is how many entries were released back to the system.

v2 mount (requested, granted, denied) The requested is the number of v2 mount requests. The granted is the number of mount requests granted access. The denied is the number denied access.

v2 unmount (requested, granted, denied) The requested is the number of v2 unmount requests. The granted is the number of unmount requests granted. The denied is the number denied.

v2 unmount all (requested, granted, denied) The requested is the number of v2 unmount all requests. The granted is the number of unmount all requests granted. The denied is the number denied.

v3 mount (requested, granted, denied) The requested is the number of v3 mount requests. The granted is the number of mount requests granted access. The denied is the number denied access.

v3 unmount (requested, granted, denied) The requested is the number of v3 unmount requests. The granted is the number of unmount requests granted. The denied is the number denied.

v3 unmount all (requested, granted, denied) The requested is the number of v3 unmount all requests. The granted is the number of unmount all requests granted. The denied is the number denied.

access cache (hits, misses, refresh) The hits are the number of requests which were serviced by an entry already in the access cache. The misses are the number of requests which were not serviced by an entry already in the cache. The refresh are the number of entries which were set to expire, but were refreshed in place. Note that in general the ratio hits to the sum of hits and miss should be high. I.e., we want significantly more hits than misses.

access cache lookup requests (curr, total, max) The curr is the number of current requests which are being serviced. This number is also reprentative of the number of entries on the export worker queue. The total is the sum of all requests. The max is the high water mark of the curr counter.

access cache (loaded, max)
The loaded is the current number of entries in the access cache. The max is the high water mark of the loaded counter.

access cache thread signals (scrub, fill, clear) The scrub is the number of times the mount assist thread has been signaled to flush the access cache. The fill is the number of times the mount assist thread has been signaled to start processing the entries on the export worker queue. The clear is the number of times the mount assist thread has been signaled to throw-away all of the entries on the export worker queue.

access cache flushes during (scrub, fill, flush) The scrub is the number of access cache entries which have been expired due to exceeding the matching timer in either the option nfs.export.neg.time_out or nfs.export.pos.timeout during a scrub of the access cache. The fill is the number of such entries expired during a traversal of the access cache as the export worker queue is serviced. The flush is the number expired during an explicit flush of the access cache.

access cache harvests during (scrub, fill, flush) The scrub is the number of access cache entries which have been expired due to exceeding the timer for the option nfs.export.harvest.timeout during a scrub of the access cache. The fill is the number of such entries expired during a traversal of the access cache as the export worker queue is serviced. The flush is currently not implemented.

access cache name service errors causing not caches (ro/rw, root)
The ro/rw is the number of name service fatal errors which have resulted in the access rights, as determined during the fill of a cache miss, not being stored in the access cache for either a rw or ro access check. The root is the corrosponding number of such errors for the root access check.

access cache name service errors via invalid caches (ro/rw, root)
The ro/rw is the number of name service lookups errors, as determined during the fill of a cache miss, which occur when there are no IP in the access rules. At such times, we silently reject NFS and mountd requests. The root is the corrosponding number of such errors for the root access check.

assist queue (queued, split mbufs, drop for EAGAIN) The queued is the number of NFS requests which are put on the assist queue. The "split mbufs" is the number of NFS requests in which data spans multiple mbufs. The "drop for EAGAIN" is the number of NFS requests which were put on the assist queue for servicing an access cache miss and for which name service errors resulted in the request being dropped.

RPCSEC_GSS context limit
The limit on the RPCSEC_GSS context table. This limit is defined via the nfs.rpcsec.ctx.high

option.
RPCSEC_GSS refers to the authentication protocol used by NFS (see RFC2203) for encapsulating Kerberos V5 authentication information. Therefore an RPCSEC_GSS context corresponds to a Kerberos V5 authentication session between the NFS client and server.

current context count
The number of RPCSEC_GSS contexts currently allocated.

maximum context count
The maximum number of RPCSEC_GSS contexts ever allocated since the last boot (if -l is specified) or the last time the statistics were zeroed via the -z option.

context reclaim callbacks

context idle/expired scans

vm pressure callbacks
Each of these three statistics counts an asynchronous attempt (callback) to reclaim least recently used contexts from the RPCSEC_GSS context table.

The "context reclaim callbacks" statistic counts the number of callbacks triggered by the RPCSEC_GSS subsystem, due to a deferred reclaim (because ONTAP could not suspend).

The "context idle/expired scans" statistic counts the number of periodic scans to look for expired or idle contexts. A context is considered idle if it has not been used for the number of seconds set in the nfs.rpcsec.ctx.idle option.

The "vm pressure callbacks" statistic counts the number of times the ONTAP VM subsystem asked the RPCSEC_GSS subsystem to release memory back to the system free list of memory.

contexts created
Counts the total number of RPCSEC_GSS contexts created.

contexts deleted
Counts the total number of RPCSEC_GSS contexts deleted.

contexts deleted due to vm pressure Counts the RPCSEC_GSS contexts deleted due to vm pressure callbacks.

contexts deleted due to context limit Counts the RPCSEC_GSS contexts deleted due to the current context count already being at the context limit whenever a new context needs to be created. The least recently used context will be deleted.

contexts deleted due to idle/expiration Counts the number of RPCSEC_GSS contexts deleted because the expired or were idle.

requests exceeding timeout
The number of NFS requests which exceeded the timer set in the option nfs.response.trig_ger.

DISPLAYS

The server RPC display includes the following fields, with separate values for TCP and UDP:

calls
The total number of RPC calls received.

badcalls The total number of calls rejected by the RPC layer (the sum of badlen and xdrcall as defined below).

nullrecv The number of times an RPC call was not available when it was thought to be received.

badlen
The number of RPC calls with a length shorter than a minimum-sized RPC call. Also counts the number of RPC requests over the TCP transport with illegal record lengths, such as those that are too small, too large, or not a multiple of the base XDR encoding unit (4 bytes).

xdrcall The number of RPC calls whose header could not be XDR decoded.

The server NFS display shows the number of NFS calls received (calls) and rejected (badcalls), and the counts and percentages for the various calls that were made.

NFSV4 STATS

An NFSV4 compound request can contain more than one operations. Therefore, the nfsstat command displays the total number of rpc requests received by the server (calls) as well as the total operations (ops) received in all the compound requests.

Only a count is displayed for NULL and COMPOUND requests. Percentages are displayed for all operations received in COMPOUND requests relative to the total number of operations. The total calls displayed under the TCP, UDP and Server nfs headings represents the sum of all RPC requests received from NFSV2, NFSV3 and NFSV4 clients and do not take into account any operations in COMPOUND requests.

BUGS

nfsstat -l reports unexpected percentages, if the nfs.per_client_stats.enable option is enabled after the system has been running for a while (typically this option should be enabled at system startup time via the /etc/rc file). Resetting the statistics via the nfsstat -z command will clear this condition.

CLUSTER CONSIDERATIONS

In takeover mode, the nfsstat command displays combined statistics for the live filer and the failed filer. From the statistics, you cannot determine how many requests were serviced by the live filer and how many requests were serviced on behalf of the failed filer.

The -h and -l options display the combined client information for both the live filer and the failed filer.

The NFS statistics are cumulative. That is, a giveback operation does not zero out the NFS statistics. After giving back the failed filer's resources, the live filer does not subtract the statistics about NFS operations it performed on behalf of the failed filer when it was in takeover mode.

VFILER CONSIDERATIONS

When run from a vfiler context, (e.g. via the vfiler run command), nfsstat operates on the concerned vfiler. Note that if vfilers are licensed when using the -h and -l options it is necessary to run this command from a vfiler context, since the interpretations of the hosts are dependent on the vfiler.

SEE ALSO

na_netstat(1), na_options(1), na_partner(1), na_vfiler(1), na_sysstat(1).


Table of Contents