NetApp Performance Measurement using Open Interfaces

1 Purpose and Scope

2 Introduction

2.1 Overview

2.2 Basics of Performance Analysis

2.3 Performance Monitoring Infrastructure

2.4 Performance Counters Design

3 NetApp Open Interfaces

3.1 Manage ONTAP APIs

3.2 SNMP

4 Performance Measurement Using NetApp Open Interfaces

4.1 Using Aggregate Counters

4.2 Using LUN Counters

4.3 Using NFSV3/CIFS Counters

4.4 Using Processor/System Counters

5 Appendix 1 - Performance Counters in Data ONTAP 7.1

5.1 Counter Types

5.2 Counter Priority Levels

5.3 List of Objects Supported

5.4 List of Counters Supported in various Objects

This document provides detailed guidelines on how to do Performance Measurement for NetApp devices using below mentioned Open Interfaces:

• ManageONTAP API
• SNMP

2.1 Overview

Performance measurement is important in understanding the usage patterns of storage which can help in catching the performance problems before they start impacting the users and applications.

Performance Measurement is useful in the following areas:

2.2 Basics of Performance Analysis

One of the fundamental theories used for Performance Analysis is Little’s Law. Following is the mathematical representation of Little’s Law:

Queue-Length = Throughput x Latency

Throughput – The rate at which any request coming into the system is served
Latency – The time taken to service a request coming into the system
Queue-Length – Average number of pending requests in the system at any given time

Little’s Law can be used to analyze the performance of any system that can be modeled as queues. For example, to analyze how efficiently a Bank is servicing its customers, we need to calculate at what rate the customers are entering the bank for a service (Throughput measurement) and how much time the teller at the counter takes to service each customer (Latency measurement). Once we get Throughput and Latency data, we can calculate the average length of the customer queue which is a direct measure of how efficiently the bank is servicing its customers and this data can be used to take a decision whether the bank needs to open one more counter or the employee at the counter needs to be made more efficient.

Little’s Law can be used to understand the performance of any storage component in the customer environment. For each component in the storage system, the queue-length data can be calculated using the I/O rate to the component (gives throughput measurement) and the I/O latency of the component. For each environment, the queue-length numbers need to be monitored and the average queue-length should be bench-marked. The performance hot-spots can be identified in the specific storage environment by tracking the consistent periodic peaks in the queue-length numbers observed over a period of time.

2.3 Performance Monitoring Infrastructure

The following figure shows the overall infrastructure that is available for the applications to monitor the performance statistics of the NetApp storage systems.

At the base of the performance infrastructure there is a set of performance counters which are accessible through existing performance commands. Above the set of these performance counters, there is a thin layer of abstraction called Counter Manager (CM).

Counter Manager provides

The following are the different channels through which performance monitoring applications can access the API set of the CM:

One of the things that are going to be added in the future is something called Filer Data Archives. Today, when you get to the performance counters you get a real-time view of these counters. If you want to save them, you can save them but that responsibility is up to the host or the administrator. With Filer Data Archives, the storage system keeps historical versions of performance counters so that it's easy to do trending and analysis without the host having to save all that information.

2.4 Performance Counters Design

Performance counters are grouped in an object-instance-counter hierarchy. Each object could have many instances of it but the object will have a set of well defined counters. So each instance of the object will have its own corresponding values for this well defined counter set. For example, 'Volume' is an Object type. In a particular storage system, there could be many two volumes 'vol0' and "vol1". Each of these volumes will be an instance of the "Volume' object. The "Volume" object has well defined counter set like "read_ops', "read_latency" etc. So the instances of the 'Volume" object in this storage system viz. "vol0' and "vol1' will have their own values for the counters "read_ops", "read_latency" etc. The example is illustrated below:

Each counter has a type, unit and priority associated with it. See ‘Appendix 1 - Performance counters in Data ONTAP 7.1’ for the description of the various objects supported in Data ONTAP 7.1, the description of the different types and priorities of the counters.

3.1 Manage ONTAP APIs

The ONTAPI interface is a set of foundational APIs for managing NetApp Storage Systems. This is a proprietary interface developed by NetApp for advance management of NetApp Storage Systems. ONTAPI includes APIs for qtrees, quotas, volumes, luns, snapshots, snapmirror, vfilers, and numerous other features.

ONTAPI interfaces use XML as a format and may be configured to use either HTTP or DCE/RPC as the transport mechanism. HTTP is important when managing devices which may be outside of the corporate firewall, and locked down so that only port 80 is available, and is the default transport configuration. Security for HTTP can be provided by using SSL.

ONTAPI is implemented as a small set of primitives that marshal and un-marshal arguments onto and from the wire, using XML as the description language. At present, bindings for C/C++, Perl and Java are provided. Development of bindings for other languages will be driven primarily by developer demand.

3.2 SNMP

SNMP (Simple Network Management Protocol) is a well-known standard for network management. NetApp storage systems support SNMP V1. NetApp only supports monitoring using SNMP which means that SNMP SET operations are not permitted.

If SNMP is enabled in Data ONTAP, SNMP managers can query your storage system's SNMP agent for information (specified in your storage system's MIBs or the MIB-II specification). In response, the SNMP agent gathers information and forwards it to the SNMP managers using the SNMP protocol. The SNMP agent also generates trap notifications whenever specific events occur and sends these traps to the SNMP managers. The SNMP managers can then carry out actions based on information received in the trap notifications.

The latest versions of the Data ONTAP MIBs files are available online on the NetApp on the Web (NOW) site at http://now.netapp.com/NOW/download/tools/mib/filer.shtml

Among the open interfaces, currently only ManageONTAP APIs provide the interface for getting the performance related data. The following sections describe how the performance counters of some of the performance objects can be used in various scenarios that would need performance measurements.

4.1 Using Aggregate counters

Consider the scenario wherein the application needs to know if more disks need to be added to the aggregate to improve the application performance. For this the application needs to do the following:

The following table provides details on API calls to get the above data:

Assuming that the data in the aggregate are striped across all the disks equally and that the benchmark numbers for the FC disks are:

Throughput: 30 – 50 MB/sec, and IOPS: 200 – 400

Also assume that the application is using 64K size blocks for read/write operations.

Total_transfers = (total_transfers at t2 - total_transfers at t1)/ (t2 – t1)
Total bytes of read/write data = ((user_read_blocks + user_write_blocks at t2) – (user_read_blocks + user_write_blocks at t1))*64*1024/ (t2 – t1)

IOPS per disk = (total_transfers)/(disk_count)
Throughput per disk = (Total bytes of read/write data) / (disk_count)

If the IOPS or the throughput per disk are not in the range as mentioned above then additional disks should be added to the aggregate to improve the performance of application that is doing IOs to that particular aggregate.

4.2 Using LUN counters

Consider the scenario wherein an application is sending I/O packets to a LUN and the response time of the I/O is poor. This could be because of one or more of the following reasons:

If the application concurrency is taken care with good multi-threaded design then the bottle-neck is most probably at the LUN.

The following table provides details on API calls to get the above data:

Queuefull/sec = (queuefull rate at time t2 - queuefull rate at t1) / (t2 – t1)
avg_latency = (avg_latency at time t2 - avg_latency at time t1)

From the above calculations, the application can determine if the queuefull rate and average latency are in the acceptable range.

If the queue full rate is not in the acceptable range then the LUN is really busy, so add more LUNs and distribute the I/O on different LUNs.

If the average latency is not in the acceptable range, then the LUN is slow so move the LUN to the volume which is based on a faster disk. If the latency at the server is much more compared to the average latency of the LUN then increase the queue-depth settings of the SCSI/FC driver on the host.

4.3 Using NFSV3/CIFS counters

Consider the scenario of an application reading data from NFS/CIFS share is facing timeouts or disconnects. The problem could be due to network latency or the storage system latency. Following are the steps to debug this issue:

Calculation of the Latencies for NFS and CIFS shares:

NFS read latency = (nfsv3_read_latency at time t2 - nfsv3_read_latency at time t1) / (nfsv3_avg_read_latency_base at time t2 - nfsv3_avg_read_latency_base at time t1)

CIFS latency = (cifs_latency at time t2 - cifs_latency at time t1) / (cifs_latency_base at time t2 - cifs_latency_base at time t1)

The maximum acceptable Latencies :

If the latencies calculated above are higher than the maximum acceptable latencies, then the problem lies with the storage system.

If storage system doesn’t show high latencies, probably it is a network issue. Compare host view of latencies versus storage system view of latencies. If different, probably it is network or host I/O driver issue.

4.4 Using Processor/System counters

One of the unique features of NetApp storage systems is, unlike other vendor storage systems, NetApp storage systems have an OS and File System. So CPU of the storage box could also be one of the potential candidates for bottlenecks in the I/O path.

Consider the scenario where there is high latency and application sluggishness. Assume that there are no disk bottlenecks or application concurrency problems. Then the problem could be with the utilization of the CPUs on the storage system. Following are the steps to diagnose the CPU bottleneck problems:

The CPU and domain utilizations can be calculated as described below. The following table shows how to use APIs to get the required counters to calculate these values:

% CPU utilization = (total_processor_busy at time t2 - total_processor_busy at time t1) / (cpu_elapsed_time at time t2 - cpu_elapsed_time at time t1)

Domain utilization:

For each processor instance, the domain utilization is derived as below:

% domain utilization = (domain[cpu-number][domain element] at time t2) – (domain[cpu-number][domain element] at time t1) / (processor_elapsed_time at time t2 - processor_elapsed_time at time t1)

The possible values obtained from the above equation are:

The total domain utilization is calculated by adding the corresponding elements in the above arrays.

Once the % CPU utilization and % domain utilization is calculated as above, the solution for the problem could be decided as below:

5.1 Counter Types

5.2 Counter Priority Levels

5.3 List of Objects supported

5.4 List of Counters supported in various Objects