Introducing the Network Appliance Filer

A note of caution: this paper is intended for a technical audience of sysadmins and developers of storage management software, well acquainted with computing practices and computers in general, but unfamiliar with Network Appliance products. If you're new to the technical arena, you may find it heavy going.

Those who've never worked with filers before frequently find their initial experiences a bit mind bending. While the filer has a command line on a console, exports files via NFS like a Unix file server and via CIFS like a Windows 2000 Server, it is neither of those things (note: trade and service marks are the property of their respective owners). This paper attempts to give you a high-level overview of what a filer is and what it does.

Physically, a filer is a head and one or more shelves of disks (the only exception is the FAS2xx line, which are low-end systems which integrate the head(s) into the shelf). The head is connected to the shelf via a FibreChannel link, which usually looks like a thin orange wire. Inside the head is an NVRAM (non-volatile random access memory) card. This is battery backed memory into which all writes are logged. In the event that power fails, uncommitted writes are replayed from the log when the filer comes back up.

Heads look like this, and are made to fit into standard 19" racks. This is an FAS960 head. It contains two Intel CPUs and 11 slots for network and SAN connectivity, and so on.

Shelves also fit into 19" racks. One head can service many shelves. This is a DS-14 shelf. It has room for 14 disk drives, and at today's capacities, can hold about a terabyte. Except for the Nearstore product, which uses ATA drives, all disks are Fibrechannel disks, the traditional choice for high-performance RAID arrays.

Filers can be clustered together for increased availability. Besides having a link between them to keep their write caches (NVRAM) coherent, each has a Fibrechannel link to all the partner's disks. When one partner fails or is intentionally taken down for an OS upgrade or maintenance, the live partner continues to serve data from the down partner's disks.

A fully configured cluster pair can serve 48 TB at present. At left is a picture of a 24TB configuration, with the heads and shelves racked into cabinets, and its proud owner.

Filers can also be Snapmirrored™ to one another. This is an asynchronous mirroring procedure which copies changed blocks from each filer to the other. In the event of a disaster, the surviving mirrors are made live, and life goes on.

Some businesses with really critical data integrity needs, such as banks, use Syncmirror™, a synchronous mirroring product along with Metrocluster™ to guarantee that all transactions are safe. In such a setup, two clusters are set up at least 100km apart, and each contains a synchronous mirror of the other's data.

The Nearstore™ product looks a lot like the main filer product line, but uses cheaper ATA disk drives. It is primarily used for online backup of both filer data, and of data stored in other host filesystems. This allows consolidation of data from various systems into one location, reduces the need for tape to occasional archival backups, and provides a staging area for the tape backups that are taken.

Network Appliance also OEMs the head to various vendors. Hitachi, for example, resells a Netapp filer to serve files from LUNs on the Hitachi storage array.

The FAS250 is pretty neat. Instead of an LRU--the card that the shelf uses to speak to the Fiberchannel loop--it has a miniature filer head. If the unit becomes full, upgrading is a matter of trading up to a higher-end filer head, popping out the baby and inserting an LRU. The FAS250 thus becomes one shelf in a larger system, with no need to copy data and with truly minimal downtime. This top-to-bottom architectural cohesiveness is a great strength of the Netapp product line. The FAS270 is the clustered version of an FAS250, with two filer heads inside a single shelf.

All Netapp products are true network appliances. Clients and workstations are connected to them over a LAN. To these clients and workstations, filers look just like Windows and Unix file servers. This means installing a filer is a matter of configuring it and plugging it in. There are no drivers or whatnot needed on the clients, as the filer speaks standard protocols already understood by the clients.

The filer exports data as files via two primary protocols, NFS and CIFS, corresponding to the Unix and Windows way of doing things. It can also export data as blocks, via FCP or iSCSI, like a traditional SAN-attached disk array. We'll talk about that later.

When you select File/Open from a normal GUI document prompt in an office setting, you browse a directory hierarchy. Some of the directories, however, are not on the local disk of your computer, but are mounted from a remote server. The name of the directory is called a mount point. All the files and directories below the mount point actually exist on the remote server (don't bother thinking about nested mounts right now if you're new to this).

NFS

Unix and Windows actually have slightly different terminology, and very different file sharing protocols. What gets exported on a Unix server is called a filesystem, and is usually specified in the system file /etc/fstab. An exportfs command is often involved on the remote host to get things going. A local host that wants to mount a remote filesystem uses a command like

# mount bighonker:/usr/users /users

to mount the filesystem /usr/users on remote server bighonker over the existing local directory /users. After this is done, the NFS file sharing protocol is used to get files from the remote server as they are accessed by the user who's logged into the local workstation. If there were files in the directory /users before the mount was done, they are no longer visible until the filesystem is unmounted. A pictorial representation of the arrangement looks like this:

In this picture, a Unix workstation has five directories on its local disk: /, /bin, /etc, /home and /applications. In the home directory are two subdirectories named users and users2. Three remote filesystems are mounted over the directories /home/users, /home/users2, and /applications. Each of these directories is called a mount point. Files in the directories are accessed via NFS, and obey Unix file semantics. To the end user, it may not be evident that any of this is going on. System admins often set up the mounts to occur automatically when users log in and when servers start up. If this is the case, the user may have no idea where their data is actually located. Usually the file servers are located in a centralized place to the enterprise, where they can be administered and their data backed up by people who do that kind of thing for a living. While it may seem that access to remote files would be a lot slower, this isn't always the case. Especially over gigabit LANs, getting data from filers is frequently faster than getting it from local disk. This is in part because of RAID striping magic.

Windows

Windows, on the other hand, uses the CIFS file sharing protocol (more on CIFS and NFS later). What gets exported on a Windows server is known as a share, and is usually specified via the GUI using the Sharing... menu item that pops up when you right click on a folder in the explorer. A local host that wants to mount a remote share called users on the server bighonker and call it drive G: usually browses My Network Places to the machine bighonker and then to bighonker's share named users, and assigns it to drive G:, or uses a command like

> net use G: \\bighonker\users

Either method mounts bighonker's share named users on drive G: and all the files and subdirectories in the remote share users appear as though they were local files in the G: drive. This drive is called a network drive. The CIFS protocol is used to get files and directory listings as they're accessed by the user logged into the local workstation. A pictorial representation looks like this:

In this picture, a Windows workstation has one local drive, and four network drives. Files in the drives are accessed via CIFS, and obey Windows file semantics. We'll talk more about how you can access files from both Windows and Unix later. Drive E: is mapped to the user's "homedir" share, which is a special feature of Data ONTAP™, the filer OS. What the user sees in E:\ is the same as that seen in D:\zoe, for the user named zoe. As in the Unix case, the end user may not be aware that any of this is going on. The case for centralizing Windows file storage on Network Appliance servers is very similar to the case in NFS. As in the NFS case, getting data from filers is frequently faster than getting it from local disk. A known exception is compilation of Visual C++ projects, which do many tiny writes to program database files, so are slowed down by network latencies.

The important thing to notice, when comparing these two pictures, is that you see the same files whether the remote filesystem is mounted via NFS or CIFS. There are some inevitable confusions over this, because the two systems have different authentication and privilege models and so on, but all in all, end users seem to get along well with the Netapp solution. A key part of this solution, for admins, is synchronizing the account names between Windows and Unix. More on how this all works later.

The filer has been since its conception a decade ago an appliance designed to be attached to the world via a network. It does most of its work over just two networked fabrics:

• Ethernet, 10/100BT and 1000BT at present (2004)
• Fibre Channel, both out the "back end", which is where the disks and shelves are attached (optionally via a Brocade switch) and out the "front end" via FCP for use in SAN-style configurations.

There are additional connections needed for power, of course, and for connecting two filers together into a cluster, but the simplicity of the network-attached approach has been a major selling point of filers from the beginning.

In spite of this simplicity, network configuration issues are one of the most common sources of phone calls to Netapp CS. Some of the most common gotchas are:

• Duplex settings. You can't get this wrong on gigabit ethernet (it's always full-duplex). On 10/100BT, however, it frequently happens that autonegotiated settings do the wrong thing, and you end up with a full-duplex device on one end and a half-duplex device on the other. This results in truly horrible performance, about five or ten percent of a properly configured filer. Experienced admins usually force both ends to full duplex instead of relying on autonegotiation.
• Improper netmasks. The filer needs to do broadcasts for some operations, so the servers it needs to talk to (DHCP or whatever) need to be within range of the broadcast allowed by the netmask.
• Old clunky machines used as domain controllers and other servers. Because of its high performance, the filer makes significantly greater demands on these servers than you may be used to.
• Poor communication paths between filers and DCs or AD servers. These should be as clean as possible. It is a good idea to have an AD server or DC as 'close' to the filer as possible to assist in speedy authentication. Poor filer/DC communications often cause poor performance and/or authentication problems for users.
• The filer and PDC must be on the same subnet for initial CIFS setup in an NT4 domain.
• Filers which are used as database data store should be connected 'back to back' with the database server. Failure to do so can leave the database's communications with the data store subject to network issues.

One would think it would go without saying that a good clean and fast network setup is necessary to effectively utilize network-attached storage of any kind (not just Netapp filers). Experience shows, however, that one would be wrong, so we'll say it again. Make every effort to have a fast and well-equipped network in place in order to effectively use the incredible performance the filer is capable of and maximize the productivity of your end users.

Iinstalling a filer may in fact bring problems to the surface that were not evident before because of its high performance. The admin-level netdiag command is very useful for flushing out these kind of dormant misconfiguration issues.

We've talked about NFS and CIFS, but those are not the only file service protocols supported by the filer. A more complete list is

• NFS
• CIFS
• HTTP
• FTP
• WebDAV

Each of these is capable of serving up any and all files stored on the filer, subject to configuration and permissions checks. HTTP and FTP, for example, both normally are set up to serve files from a given "home" directory only, but this is a means of restricting what people may get, not a fundamental limitation of the protocol or the filer.

The filer supports many other protocols. A partial list of them and their purposes is

The filer also has two RS/232 serial ports, the first of which is used to connect to the console CLI. The second port is used in-house by Network Appliance personnel for diagnostics purposes.

Many of the protocols listed above are management protocols, and as such expose the system to security risks. The filer can be configured to restrict access in many cases by IP address, in others by or also by user ID, and at a minimum by turning off the protocol. Secure LANS are an essential part of any filer security program

A schematic of a high-end filer setup is as follows. Two filers are clustered for HA purposes. Of course in a real setting there would be more servers and workstations and clients.

In this picture, the primary data paths are in red and blue. Servers which expect raw disk storage over Fibrechannel are served via FCP. Clients and servers which use file-based storage or raw storage via iSCSI are served over ethernet.

This picture also shows the data paths between the filers and their disks. Filers in a cluster communicate via cluster interconnect, also shown (ServerNet, troika and infiniband are the current possibilities, depending on filer model). Each has two fibrechannel loops out the "back end", one for its own disks and one for the disks belonging to the partner. In ordinary operation, each serves data from its own disk. In the event that one of the filer heads fails, the other does a "takeover", and begins serving data from both sets of disks. To do this, it has to take over the ethernet address of the downed filer and pretend to actually be that filer, so far as the outside world is concerned. For this reason, a filer which has been taken over but not powered off stays in a reboot loop, waiting for the signal from the takeover partner over the cluster interconnect to resume normal operation.

It's quite common to use a serial port concentrator (Portmaster and Annex are two well-known brands) to manage the serial port of the filer. Using one of these, you can access the serial port from any client on your network. The serial port is normally no more useful than a telnet session to the filer (they're effectively the same thing), but it is important when the filer breaks, as it may drop down to the firmware prompt. At this point there's no OS, so telnet is gone and the serial port is the only means of managing the filer.

A block diagram of the internal workings of the system looks something like this
             

Clients connect physically through the network and host bus adapters. Remarkably, the filer uses a single data path from WAFL to disk for both block- and file-based storage. For this reason it's called a Unified Storage Appliance, about which more later.

The core technologies that have built the filer's success in the marketplace are based on the novel and patented design of its filesystem. The WAFL filesystem always writes new blocks on disk when a file is written, instead of overwriting already allocated blocks. Out of this one concept, the following advantages flow.

• Taking a snapshot of the system is simple. All that is needed conceptually is a copy of the root inode and a new "blockmap" file describing which blocks are in the active filesystem. Freeing up blocks when they are no longer allocated in any snapshot is relatively simple. Because data is always written to new blocks, a block only transitions to unused once going back in time. So a binary diff of any two consecutive blockmap files yields a bitmap of blocks that can be freed.
• Writing is much faster. Because writes are always to new blocks, WAFL can write according to where the disk heads are, instead of having to seek all over the disk for old blocks to rewrite.
• Reading is faster. Because of WAFL's write allocation policies, blocks in a file can frequently be clustered near each other on a disk, resulting in fewer disk seeks per read. This happens whether or not the application lays out the file linearly or not.

The filer uses battery-backed non-volatile RAM (NVRAM) to store incoming writes. It returns success to the application when the data is safely on NVRAM, so write latencies are much smaller than in normal filesystems. Every 10 seconds, or sooner if things are busy, WAFL does a "consistency point", and initiates a "write episode". The filesystem proceeds to write all the data from the NVRAM to disk, and at the very last moment, a new root inode is written. At that point, the changes are committed and the filesystem is in a consistent state. A snapshot is basically a "promoted" consistency point, so snapshots represent consistent views of the filesystem as well.

It's important to understand that each additional snapshot uses almost no extra space, so long as the filesystem is not changing rapidly between snapshots. It's also important to distinguish between Netapp copy-on-write snapshots and traditional copy-on-write snapshots, which also don't consume extra space until a write happens. At Netapp, we call those "copy-out-on-write" snapshots. Copy-out-on-write snapshots require the filesystem to copy out the block that's about to be written and do bookkeeping about where it went, then write the changed block. When a file is written in WAFL, it just gets written to a new block as always, a much more efficient process, as only one block is written, not two. The WAFL scheme also does not require changes to snapshot filesystems, where copy-out-on-write schemes require updates to the inode blocks in every snapshot the first time a block is changed.

Snaprestore essentially reverses the process by which a snapshot is taken, and turns a snapshot into the live filesystem (all snapshots newer than the one being reverted to are lost). This makes it possible, for instance, to revert a corrupted database in seconds or minutes instead of the hours watching tape reels spin that are usually required.

Snapmirror copies snapshots asynchronously to a remote filer over ethernet, and mirrors changes in that filesystem back to the master as well. In the event of a disaster, the remote filer may be made the master copy of the mirror. The interval of asynchrony is not that long (minutes). Most business units can tolerate a couple minutes of data loss and do not require more protection than this.

For businesses which cannot afford to ever lose a transaction, there is SyncMirror. This is actually a RAID-level mirroring operation over distance. We will dispense with this topic by saying that the intricacies of no-data-loss filer configuration are beyond the scope of this paper. It is worth outlining the storage hierarchy, however:

Filer

Zero or more spare disks (the spares pool)

One or more volumes

Each volume contains one or two plexes. If there are two, they are synchronously mirrored. The topology of the two plexes must be identical.

Each plex contains one or more RAID groups (usually one)

Each RAID group contains at least one disk and a parity disk. If a disk goes bad, the parity disk takes its place and parity is reconstructed onto a new disk taken from the spares pool. While this process is taking place (up to some hours, depending on volume size), the RAID group is in "degraded" mode.

Because of increasing disk sizes--with consequently larger physical volume sizes and ensuing longer reconstruct time--it's not too uncommon that a second disk fails while parity is reconstructing. In traditional RAID 4 and RAID 5 used by other vendors as well as Netapp, this is guaranteed to cause at least some data loss. In Data ONTAP 6.5, NetApp has introduced an innovative new double parity scheme called RAID DP™. This scheme uses a second parity disk and a mathematically optimal algorithm to achieve parity protection that can survive the loss of two disks at once. The algorithm is much faster than Reed-Solomon or other Hamming code based schemes which achieve the same protection level.

Snapvault™ is a technology very much like SnapMirror. It is a one-way transfer, however. Whereas SyncMirror keeps two active filesystems in sync, the snapvault target cannot be written by clients, only read. Snapvault is intended for use in near-line storage on platforms such as Network Appliance's R100, R150 and R200, which use large slow ATA drives instead of the pricier fibrechannel drives in the filer product. RAID and WAFL protection make nearline storage on a Nearstore safer than storage on tape. While tape may be necessary for long-period archival purposes or remote offsite storage, nearline storage can be used as the source for the tape dump, freeing the filer of that load. Additionally, weekly and monthly snapshots may be kept on the R100, resulting in reduced frequency of tape backups. This reduces the need to buy new tape—a major expense at large shops—and may make it possible to use less expensive tape arrays.

Also in Data ONTAP 6.5 is the SnapLock™ product, an option which turns a volume into a WORM volume which is compliant with the Sarbanes-Oxley Act and other regulatory requirements.

In the first section we mentioned that users coming in through various protocols see the same set of files. What we didn't mention was how the filer figures out whether you have rights to perform operations on those files. Currently this is done by deciding that you're either

• a Windows user, or
• a Unix user

This decision is straightforward, and is done via a combination of protocol and authentication method:

After this, things get more complicated. There are four cases:

1. A Unix user requests a file with Unix file permissions (perms) on it.
2. A Windows user requests a file with an Access Control List (ACL).
3. A Unix user requests a file with an ACL.
4. A Windows user requests a file with perms on it.

When a request comes in via Unix, it contains things like FSIDs, a UID, GIDs, inode #'s and so on. A Windows request comes in with a pathname, User SID, Group SID list, file offset and so on. There's no one-to-one mapping between these things, nor between perms and ACLs (ACLs are more expressive than perms). What there is a one-to-one mapping for is a user's identity in the Windows space and their identity in the Unix space. For this reason, the filer does username mapping. Normally, if your account name in Windows is the same as your Unix account, by default your Unix account will be used if you come in via Windows to use a Unix file, and vice versa. The special name mapping file /etc/usermap.cfg allows other mappings. Expanding the table of four possibilities to include what the filer does gives

1. A Unix user requests a file with Unix file permissions (perms) on it
   1. Filer compares uid and gid list and requested operation against file owner, group and perms and grants or denies access
2. A Windows user requests a file with an Access Control List (ACL).
   1. Filer compares sids and requested operation against the Access Control Entries (ACEs) in the ACL until access is granted or denied.
3. A Unix user requests a file with an ACL.
   1. Filer uses the nameservice mechanism (/etc/passwd, NIS, LDAP, as specified in /etc/nsswitch.conf) to get the account name of the user making the request
   2. Filer uses the name mapping feature to map the Unix name to a Windows account name
   3. Filer queries the Domain Controller or applicable Windows authority to get the account SID (Security Identifier) and group SID list.
   4. Because DC lookup is slow, and this has to be done on every NFS request, not just at session setup time as in Windows, the filer caches the result of this query for a few minutes.
   5. Authorization continues as in step 2.
4. A Windows user requests a file with perms on it.
   1. Filer queries the DC for an account name corresponding to the SID in the request. This only needs to be done once per session, so the result isn't cached.
   2. Filer uses the name mapping feature to get a Unix account name
   3. Filer queries the nameservice in use to get the uid and gid for that account
   4. Authorization continues as in step 1.

Very well, you may say, so now I know what happens when I need to get a file. But what happens when I need to set permissions on a file? The filer allows three kinds of "qtrees" (basically virtual volumes), which help you manage this.

• "unix" qtrees
  • The only kind of permissions that can be set in a unix qtree are traditional Unix perms
  • Because you can change qtree "security styles", it's possible that files may have ACLs on them. These are discarded the next time permissions are set on the files.
• "ntfs" qtrees
  • The only kind of permissions that can be set in an ntfs qtree are traditional Windows ACLs.
  • Because you can change qtree "security styles", it's possible that files may have perms on them. Setting permissions on these files writes an ACL that overrides the perms.
• "mixed" qtrees
  • A file's permissions in a mixed qtree are determined by the last system to have set them. Setting unix perms will blow away any ACL already on the file. Setting an ACL will override any perms already on the file.

It's actually harder to explain this than to use it in practice. It turns out that people mostly touch the same files from the same systems. Even when they touch them from both directions, as happens in Engineering environments, people normally set permissions using one system or the other, and don't mix it up much. Several years of experience with this system have proven that for the vast majority of users it just works.

Filers' storage space may be carved up into "vfilers", each of them run by a separate administrator and available on a separate network interface. Vfilers cannot see the storage owned by other vfilers (except for the special vfiler "vfiler zero", which is actually the physical machine). A variety of replication and migration options are available for vfilers. The NOW site and the SAG (systems administrators guide) have more information on the capabilities of vfilers.

Unified storage

In the first couple sections we showed how the filer serves data either as blocks in a SAN setting, or as files in a NAS setting. The wowie-zowie thing about this not so much that it happens (other vendors can kinda sorta do this too). It's that when the filer exports a LUN, it is for all practical intents and purposes just another file in the filesystem. This means that LUNs can take advantage of snapshots, snapmirror, snaprestore (there is a special snaprestore called single-file snaprestore that's real handy for LUNs), and so on. For example, multiple clients may boot off a base filesystem image by each mounting a separate snapshot of the base filesystem LUN, and only the config files used by each client will vary from snapshot to snapshot.

SRM and the Filer

The challenge for SRM vendors is to present a unified picture of this unified storage platform. Network Appliance is committed to ongoing dialog with 3rd party management partners to advance this goal.

A few things should be mentioned to SRM developers who may be unfamiliar with the implications of the Netapp filesystem design.

• Deleting files doesn't immediately free the underlying storage unless they're deleted before a snapshot is taken. Once the blocks are in a snapshot, the storage stays used until the snapshot ages out (may be weeks) or is deleted manually.
  
• The filer exists on multiple levels of a traditional architectural diagram. On the one hand, it serves up "raw disk" access via LUNs. On the other hand, it serves up files. This is part of the problem pointed to just above.
  
• Netapp occasionally gets requests for doing file walking within the filesystem for performance reasons. While this makes sense on the surface, it isn't clear that the CPU should be used for that, in the larger scheme of things. It's unlikely also that we'd be able to make such a facility general enough. Good threading practices on the client side generally solve the perceived performance problem. Please see the fastfilewalking code provided in the SDK.

This simplified UML diagram shows the principal filer components as they relate to each other. In versions of ONTAP prior to 7.0, the "aggregate" and "volume/filesystem" elements were effectively merged.

Data ONTAP 7.0 introduces the idea of flexible volumes and aggregates. These replace "volumes" in 6.5 and earlier. Aggregates are the physical containers formerly represented by volumes, and flexible volumes are the logical component, the filesystem level entities if you will. Flexible volumes may be grown and shrunk, and enable "thin provisioning", which allows customers to "overprovision" storage space on a given aggregate. Using thin provisioning, for example, you could allocate 12 2-TB flexvols on one 2-TB aggregate, only adding space to the aggregate when actual space usage in the flexvols approaches the size of the aggregate.

FlexClones allow the creation of writeable snapshots of flexvols. These can be exported as either filesystems or LUNs

References

Software Setup Guide
Storage Management Guide
File Access and Protocols Management Guide
Data Protection Online Backup and Recovery Guide
MultiStore® Management Guide
Network Management Guide
Block Access Management Guide for iSCSI and FCP
Commands: Manual Page Reference, Volume 1
Commands: Manual Page Reference, Volume 2
Core Commands Quick Reference

Most Windows and Unix applications don't need to do anything special to talk to filers, which do everything in their power to convince the application the files they're seeing are local. Developers writing server management applications, on the other hand, may want to license the Manage ONTAP SDK. Please inquire at dl-manage-ontap-api@netapp.com.