About: Handling of Empty CIFS Notifications in NSS

CIFS Notifications

Northern Storage Suite uses CIFS notifications to track changes on the file system. This asynchronous, real-time tracking method is what is used to monitor quota usage levels, primarily on EMC Unified Storage, NetApp FAS systems and other IP storage systems.

When a quota is set on a path Northern Storage Suite will subscribe with the target storage device to receive notifications of changes made within that path. These CIFS notifications include certain details about the change; the path where the change occurred, the type of operation that was performed, etc. NSS uses the path information included in the notification to trigger a scan of the individual directory where the change occurred and then update the quota usage level to reflect the change.

Notification Buffer

CIFS works through a client-server approach, where a client makes specific requests and the server responds accordingly. NSS is the CIFS client requesting information from the target storage which is then the CIFS server.

BUFFER SIZE

CIFS notifications are always buffered on the CIFS server [target storage] and delivered to the CIFS client [NSS] in batches. The buffer has a fixed size on all platforms, although that size varies between platforms.

BUFFER DELIVERY INTERVAL

The buffer is normally delivered and emptied as soon as the CIFS client requests to receive the collected batch of notifications. NetApp systems differ slightly in that the buffer is delivered only at a fixed interval. The default interval for NetApp systems is 0.5 seconds but can be controlled by a parameter configured within the operating system of the NetApp device.

Empty Notifications

In certain situations it is possible that the message does not include any information; an empty CIFS notification. Empty notifications are allowed by the CIFS protocol specification. They are used to highlight specific circumstances and elicit a specific response.

CIFS, and the underlying SMB and SMB2 protocols, are owned and administered by Microsoft. Microsoft describes empty CIFS notification as follows:

If the number of bytes transferred is zero, the buffer was either too large for the system to allocate or too small to provide detailed information on all the changes that occurred in the directory or subtree. In this case, you should compute the changes by enumerating the directory or subtree.

See msdn "ReadDirectoryChangesW Function" article for more details.

According to this specification a 0 byte or empty notification is sent to the CIFS client when the server-side buffer is exceeded. [This is preferable to not sending any kind of notification as the information that something has changed is still communicated to the CIFS client, even though there is no detail as the type of change or change location.]

The specification also defines how the CIFS client should respond to an empty notification; by re-scanning the part of the file-system that was being monitored.

Effect of Empty Notifications on NSS

For Northern Storage Suite the most important element of a CIFS notification is the path data; the directory where the change occurred. NSS uses this value to understand where, at the file level, a re-scan is needed. An empty CIFS notification does not include this data. When NSS receives a notification without path data it knows that a file system change has occurred but it does not know where. Two courses of action are now open:

  • Ignore the notification, in which case the displayed usage level of one or more quotas will be incorrect
  • Rescan all quotas to re-calculate usage levels

Ignoring the notification and allowing quota usage to diverge from the actual state may be a viable option in some specific circumstances but it is expected that these circumstances are rare.

Provided that these empty notifications occur relatively infrequently, in the range of one per day, rescanning the quota space is a viable solution that causes little or no deterioration in service level.  However, if the rate of empty notifications is in the range of one per hour then the likely outcome is that all NSS scanning threads eventually become dedicated to re-scanning. The result of this for end-users is a slow response in quota updating, locking and unlocking. This lag can be anywhere from minutes to, in large and busy environments, hours.

Rate of Empty Notifications

Empty CIFS notifications are caused when the CIFS server buffer is exceeded. There are three parameters that influence the frequency of this state

  • The rate of activity in the environment
  • The buffer size
  • The buffer delivery interval

The rate of activity is a factor but it cannot be effectively controlled by settings within either the CIFS server or the CIFS client. The remaining two factors, and how they can be controlled, are described in more detail below.

BUFFER SIZE

According to SMB specifications the largest possible size for the CIFS notification buffer is 64KB. The default size of this buffer in EMC and Microsoft CIFS Server implementations is 64KB. The default size of this buffer in NetApp's CIFS server implementation is a much smaller 4KB. A buffer of 4KB will obviously accommodate sixteen times fewer notifications and is, as such, at far greater risk of becoming full.

BUFFER DELIVERY INTERVAL

If more notifications occur between two requests/deliveries than can fit into the buffer then an empty notification will occur; either the CIFS client is waiting too long before returning for a new batch of notifications or the CIFS server is waiting too long before it delivers them.

Northern Storage Suite will query the CIFS server every 1-5 milliseconds. On those platforms where the delivery of the buffer is made immediately upon request this means that the delay between requests is only 1-5ms. On NetApp systems the CIFS server delivers notifications at a default interval of 0.5 seconds [500ms], regardless of how often the buffer requests are made. A very large number of notifications can be generated over such an extended period of time.

Reducing the Rate of Empty Notifications

WINDOWS PLATFORMS

Northern has very few reported cases of empty CIFS notifications being generated on Microsoft platforms. The rate of empty notifications has always been well within the design scope of NSS and service levels have not been affected.

EMC UNIFIED STORAGE/CELERRA

Northern has very few reported cases of empty CIFS notifications being generated on EMC platforms. The rate of empty notifications has always been well within the design scope of NSS and service levels have not been effected.

NETAPP FAS

It is clear that differences in the CIFS implementation on NetApp systems can make those systems significantly more likely to deliver empty notifications to CIFS clients. The default settings for buffer size and delivery interval are dramatically different from those of Microsoft and EMC for example. 

The default values for buffer size and buffer delay have always been parameterized but were hidden from users until the release of Data ONTAP 7.3.3 P3. This patch is described in the NetApp Bug Report 380382. The parameters to control buffer size and buffer delay are 'cifs.neg_buf_size', 'cifs.changenotify.buffersize' and 'smb_boxcar_expire_ms'. The 'smb_boxcar_expire_ms'-parameter can only be seen and changed in privileged mode.

The ranges for these values are as follows:

  • 'cifs.neg_buf_size': 4356 to 65535 (default = 33028)
  • 'cifs.changenotify.buffersize': 4 to 64 (default = 4)
  • 'smb_boxcar_expire_ms': unlimited (default = 500)

Northern's experience shows that the following procedure greatly mitigates the rate of empty CIFS notifications being generated in NetApp environments:

  1. Set 'cifs.neg_buf_size' to 65535
  2. Set 'cifs.changenotify.buffersize' to 64
  3. Reduce 'smb_boxcar_expire_ms' to 200ms and monitor for acceptable rate of empty CIFS notifications, further reduce to 100ms, 50ms, etc. as needed.

NOTE: The above recommendations have been established through dialogue with NetApp's technical support staff and extensive experience in production FAS environments. However, if you have any questions about these settings, please contact NetApp support for clarification.

CHANGES IN NORTHERN STORAGE SUITE

Northern Storage Suite is designed to accomodate empty CIFS notifications. Due to the unique design of the CIFS implementation on NetApp FAS, empty notifications are generated at dramatically higher rates than any other platform. In an effort to reduce the impact of this design choice Northern has made certain changes within Northern Storage Suite.

NSS 8.27, in version 4008.27.16750.1209 or higher empty notifications trigger a full re-scan of target shares. In older versions of this branch empty notifications are ignored.

NSS 8.5, empty notifications triggered a full re-scan, as described in the CIFS specification. This will lead to slow quota update in some cases.

NSS 8.5 SR3, the quota space is re-scanned when empty notifcations are received but these re-scans cannot be triggered with greater frequency than once every 30 minutes. As an additional optimization locked quotas are rescanned after every empty notification. These quotas are prioritized in case the empty notifications are being caused by users deleting large number of files in an effort to get back within quota thresholds.

NSS 8.5 SR4, an additional change has been made to mitigate the effect of high rates of empty CIFS notifications. This change affects the NetApp platform only, where high rates of empty notifications are far more common. The effect of the change is to increase the amount of empty CIFS notifications NSS can tolerate before quota update speeds become affected.

In this change, NSS continuously monitors the file system activity using the NetApp FPolicy api to keep a list of all directories that have been "touched" in the preceding 5 seconds. When an empty CIFS notification is received NSS will rescan all the directories which are in this list; effectively re-introducing the path data to the empty notification. This greatly reduces the amount of rescanning.

Note that it is possible for the empty CIFS notification to relate to a directory that is not in the "touched" list but this would require a rare burst of notifications across multiple unrelated directories. The bursts of activity that give rise to empty notifications are, in Northern's experience, always centered on a single directory.

Recommendations

For NetApp customers experiencing degradation of quota service levels due to a high rate of empty CIFS notifications Northern recommends upgrading to the latest release of Data ONTAP and consulting with NetApp engineers on how to configure the buffer size and buffer delay parameters. Northern also recommends upgrading to the latest version of Northern Storage Suite. This can be downloaded from the secure community site.

For customers using Northern Storage Suite to manage other platforms it is not expected that any action should be required. Occasional empty CIFS notifications are part of the CIFS specification and are expected. Northern Storage Suite is designed to accommodate them.

ADDITIONAL RESOURCES

  • KB1729 What is/are: The Interfaces Used in Quota Operations
  • KB Article: 1785

    Updated: 8/24/2017

    • Category
      • Concept
    • Affected versions
      • NSS 9.0
      • NSS 9.5
      • NSS 9.6
      • NSS 9.7
      • NSS 9.8

    North America HQ

    NORTHERN Parklife, Inc.
    301Edgewater Place, Suite 100
    Wakefield, MA 01880
    USA

    Voice: 781.968.5424
    Fax: 781.968.5301

    salesUS@northern.net

     

    Additional Contact Information

    EMEA & APAC HQ

    NORTHERN Parklife AB
    St. Göransgatan 66
    112 33 Stockholm
    Sweden

    Voice: +46 8 457 50 00

    salesHQ@northern.net

    Northern Parklife



    ©2018 northern parklife

    privacy statement 
    terms of use