About: The significance of network latency in report scanning speeds

Summary

Northern Storage Suite reporting (including billing reports) is directly affected by network latency. The effect is shown in the speed with which file-level metrics can be collected from target storage devices and the speed with which NSS stores processed information into its database.

The network latency that exists between NSS and the target storage is far more significant than the latency between NSS and its database. This is due to the fact that NSS collects raw data but stores processed data; the amount of traffic between NSS and the target storage is far greater than the amount being transferred between NSS and its database.

This article will focus on the effect of latency in report scanning operations.

Calculating the Effect of Latency

Northern Storage Suite host scanning, to populate the interface of the application, is the most commonly run scan. This scan is also usually the widest in scope. As such it will be used as the primary example when describing how to calculate the effect of latency. In order to make that calculation it is necessary to understand the latency experienced in the system, how much data is transferred and the speed at which data requests are processed by NSS itself.

Calculating Network Latency

Calculating the latency in the connection between the NSS Managing Host and the target file shares is a simple method of pinging the target storage device from the NSS host, or host in the same pool, and collecting average times over several pings.

Calculating The Amount of Data Transferred

Host scanning, as with all NSS reporting, is achieved through the CIFS API; file-level meta data is collected with CIFS operations. During a host scan a single CIFS operation at the folder level is needed to collect file type, size and age information for all files in that folder ('SMB_COM_QUERY_INFORMATION'). Four more CIFS operations per file are then needed to collect file ownership information ('SMB_COM_OPEN', 'SMB_COM_READ', 'SMB_COM_SEEK', 'SMB_COM_CLOSE').

In the average environment you should expect to see approximately 10 files per folder. This will vary from folder to folder but should, in Northern's experience, be accurate across any medium/large file system.

With 1 CIFS operation per folder, 4 CIFS operations per file, and ten files per folder it is possible to determine that 4.1 CIFS operations, and therefore 4.1 CIFS Packets per file must be transferred between the target storage and the NSS managing host.

The Speed of NSS Data Requests

The speed of data transfer for NSS is affected by several factors but if the guidelines for minimum system requirements are followed, and the network itself is in reasonable repair, then any one NSS managing host should send and receive 1 CIFS packet every n milliseconds (where n is the network latency) per Network Interface Card (NIC).

Formula for Calculating the Effect of Network Latency

The formula for calculating the theoretical scanning speed of an NSS host scan through a single NIC, in files per second, is as follows:

Number of CIFS packets that can be sent per second = (1000/n)
Number of CIFS packets required per file in host scan = 4.1
Theoretical number of files scanned per second = (1000/n)/4.1

Using 0.2 milliseconds as an example = (1000/0.2)/4.1 = 1220 files per second, per NIC

If longer latencies are considered, in the region of >1ms, then the numbers of files scanned per second is low. In such situations it is important to have the correct hardware or architectural configuration in order to maintain strong solution performance.

NIC Teaming to Overcome Long Latency

Northern recommends that NIC teaming be used in environments that experience long latency between the NSS managing host and the target storage devices.

Link aggregation or NIC teaming is a strategy of combining (aggregating) multiple network connections in parallel to increase throughput beyond that which a single connection could sustain. NIC teaming can be applied in both physical and virtual environments but care must be taken in virtual environments to ensure that underlying physical hardware is available to support the virtual configuration.

Considerations for Distributed Environments

Due to the significant difference in the amount of data collected through report scanning and the amount of data stored into the NSS database, it is always more advantageous to locate NSS application servers closer to the target storage than to their centralized database. With this configuration the effect of latency is felt only in the database activities, where far less information is being transferred.

KB Article: 2857

Updated: 6/27/2016

  • Category
    • Concept
  • Affected versions
    • NSS 9.0
    • NSS 9.5

North America HQ

NORTHERN Parklife, Inc.
301Edgewater Place, Suite 100
Wakefield, MA 01880
USA

Voice: 781.968.5424
Fax: 781.968.5301

salesUS@northern.net

 

Additional Contact Information

EMEA & APAC HQ

NORTHERN Parklife AB
St. Göransgatan 66
112 33 Stockholm
Sweden

Voice: +46 8 457 50 00

salesHQ@northern.net

Northern Parklife



©2018 northern parklife

privacy statement 
terms of use