SAS I/O performance: unprecedented flexibility in I/O bandwidth selection.
Configurations & Metrics
Many current systems employ Ultra 320 parallel SCSI for boot drives, local data storage, or external arrays of modest size. Systems emerging over the next two years will employ SAS disks for boot drives, local data storage, and external arrays of modest to large size. SATA makes use of the same SAS infrastructure and adds vast storage capacity at relatively low cost.
Two primary metrics of storage subsystem performance are data throughput and I/O rate. Data throughput is typically stated in terms of MB/s, and measures the maximum sustained data rate. Usually maximum data rates are seen with sequential data streams that are either entirely Read or entirely Write operations, and have data block sizes of 64 KBytes or larger. I/O rate is the maximum number of I/Os the system can complete per second. Usually maximum I/O rates are seen with sequential data streams, which are either Read or Write operations, and have data block sizes of a single disk sector, or 512 Bytes.
Much I/O from user applications is random I/O. That is, requests for data may jump around to different locations on the disk. Random I/O will never occur at throughput levels of sequential I/O since a random I/O involves moving the disk head (seek time), waiting on the disk platter to spin to the right location (rotational latency) in addition to the time required to move the data. Data caches on disk drives, in RAID controllers, and in the file system can mitigate the effects of random addressing, but cannot completely eliminate them. Spreading a data volume across multiple disks, and using only a portion of the disk for storage is a common technique for increasing random I/O throughput. SAS and SATA, with their increased connectivity permit use of volumes that are larger than with parallel SCSI, and potentially have higher random throughput performance. The I/O solution with the highest sequential data throughput has the best chance of supporting the highest overall I/O throughput when random and sequential I/O are both considered. For performance planning purposes, it's best to make sure the storage subsystem has sufficient sequential bandwidth and connectivity before considering methods to improve random throughput.
SAS and SATA solutions have been in development laboratories long enough now so that we have a good under standing of the actual capability of the interfaces and disk drives in a variety of storage configurations. Throughput values used in this paper are measured results from best-in-class solutions. These results can be used as a basis for planning storage configurations to meet specific application requirements.
Performance Variation: Parallel vs. Serial disks
Connectivity and performance are tightly linked with parallel Ultra 320 solutions. Controllers typically have one or two SCSI channels, each supporting a maximum of 270 MB/s. Each channel can connect from 1 to 15 devices, sharing the 270 MB/s bandwidth. Performance variation in an Ultra 320 SCSI implementation is limited, and can come from:
* Number of disks (1 to 15 per channel)
* Mode of operation (Ultra 320, Ultral 160, Ultra 2, etc.)
* PCI-X bus width and frequency (32 & 64 bits; 64, 100 & 133 MHz)
With the serial point-to-point implementation of SAS and SATA, connectivity and performance are separated and can be individually optimized for the needs of each system.
Connectivity variation can be managed through topologies of controllers, expanders, and disks, up to the limit of number of addresses supported by the active components.
Highly available parallel SCSI, SAS and SATA storage subsystems are built using various RAID controllers and protection levels, as well as redundant paths and failover of controllers. SAS enhances that by supporting redundant paths to the disk drive. SAS disks are typically tested to higher MTBF levels and lower bit error rates than SATA disks.
The following Figure shows the maximum achievable throughput of Ultra 320, SAS and SATA technologies. Each vertical bar indicates the highest large-IO READ throughput that can be supported by current controllers, host computer systems, and disks. Separate bars are given for 1 and 2 channel Ultra 320, since controllers exist in either configuration. Separate bars are given for SAS and SATA with narrow (x1) links and wide links of widths 2, 4, and 8 (x2, x4, x8).
[FIGURE 1 OMITTED]
The horizontal lines in the chart show bandwidth limits of various host side links used in these controllers. The Ultra 320 solutions employ native PCI-X, which is available in widths of 32 & 64 bits and frequencies of 66, 100, and 133 MHz. Some SAS and SATA solutions are available with the same PCI-X host link. SAS and SATA are also available with native PCI Express with link widths of x1, x2, x4, and x8.
In most cases, WRITE throughput is very close to the READ throughput limit for sequential I/Os. Of course, I/Os going to RAID volumes may be much slower for WRITE if parity or other redundant data techniques are used. In this article we focus on READ throughput since it typically sets a higher throughput limit.
In the Figure, we can see that the maximum throughput of an Ultra 320 controller is a third of the maximum for a SAS controller. Also, the new serial interfaces offer the opportunity to tune data throughput capability by selecting the width of the disk-side link. In using the chart, data throughput can be as high as the lowest of: the vertical bar for the disk-side link, or the horizontal line for the host-side link. The lower bottleneck will set the highest throughput.
Several examples illustrate how this chart can be used to plan a storage configuration to meet a data throughput goal.
A SATA x1 link is limited to 115 MB/s regardless of the host-side bus. One can connect as many disks as desired through expanders or port multipliers to achieve connectivity or availability through a RAID volume. But the maximum throughput will not exceed 115 MB/s with an x1 narrow link.
Disks connected with a SAS x2 link can be limited to 195, 390, 480, or 540 MB/s if connected to a PCI Express x1, PCI Express x2, PCI-X 66, or anything faster or wider than these three. Again, expanders can be used to increase the drive count, but throughput will be limited by either the host side link or 540 MB/s, the limit of an x2 SAS wide link.
Disks connected with a single Ultra 320 channel are limited to 270 MB/s aggregate throughput, regardless of the host-side link.
For Ultra 320 SCSI, bandwidth planning was much simpler, but also with much lower capabilities. The choices were basically 1 or 2 SCSI busses, with a maximum of 15 devices per SCSI bus. The host side connection was PCI or PCI-X at 133 MHz or slower. Some PCI Express Ultra 320 adapters are available, but they all use controllers with a native PCI-X connection. So the speeds of PCI Express x8 are not available with a single Ultra 320 controller.
Disk Performance Data
Currently enterprise class disks are offered in two form factors. The form factor has a significant impact on maximum sustained data throughput. For Ultra 320 and SAS, a 3.5" disk will typically support 90 MB/s sustained throughput. There is some variation from between manufacturers and depending upon what part of the disk surface is tested, but this is a typical value useful for planning. A 2.5" Ultra 320 or SAS disk will support 60 MB/s sustained throughput. When we look at SATA disks, the values are lower. A 3.5" SATA disk will support around 60 MB/s sustained throughput, while 2.5" disks support up to 45 MB/s. These throughput numbers assume a sequential data stream, and are not affected much by spindle speed. For random I/Os, the supported data rate is entirely different, and driven by rotational speed, stroke of the disk (amount of the disk capacity that is configured for the volume), I/O size, and number of queued I/Os. However, the sustained sequential throughput sets the upper bound of what can be expected from each disk.
Understanding where these interfaces are going to be in the future could have a large influence in selecting a solution for today. Ultra 320 is the end of the line for parallel SCSI. There are no plans for increasing the data throughput or connectivity of Ultra 320. This interface has grown tremendously over the past 20 years, and has been very successful.
The new serial SCSI interfaces are at the beginning of their development, and will grow in terms of connectivity and performance over the next several years. Standards are being worked for 6 Gb/s SAS, and there are plans for extending it to 12 Gb/s. Some SATA solutions are emerging that implement SATA 2 with a 3 Gb/s serial link rate. SATA configurations supporting over 1.0 GB/s are expected soon. Extending the capability of SAS and SATA with management utilities, expanders, port multipliers, and faster and higher capacity disks is also occurring. SAS and SATA will continue to develop with increases in performance, connectivity, and management capability.
Charles E. Gimarc, Ph.D. is a Performance Engineer for the Storage Components Group, LSI Logic (Milpitas, CA).
|Printer friendly Cite/link Email Feedback|
|Author:||Gimarc, Charles E.|
|Publication:||Computer Technology Review|
|Date:||Oct 1, 2005|
|Previous Article:||Data encryption strategies; Part 2: encrypting high-performance, high-volume storage.|
|Next Article:||Backup and disaster recovery techniques: a look back and forward.|