The Nearline Effect: tape vendors will love it; disk vendors won't.

Nearline defines the level of storage between online disk storage and offline, manually mounted data storage devices. This level of the storage hierarchy has traditionally been the domain of automated tape libraries, though other architectures are now beginning to enter the scene. The implementation of a nearline storage solution most often has a significant impact on changing the ratio of data residing on disk and tape. The results of implementing a nearline solution define The Nearline Effect.

What's Happening with the Disk Industry?

There are many dynamics at work in today's magnetic disk-drive industry. While disk drive recording density advancements continue to increase drive capacity at 60% or more per year, the performance of a disk drive improves at less than 10% annually. Access density is the ratio of IOs per second, per gigabyte, for a given disk drive. This scalability disconnect creates an ever-increasing access density problem and, as disk capacity increases, disk utilization is showing signs of declining again in order to maintain acceptable drive response times. This is unfortunate, since the trend for the past five years has shown disk utilization levels beginning to increase for non-mainframe systems.

Typical allocation levels for Unix, Linux and Win2K/NT systems are normally 40% or less, while utilization levels range as high as 80% for DF/SMS-based z/OS mainframe systems. What is the benefit of larger capacity disks if the additional capacity can't be effectively used without dealing with performance tradeoffs? Though the trend to larger capacity disks appears inevitable and drives the hardware costs lower, the usability of disk drives is beginning to change. Look for a second level of smaller capacity (and diameter) disks to coexist with high capacity disks to address the requirement for higher performance.

Pricing for disk storage has been on approximately a 40% annual decline for more than 10 years and this price erosion trend is expected to last for several years. Price erosion and the soft economy have had a serious impact on the disk industry. For the first six months of 2003, worldwide disk industry revenues were down 1% compared to the first six months of 2002. Storage management challenges continue to mount as disk growth exceeds management deployment each year. This continuing saga builds a widening storage management gap that will take exceptional technological solutions with vendors working together to eventually close. The prospects for this aren't particularly encouraging at this point and more proprietary solutions are on the horizon.

What's Happening With the Tape industry?

Similar to the disk-drive industry, there are many dynamics at work in the tape industry. The tape industry is comprised of tape drives, tape libraries and tape media. Tape pricing is falling around 40% annually, more similar to disk price erosion patterns than at any previous time. Tape industry revenues are slightly down, though the shipments of tape libraries continues to increase. Tape cartridge capacity has been increasing at 60% or more, greater than disk drive areal density trends. In the late 1990s, multiple tape cartridges were required to contain the contents of a single disk drive. Today, the reverse is true as a single native capacity tape cartridge can hold the contents of any magnetic disk.

Along with the higher capacity, non-mainframe tape is experiencing higher utilization levels than non-mainframe disk storage, as the need to trade disk capacity for improved performance is less for these systems. Tape drives are benefiting from faster drive data rates and search times to help compensate for the increased cartridge capacity. For enterprise z/OS or mainframe systems using virtual tape, cartridge utilization often reaches 80% or more. Data compression further enhances tape storage efficiency, as it is used on nearly every tape drive and normally yields an average loss-less compression factor of 2X for tape data. Thus, a 200-gigabyte cartridge can actually contain 400 gigabytes or more of actual data in compressed format. As the amount of data increases, it becomes harder to manage. This is true for both disk and tape storage systems. Automation remains the single biggest factor that can reduce both tape management complexity and disk hardware expense. Tape library SANs have also increased in popularity as a way to reduce management complexity by providing a single point of management control.

The Nearline Effect

In addition to reducing the costly human efforts in manual tape or other media handling, tape automation has also had a profound effect on reducing disk storage consumption. When coupled with virtual tape subsystems, the number of tape drives and the amount of media consumed is reduced as a result of the intelligent data, volume placement and stacking used by virtual tape systems. Since STK defined the initial use of the term "nearline" in 1987, in conjunction with their Nearline automated tape library, these systems have enabled a significant percentage of moderately active or less active data to be offloaded from more costly online disk subsystems to less expensive tape libraries. The nearline level is no longer the exclusive domain of tape library subsystems though now there are over a dozen tape library suppliers. Nearline storage is being joined by a new family of low-cost SATA disk arrays implemented in the new MAID (Massive Arrays of Inactive Disks) concept.

HSM (Hierarchical Storage Management) or other data movers have been an instrumental component for driving the Nearline Effect. Much of the HSM usage has been centered in traditional mainframe environments where the use of HSM is a well-defined, standard 7x365 practice; but it is now beginning to spread to the much larger Unix, Linux, and Win2K storage users. These three operating systems are estimated to generate over 85% of the world's digital data. With non-mainframe data centers doubling their storage on an average of every 18 months or less, the Nearline Effect will begin to show similar benefits in these markets in the near future.

There are four storage topologies that illustrate the Nearline Effect. Like all storage statistics, wide variances may occur. Note: The Nearline Effect increases the amount of data on tape and reduces the amount of data on disk, thus lowering the ratio of data on disk to tape.

Industry average for all sites with and without Nearline. Though actual numbers vary by business, general measurements indicate that the ratio of the amount of tape to disk data for all data centers typically ranges between six and eight to one. Yes, there is more data stored on tape than on disk, primarily because the original copy or instance of data resides on disk and multiple backup copies or images of the original copy reside on tape.

Sites without Nearline. For those sites with no nearline implementation, the ratio of the amount of tape data to disk data typically ranges between three and four to one. This is the result of more data being kept on disk longer with less data being migrated to lower cost, automated tape libraries than the industry average.

Sites with Nearline. For sites with nearline implemented, the ratio of the amount of tape data to disk data typically ranges from eight to twelve. This ratio results from effectively migrating low to moderately-active data from disk to nearline storage, shrinking disk space requirements and increasing the amount of data on tape.

Virtual tape provides additional benefits. With virtual tape, the disk storage requirements and the tape storage requirements are both further reduced. The quick response time of the disk cache in a virtual tape system enables more disk data to be migrated to nearline since the recall time is based on disk performance. Secondly, virtual tape systems write multiple files on tape cartridges to optimize utilization and therefore reduce cartridge consumption.

The benefits of the Nearline Effect have been borne out since 1987 and can have a significant impact on disk consumption. The Nearline Effect is beginning to extend to non-mainframe systems through traditional automated tape libraries and a number of innovative SATA disk solutions coupled with the increased usage of HSM and other data movers. There are many aspects to improving storage management and reducing storage infrastructure expenses; the Nearline Effect continues to be one of the most beneficial.

Typical Expectations From the Nearline Effect

                           Disk to tape
Storage Environment        storage ratio   Comment

Overall industry disk       1x to 6-8x     All sites are included, both
and tape data ratio                        with and without Nearline.

Sites with no Nearline      1x to 3-4x     Sites with no Nearline, more
                                           data resides on disk
                                           compared to tape

Sites with Nearline        1x to 8-12x     Sites with Nearline
                                           aggressively move moderate
                                           to low activity data from
                                           disk to Nearline, increasing
                                           tape storage and lowering
                                           disk storage

Sites with Nearline and      1x to 4x      Sites with Nearline and
virtual tape                               virtual tape consume fewer
                                           disks and less tape than
                                           industry norms.