The Hole In The Hierarchy.

Last month's CTR article focused on data intensive applications and discussed their access and storage characteristics. Some applications are I/O intensive while others are transaction intensive. Some are read intensive and others are write intensive, some favor sequential access and some others are always accessed randomly. Some files are active but only at certain times. Understanding the application's profile is achievable but not always a trivial task. Sometimes the profile is obvious like video streaming, which is read intensive, 110 intensive, and is accessed sequentially. More often however, storage and performance measurement tools are needed to determine what an application is actually doing.

Once the application's profile is understood, determining the most cost-effective storage solution for the application is the next task. For much of the past decade, the storage solution was solved by disk drives. Disk storage was viewed as inexpensive, didn't seem to need much management compared to other alternatives, and could handle nearly all types of application storage profiles. After all, storage hardware prices per megabyte purchased were (and are) falling at more than 30% annually so why bother with exploiting the full hierarchy of technologies available? A new century brought a new understanding to the storage management equation. The industry is quickly becoming aware that the total cost of managing disk storage hardware (hardware, software, and personnel) now ranges from 3 to 10 times the acquisition cost of the hardware and is growing. A new factor for storage to deal with is higher energy consumption and increasing cost. When will energy costs exceed the cost of data center hardware? We have a ll seen reports that energy consumption for computing equipment is growing at a staggering rate and shortages or power outages are becoming more common. This question will loom larger as storage prices decrease more than 30% annually while energy costs are now increasing at well over 30% annually. Having data spin on disks for days or weeks without any activity is rapidly becoming less cost-effective as the total cost of ownership becomes a more visible issue. This issue alone may encourage storage administrators to keep only the most frequently accessed data online.

Hierarchy Components

The storage industry has used a pyramid or triangle for years to depict a hierarchy of data storage products that span all levels of price, performance, and capacities available.

For channel attached storage, Solid State Disks (SSDs) provide the highest I/O performance of any storage device. SSDs consist of DRAM chip technology eliminating the latency and seek components from a traditional disk I/O. SSDs are frequently used as an application accelerator, especially for databases.

Magnetic disk holds approximately 10% of the world's digital data and more than 70% of all of the world's mission critical data. Magnetic disks have clearly defined both a high-performance and high-capacity level in the hierarchy. In parallel, the capacity of magnetic disks typically increases at an average of 60% annually. Another disk, optical disk, has been largely squeezed from the general storage hierarchy, as it has not kept pace with magnetic disk and magnetic tape developments in areal density, price, and performance. DVD offers some promise, though standards issues and low performance may limit its potential to that of being a replacement for CDs. In reality, optical disk is a proven and very successful consumer technology but is no longer viewed as a data center storage solution.

Nearline defines the level of storage between disk and far-line or shelved storage. Nearline is removable storage and uses robotic technologies to retrieve storage media automatically. The media used in nearline storage is most often magnetic tape, a sequential access technology. This level is the basis for the vision of the future for mass-digital storage as today more than 85% of the world's digital data resides in removable storage. For data centers, this is primarily tape. In addition, over 90% of the world's data are now born in a digital format according to the landmark Berkeley digital storage studies, see (http://www.sims.berkeley.edu/research/proj ects/how-muchinfofindex.html). Far-line, or manually retrieved storage from the shelf, still represents the vast majority of the world's data, but the path to digitization is slowing, not halting, its rate of growth.

The Hole In The Hierarchy, A Possible Solution

If the application profile is known and the storage device parameters are known, can we optimally map any application to the available storage solutions? The answer is not optimally. Where does a hypothetical application with very high capacity, random access that is infrequently referenced optimally reside? It could optimally be stored on disk except for the infrequently accessed requirement. It could be stored on tape except for the requirement for random access. Optical storage theoretically was aimed at handling both of the requirements of random access and infrequent use but it didn't solve the problem and was surpassed both in capacity and performance by magnetic storage by the middle 1990s.

The "hole in the hierarchy," also referred to as the "access gap," is defined as the requirement for high capacity, random access, and removable storage as a result of the need for infrequent access. This is the final unfilled gap in storage hierarchy. The requirement for random access points to disk and the requirement for infrequent access points to a removable media if, for no other reason, the energy inefficiency of keeping low activity data on spinning disks for long periods of time. What if a robotic library contained hundreds of cells each containing a magnetic disk? After all, disk drives have shrunken in size and mass to the point that a high capacity 2.5-inch diameter disk fits nicely in a robotic cell and can be handled with a robotic hand. The hypothetical application could request the disk(s) to be mounted whenever needed meeting the requirement for random access and then dismount the disk(s) meeting the infrequently accessed requirement. The time presently needed to mount a copy of the applicat ion or file data residing on tape and stage it to a disk to enable random access would also be eliminated as its primary residence would be disk.

Hardware prices for magnetic disk storage are projected to continue the 30% or greater annual decline for the foreseeable future. Keeping inactive disks in a removable storage cell significantly reduces needless energy consumption, becoming a significantly more cost-effective element of the storage hierarchy. Finally, future considerations for this concept in conjunction with a SAN are appealing. SAN virtualization schemes enable all disks in the removable library to appear to be online. This fits nicely into the evolving role virtualization plays for SANs. (See "Storage Virtualization--Strategic For SANs", p. 1, March CTR).

The challenge for someone to effectively complete mapping all the DNA of data to all the DNA of technology remains. Though much of the storage hierarchy has been in place for years, the welldefined hole in the hierarchy now stands out. With many new and emerging applications requiring large amounts of storage capacity steadily appearing on our horizon, the demand to fill the final hole in the hierarchy has arrived. The question, who will seize the opportunity first, remains.