The abundance challenge: create more storage demand; capacity far outstrips ability to utilize it.

Most people believe the storage industry is exploding. But you might ask: "Which part of the storage industry is exploding?" The overall IT market has experienced minimal revenue growth rates since 2000 and is projected to reach $965B in annual revenues by the end of 2004, according to market research firm IDC. Generally divided into three major segments--services, applications and software, and hardware--overall growth rates are projected to be in the 5% range annually for the foreseeable future.

Projections for 2004 indicate that the largest IT revenue segment will be services, totaling just over $400B while generating 41% of all IT revenues. This segment includes storage services, IT consulting and education, operations management, break/fix and implementation.

Implementation services represent the largest component of this segment accounting for 36% of all service revenue, and it reflects the growing requirement businesses face to successfully implement complex and non-open computer technologies after reducing staffs for the past three years.

Storage services represent the smallest services component, totaling just under $25B and accounting for just 6% of all IT services spending for 2004.

Applications and software represent the smallest IT segment and are expected to generate just under $200B, accounting for 20% of all IT revenues by year end 2004.

Growing at just over 5% annually, storage software is the smallest software segment accounting for nearly 4% of all applications and software revenues. Storage software revenues had been projected to be more than twice current levels but have declined as reduced staffing levels, the slow progress of interoperability, and budget reductions continue to slow the rate of storage management software deployment.

Hardware presently accounts for approximately 39% of all IT revenues. Storage hardware accounts for nearly 9% of all IT hardware revenues and is the smallest hardware revenue component, totaling $31.5B. Servers account for almost $51B or 13% of all hardware revenues.

Clearly, people spend more on computing than on storage. Overall, worldwide disk industry revenues generated approximately $31B in 2001 and climbed back to just $20B in 2003. The magnetic tape industry revenues consisting of drives, libraries and media have ranged between $4B-$5B since 2001. A consistent oversupply of storage products resulting from lower global demand than expected continues to propel hardware price erosion at 30-40% per year, per gigabyte.

Where the Data Comes From

The question of how much digital data exists in the world is best addressed by the second University of California, Berkeley, study on digital data creation (www.sims.berkeley.edu/research/projects/how-much-info-2003).

This comprehensive study defines magnetic, optical, print and film as the four types of physical media where new data is stored. These four media produced nearly 5 exabytes of data in 2002. Magnetic media (disk and tape) accounted for 92% of the total amount of data stored; film represented 7% of the total, with paper and optical media storing the remainder. This means that 92% of the world's data is now born digital while just 8% is created in analog format. E-mail is presently generating about 400 petabytes of digital data per year, making it perhaps the most storage-intensive application. Unfortunately, unwanted spam represents over 60% of all e-mail messages sent today.

Most of the world's digital data is generated by the four computer operating systems. Unix is expected to generate 47% of the world's digital data, making it the largest overall storage market by the year 2007. Windows operating systems are projected to generate 38% of the digital storage market. Linux is the fastest growing operating system in terms of storage growth, presently experiencing 102% compounded annual growth rate; however it will represent just 7% of the total digital storage market. Linux systems are increasing in popularity though deployment of business-critical applications remains minimal. The traditional mainframe system, now named the z/Series, is expected to account for 4% of the world's digital storage by 2007. The ultra-high availability of the z/Series enables it to manage well over 50% of the world's mission critical data.

Where Will Data be Stored?

The world's digital data is stored on disk, tape and optical media. More data is stored on removable storage than on fixed storage. The amount of digital data stored on removable storage ranges from four to fifteen times the amount stored on disk, depending on the business and operating system. Enterprise or mainframe systems have a higher ratio of data on tape to disk than non-mainframe systems, as the widespread usage of HSM (Hierarchical Storage Management) software significantly optimizes consumption of more expensive disk storage by moving less-active data to lower cost storage. The use of HSM on non-mainframe systems is not yet pervasive, though its time is rapidly approaching.

Magnetic media is the predominant data storage technology and no replacement is in sight. Magnetic tape cartridge capacity is now growing faster than disk-drive capacity. This is particularly significant because disk recording densities had been increasing on an average of 60% annually since the early 1990s. Disk recording densities finally displayed signs of slowing from these traditional levels in 2003. Disk-drive projections indicate capacities of nearly 5 terabytes in 2013 and native tape cartridge capacities ranging as high as 10 terabytes by 2013.

As disk-drive capacity increases, the actual allocation levels for high-capacity disks are gradually falling as a percent of total capacity. This is a direct result of the decline in disk access density as disk capacity has been increasing at 60% annually while disk performance is increasing at well below 10% annually. Higher capacity disks contain increasingly more data than their predecessors and therefore potentially support more concurrent user requests, increasing the probability of device contention and longer device I/O response times. As a result, allocation levels are falling as a percentage of total disk capacity to maintain acceptable response time levels. Many storage industry vendors won't recognize that scalability means more than just increasing capacity and requires scaling in two dimensions: capacity and performance. Without scaling performance in parallel with capacity, effectively utilizing the capacity of higher capacity disks is becoming increasingly more difficult.

Conclusion

Some exciting storage milestones have been reached this year. The industry's first 500-gigabyte disk drive was just announced using four 125-gigabyte platters, while the first one-terabyte native capacity tape cartridge has been previewed. The ability to deliver continually higher capacity devices has been never been a major concern for the storage vendors. The ability to utilize these devices effectively has now become a major concern for storage consumers. The continued over-abundance of storage capacity ensures that storage demand can be met at any level. The size of the worldwide storage industry can be measured in two ways: revenues and demand. Overall, the worldwide IT community is spending less on storage hardware than it did in the year 2000. Though annual demand for storage is expected to range between 30-50% per year for the next few years--indeed a significant growth rate--revenue levels are expected to grow no more than 3-5% annually. Clearly, the explosion in storage is in demand, not in revenue.