Digital Data's Future--You Ain't Seen Nothin' Yet!

The new millennium has promised to formally usher in the Information Age. The question of how much digital data exists now comes too frequently to simply leave it as one of academic interest only. As we move step-by-step closer to delivering an information utility, the need to at least get a handle on the world's digital storage profile mounts.

Assessing the amount of digital data in the world is no easy task. Though no one actually counts each byte of digital data created or knows its life span, best estimates are available and now indicate that as much as two-thirds of the world's data is being "born digital," meaning that its original occurrence was in a digital format generated by a computer. By the year 2005, it is projected that as much as 18 percent of the known data in the world will be captured in machine-readable (computer recognizable) digital format.

By 2005, nearly 82 percent of the world's data will remain on paper, microfiche, graphs, charts, various films or other non-machine-readable formats. Much of this data has existed in its non-digital form since its creation. The degree of digitizing non-digital data is relatively insignificant in the overall digital data progression. The costs of this conversion are very large and converting from nondigital to digital often involves additional application or workflow conversions, driving costs even higher.

Growth rates for digital data vary considerably. We observe traditional applications growing at 45-60% per year with some Internet, e-commerce and B2B based applications growing in excess of 100% annually. Recent and well-publicized growth rates from excite.com, who added 45TB of disk storage in less than two years and mail.com, who add approximately 27TB of disk storage every 45 days are prime examples of the upcoming storage demand profile of our future. Do these businesses back up their data, creating even more digital storage demand? Do they send additional copies to branch or regional offices or business partners? How often do they delete or retire obsolete data? As an industry in general, less data is being deleted (look at email for example), archives last longer, more and various types of backup copies are created, and applications such as e-mail replicate the same digital date for numerous users at numerous locations.

Accumulating data indefinitely without implementing archival and retirement policies can turn storage management into waste management. In addition, we see the costs of managing storage continue to range from three to ten times the cost of storage hardware. With hardware prices falling at 35-40% annually and management costs, which include an increasingly short supply of storage trained people along with software, rising at over 15% per year, the notion of not managing storage and simply buying more hardware is sometimes being mentioned as the easiest way out.

The digital data storage technologies map into a storage technology hierarchy consisting of magnetic disk and several removable media products that are making mass storage, data archiving and electronic data vaulting affordable realities for many businesses. Approximately 10 percent of the digital data in the world resides on magnetic disk storage and an estimated 90 percent of digital storage resides on removable storage. The amount of digital data on tape now ranges from four to fifteen times the amount of data stored on disk. The percentage of data stored on a technology does not represent the size of the technology's storage market however.

The disk storage market totals approximately $34 billion annually while the tape drive, library, and magnetic tape media markets total between $6-7 billion annually. The amount of digital storage is being driven up beyond historical levels by many new storage-intensive applications such as the Internet, e-commerce, and B2B, while data is being retained for longer time periods as attempting to manage very large storage pools becomes more difficult and costly. One page of text requires 2,400 bytes of storage. An 8-in.x 10-in. color photo will use about 38MB to store the digitized image. A standard x-ray takes 12MB to be stored digitally. A typical hour of telephone recording requires 22MB of storage. Fingerprints typically require 2MB per fingerprint set with an average of 4.2 fingerprints per arrest. Digital security beyond fingerprinting using retinal scans, blood types, DNA, and other biometric measures remains in its infancy but storage demand for this application is projected to grow exponentially.

Automated libraries architectures using magnetic tape, possibly small form-factor magnetic disks, the CD-DVD and possibly other emerging storage media are becoming the storage foundation containing most of the mass-storage demand growth though they will not account for the biggest piece of the hardware revenues. Making this possible are the unprecedented tape capacities now being delivered. New and emerging digital applications will continue to fuel a period of explosive growth for storage well into the next century. Terabyte-plus databases for a variety of new applications, data warehouses, electronic voice and in particular video mail attachments will drive storage requirements for "mail" applications beyond anything we've seen before. A ten-second video attachment is one thing, a two hour video attachment will be another. We will soon describe storage in new terms such as zettabytes (1021 or 1,000 exabytes), and yottabytes (1024 or 1,000 zettabytes).

The amount of storage (and bandwidth) generated by the Internet has yet to be estimated and will create countless new storage management and application developments. Beyond the Internet, entertainment may be the largest single driver of storage demand on a global basis as we demonstrate an endless appetite for any form of digital entertainment in the home, away from home and even at work. Compression techniques can create the impression of having more data than hardware capacity. Estimates for entertainment are even more difficult as the number of titles and the number of copies (legal or otherwise) per title are more difficult to track.

How much digital data is there? Estimates indicate that there will be approximately four exabytes of digital data in existence by year-end 2000. Given the assumptions you choose to use regarding the creation and deletion of data, data life cycles, replication, new application deployment, any improvements in the ability to manage data, determines the amount of digital storage to be forecasted. Using four exabytes as a baseline, and an overall growth rate of 60% (based on the assumptions) we can simply straight-line project (See Table).

Knowing the size of applications and their storage requirements gives us a basis of planning for growth, backup and recovery, archive, and when to plan for capacity upgrades without first running out. It could lead to outsourcing storage. The Storage Service Providers (SSPs) will soon have a particular need to understand the storage demand equation as the SSP business gets more competitive. Data is facts or figures to be processed, evidence, records or statistics from which statistics can be determined yielding information. Knowledge is the full range of facts, information, and principles accumulated by mankind over time. The millennium has promised to be the information age, not the data age, however. Even with growth of data expanding at unprecedented levels, the path to convert data to information and then to knowledge, promise to make any historical growth rates for pure data storage seem insignificant. Like they say, "you ain't seen nothin' yet".

Table
Year                       2000  2001   2002  2003  2004   2005
Digital data (exabytes) -     4   6.4  10.24  16.3  26.2  41.94
estimted per year at
60% annual growth