Selecting Storage Media for Long-Term Access to Digital Records.

The periodic replacement of digital media is a threshold solution that helps mitigate the consequences of media fragility and technological obsolescence. The replacement of digital storage media involves the use of selection criteria by information professionals to assess current and future magnetic and optical storage media. The criteria include (1) high storage capacity, (2) high data-transfer rate, (3) life expectancy of at least 20 years, (4) established and stable marketplace presence, (5) affordability, and (6) suitability. Application of these criteria leads to the conclusion that digital linear tape is the preferred storage medium to support the long-term storage of digital records.

This study explores the selection of storage media for long-term access to digital records from four different but interrelated perspectives. The first viewpoint is an overview of the current digital storage environment with an emphasis on magnetic and optical storage media. It is followed by a discussion of the so-called problem of digital storage media. The third perspective is a review of evaluation criteria for media selection. An assessment of current storage media available today comprises the fourth perspective. (Author's note: much of this discussion is indebted to the author's soon-to-be released work Authentic Electronic Records: Long-Term Access Strategies, which is being published by Cohasset Associates in 1999.)

Although media selection is a threshold issue for long-term access to digital records, it is only one component of a long-term strategy for access to digital records. Other components include maintenance of a stable storage environment, periodic media renewal, conversion to new technology platforms, and quality control procedures.

The Digital Storage Environment

The hierarchical storage model (HMS) defines digital storage media as (1) on-line, (2) near-line, and (3) off-line, based upon frequency of access. In this model, on-line storage creates higher costs, offers faster speed, but provides lower capacity than near-line storage. Near-line storage consists of optical media jukeboxes and automated tape-retrieval libraries, while off-line storage consists of shelved optical and magnetic media. Off-line storage has lower costs, slower speed, and higher capacity than near-line storage

Electronic records that have been set aside for long-term storage are no longer required for operational purposes so they fall into the off-line storage environment. Two current recording technologies -- magnetic and optical -- are used for off-line storage.

Off-line Magnetic Media

There are three different kinds of off-line magnetic media in use today: longitudinal, longitudinal serpentine, and helical. Longitudinal recording media have a fixed number of parallel tracks (e.g., 18 tracks for 3480 tape cartridges) laid across the length of the tape that are written to or read from the beginning of the tape to its end.

Longitudinal serpentine recording media also have a fixed number of tracks (128 to 204) for digital linear tape (DLT) laid down the length of the tape. Multiple write/read heads record or read simultaneously the first two or four tracks sequentially from the beginning to the end of the tape, and the next two or four tracks are read from the end of the tape, back to beginning. This stepping down technique continues until all the tracks are read or are written on.

Helical recording media involves short tracks that are recorded diagonally (at an angle of about 11 degrees) across the width of a 4mm or 8mm tape. Both helical and longitudinal serpentine recording media have a storage capacity and data transfer rate that is five to six times greater than that of longitudinal recording media.

Off-Line Optical Media

There are three different types of optical storage media: read many times (ROM), recordable (write once read many or WORM), and rewritable (RW). These three types of optical storage media share several features. Their shared physical media characteristics include

1. a rigid translucent polycarbonate substrate containing thousands of tracks

2. a reflective coating over the substrate

3. spots in the tracks, usually called pits, that represent binary 1s and "lands" (spaces without pits) that represent 1s

4. a protective clear lacquer or acrylic overcoat

These three types of optical storage media also use a low-power laser to read the representations of binary 1s and 0s. Reading occurs when a low-power laser beam is focused on a track and reflectivity is measured. Alterations (e.g., a bubble or a pit) in the smooth recording surface causes the laser beam's reflected light to disperse. A timing mechanism tells the optical head when to expect reflectance, and the optical head interprets the absence of reflected light as a transition from one or more binary 1s to one or more binary 0s. The presence of reflected light denotes a transition from one or more binary 0s to one or more binary 1s. ROM, WORM, and RW optical storage media can be differentiated by the way a binary stream of 1s and 0s is recorded and whether the recorded binary bit stream can be changed.

Read-Only Technology

The manufacture of read-only optical media involves a process whereby the binary bit stream of an information object (e.g., a book, software, or music) is replicated in the on and off modulations of a high-power laser beam that etches pits and lands on a glass master disk coated with a special photoresistant material. The pits and lands on the tracks of the master disk are replicated in a metal mold into which molten polycarbonate is injected. After the molded plastic disk cools, its underside is coated with a very thin, highly reflective aluminum coating.

Reading occurs when a low-power laser beam is focused on a track and the presence or absence of pits and lands is measured by the amount of reflected light. Hundreds of discs can be fabricated with the same mold without any error. The pits and lands in CD-ROM are irreversible, leading to the term "read only." In addition, no new digital information can be later added to the disc.

Read-Only Systems

There are two read-only technologies in use today. One is the CD-ROM disc that enjoys a wide market penetration and is an established technology. The second is the DVD disc that is an emerging storage technology with considerable future potential for the long-term storage of electronic records because of its high storage capacity and high data-transfer rate. DVD has even begun to appear in the consumer market in both computer and entertainment functions. As DVD technology matures over the next several years, it will likely achieve the same level of market penetration that CD-ROM technology has today. (For more information on DVD technology, see Taylor 1997 and Silver 1999.)

Recordable Optical Media

Recordable optical media, known generally as "write once read many" (WORM), have the same features of read-only optical media with one key difference: the method of recording 1s and 0s. In WORM recording technology, the on and off states of a laser replicates the 1s and 0s of a binary bit stream. In the on state, the heat from a focused laser causes microscopic, irreversible physical alterations on the surface of the recording material, which represent 1s. No alteration occurs in the off state.

Rewritable Optical Media 

The essential characteristic of the rewritable optical media is that its data-bearing surface deformations are reversible. Rewritable optical disks are available as magneto-optical (MO) and phase-change.

In MO technology, recording occurs when a laser beam heats a microscopic area of thin-film magnetic material in a track and its coercivity is reduced so that as the 1s and 0s of a bit stream pass through a write-head magnet, the polarity of the area heated by the laser beam is changed to match that of the bit stream. (Coercivity is the capacity of a magnetic field to resist erasure or alteration. The level of coercivity depends upon the chemical properties of the recording material.)

Reading occurs when a low-power laser beam is focused on the thin-film magnetic material in a track. A phenomenon known as the "Kerr effect" causes a different angle or incidence of reflectance for 1s and 0s respectively that is detected by a sensor and then translated back into a bit stream.

In phase-change recording technology, a laser beam heats a thin-film material that can be in one of two states: (1) amorphous, in which there is very low reflectance, or (2) crystalline, in which there is very high reflectance. The power of a focused laser beam is modulated (high power/low power) so as to duplicate the 1s and 0s of a bit stream. When the focused laser beam is at the high-power level, it causes the film to form an amorphous structure about 1 micron in diameter. When the focused laser beam is at the lower power, it causes the amorphous structure to relax and return to a crystalline state. (If the structure of the spot is already crystalline, the low-power laser beam does not affect it.) Phase change recording technology is used in DVD discs.

The Storage Media Problem

Magnetic and optical storage media are inherently fragile, so their readability and longevity are at risk. A readable digital record is one whose underlying bit stream can be processed on

1. the computer system or device that initially created it

2. the computer system or device that currently stores it

3. a computer system or device that will be used to store the digital information in the future

Digital records can become unreadable in two different ways. One is the degradation that results from exposure to a hostile storage environment. So far as magnetic media are concerned, their composition -- the magnetic particles on which signals are recorded, the substrate, and the binder that holds the magnetic particles to the substrate -- becomes unstable in a hostile storage environment. The strength of signals recorded on the magnetic particles naturally degrades or fades over time, but the rate of degradation generally increases with elevated temperatures. Also, the substrate tends to expand and contract as temperatures cycle from high to low and back to high. This event can lead to "cinches" (wrinkles) or scratches in the magnetic particles. The binder that holds the magnetic particles to the substrate is susceptible to binder hydrolysis when exposed to high relative humidity. The effect causes the magnetic particles and the bit stream recorded on them to separate from the substrate.

Optical media, of course, are relatively less vulnerable than magnetic media. However, the recording material of optical media can degrade with prolonged exposure to high temperature and high humidity (NIST 1991). In addition, the binder that holds the recording material to the polycarbonate substrate is subject to hydrolysis when exposed to high humidity. This condition can lead to small particles of the recording material separating from the substrate.

Research conducted by the National Media Laboratory (NML) on predicted life expectancy of magnetic and optical storage media confirms the negative impact of temperatures in excess of 26 [degrees] C (74 [degrees] F) and a relative humidity of 70 percent or more (Bogart 1995). According to this research, lower temperatures and lower relative humidity levels can contribute significantly to the predicted life expectancy of all media but particularly to the life expectancy of magnetic and optical media. John Van Bogart, director of media research at NML, who conducted these studies, believes that 10 [degrees] C (50 [degrees] F) and 20 percent relative humidity provide the ideal storage environment for those electronic records retained for long periods of time. (This is also the recommendation in ANSI/PIMA IT9.23-1998: Imaging Materials -- Polyester Base Magnetic Tape -- Storage Standard.)

Attaining the predicted life expectancy of digital storage media through storage in a controlled environment is important but can not resolve a more fundamental aspect of maintaining the readability of electronic records: media obsolescence.

Media obsolescence occurs when the storage medium used (e.g., a tape or disk) is physically incompatible with the available computer hardware and therefore cannot be read. Media obsolescence seems inevitable because advances in digital storage technology have introduced changes in the way the underlying bit stream that constitutes the records is physically represented. Consequently, older storage media are incompatible with those used in the present, and those in use today are likely to be incompatible with those developed in the future.

A case in point is the IBM 3590 MagStar tape. Although the 3590 MagStar tape cartridge has the same physical dimensions as the IBM 3480 and 3490/E tape cartridges, it is not backward compatible. A primary reason for this is that the 3590 has 128 recording tracks, the 3490 36 tracks, and the 3480 18 tracks.

There is no ultimate or permanent solution to media obsolescence (Ross 1995). The most effective way to mitigate its impact is to periodically reformat or copy electronic records from old media to newer media. If executed correctly, this solution to media obsolescence is likely to entail considerable cost each time such a large-scale update takes place.

Selection Criteria

The selection of a storage medium for long-term storage of digital records is critical. Media selection criteria for long-term access to digital records include

* high storage capacity

* high data-transfer rate

* minimum projected life expectancy of 20 years

* established and stable marketplace presence

* affordability

* suitability

High Storage Capacity

Over the last three decades, an exponential growth in the storage capacity of digital storage media has been accompanied by a decrease in physical dimensions. In other words, more and more bytes can be stored in smaller and smaller surface areas of storage media. A decade or so ago this was measured in megabytes (one million bytes). In the early 1990s it was measured in gigabytes (1,000 megabytes); in the late 1990s it is measured in terabytes (1,000 gigabytes). In the early years of the 21st century it will be measured in petabytes (1,000 terabytes).

The storage capacity of magnetic storage media has tended to match, if not exceed, that of optical media, although DVD technology offers a significant increase of storage capacity in a small physical form (i.e., the CD-ROM form factor of 4.72 inches in diameter).

Intuitively, a high-storage capacity seems to be an absolute benefit; yet it does involve certain trade-offs and risks in implementation. Take, for example, what is involved in transferring the content of 3480 tapes to DLT with a storage capacity of 10 gigabytes. It would take approximately 50 3480 tapes to fully use the 10 gigabytes of storage on a single DLT. The metadata and associated documentation for each of the 50 3480 tapes, including where each begins and ends, must be transferred to or otherwise associated with the single DLT. This is possible but must be done carefully to avoid inadvertent misidentification of digital records that could have the practical effect of their being lost. This problem can become acute when the number of 3480 tapes is quite large (Olsen 1999).

High Data-Transfer Rate

The data-transfer rate of the drive for a particular digital storage medium is defined as the period of time required to transfer one megabyte of data. The higher the data-transfer rate, the less time required to read or transfer data from one storage medium to another, an activity required for media renewal. The significance of a high data-transfer rate can be illustrated with DLT and CD-R discs. It would take about 55 hours of continuous operation (assuming no read errors) to transfer the content of 50 DLT tapes containing 500 gigabytes of electronic records. To transfer the same amount of information from 1,600 CD-Rs would require approximately 800 hours. (This estimate assumes that a 2X drive is being used. For more information on this topic see Pahwa 1994.) Typically, the higher the relative cost of the drive, the higher the data-transfer rate. A relatively low data-transfer rate entails a lower front-end cost but a higher back-end cost. In this context, it is indeed a question of paying now or paying later.

Life Expectancy of at Least 20 Years

A projected life expectancy of 20 years seems rather modest given vendor claims of 50 to 100 years life expectancy of certain CD-ROM and WORM optical media. Bear in mind, however, that the longevity of digital storage media exceeds the life expectancy of drives to read them, and the life expectancy of the drives to read the media exceeds the life expectancy of the software application used to process and render the digital records. Given the rate of technology changes during the last three decades, a combined life expectancy of 20 years for specific storage media, along with the necessary drive and software, is a prudent requirement. This means that the only way to extend the usability of digital records is to transfer them periodically to new storage media supported by new drives and software. And, as noted earlier, it is at this point that the data-transfer rate which the drive supports becomes crucial.

Established and Stable Marketplace Presence

This criterion is probably the single most important for selection purposes because it is what helps ensure persistence of a specific storage technology type over time. No matter how promising a storage technology may be in a development or prototype environment, it is unlikely to persist over time unless there is a substantial customer base. Such a customer base comes into being because the technology serves a mainstream market need and because there are multiple vendor products from which to choose. The withdrawal, for example, of Eastman Kodak's support for the 14-inch WORM optical disk technology highlights the importance of using a storage technology that has an established and stable marketplace presence.

Affordability

Affordability means that the cost for a particular digital storage technology is within currently available and continuing financial resources. One factor that helps to drive costs down is competition among vendors. A word of caution, however, because in some instances a digital storage technology may appear to be affordable, but substantial long-term costs may arise later. For example, most low-cost drive devices also have a low data-transfer rates, which means that considerable time may be involved in transferring records to a new storage medium.

A related consideration occurs when a vendor makes available a new, cutting-edge technology with great promise at a greatly reduced priced in order to gain market visibility. The vendor, however, may never be able to translate this initiative into a substantial customer base.

Suitability

The last media selection criterion is that of suitability, a concept which encompasses the notion of the congruence between the fundamental purpose of the storage technology and various long-term access requirements and considerations. In some respects, the notion of suitability is in opposition to the general practice of discovering new uses of existing technologies. Not every digital storage technology, however, works well in a long-term access environment. A case in point is the helical recording technology that was widely used in the video recording industry and then adapted to backup digital storage technology using 4mm, 8mm, and 19mm tape cartridges (DD-1). The 4mm and 8mm tape technologies are at the low end of the cost spectrum while 19mm tape cartridges are at the high end.

Helical recording technology involves exposure of large parts of the tape surface to the read/write head(s), which subjects the tape to substantial mechanical stress and necessitates periodic replacement of the read/write head(s). In addition, most of the error correction codes (ECC) for helical recording are not as robust as those for longitudinal digital recording technology (e.g., 3480 tape cartridge). Based upon his research experience with 4mm and 8mm tape used for backup purposes, NML's Van Bogart is doubtful that it would be possible to read the information back from 4mm and 8mm tapes 20 years later (Bogart 1998).

Digital Storage Media Assessment

What digital storage medium should be used? WORM optical media, it is sometimes argued, should be the preferred long-term storage medium because with WORM the underlying bit stream that comprises digital records cannot be changed or otherwise altered. This argument, however, does not take into account the fact that storage media are vulnerable to alteration or corruption when electronic records are reformatted, copied, converted, or migrated. In each of these activities, the so-called "permanent" recording of WORM optical media provides no protection because the physical representations of s and 0s must be translated into electrical signals that computers can understand and process. In this regard, therefore, WORM optical media are no different than magnetic media; on balance, the latter provides greater support for greater long-term access.

There are several reasons for this conclusion. First, the storage capacity of magnetic storage media, such as DLT, continues to exceed that of optical media. Second, the data-transfer rate for DLT, 3490 tape, and 3590 Magstar tape is quite high when compared with most removable magnetic media such as 4mm and 8mm tape and optical media.

For example, the data transfer rate for DLT drives is 5MB per second while that of the IBM 3590 Magstar is 9MB per second, a data-transfer rate that no optical media and drives and no low-cost magnetic storage media and drives can match. Third, although the predicted life expectancy of magnetic media is less than that of optical media, it still is within a 20-year time period within which digital storage technologies are likely to remain relatively stable.

The fourth factor is marketplace penetration and stability. Except for the 3490, 3590 Magstar, 4mm and 8mm tapes and drives, magnetic media have an established track record of reliability when compared with optical media. The magnetic marketplace is much more stable than the optical media marketplace as suggested by established product lines that tend to have a relatively long life. For example, DLT and drives, which were introduced in 1985, had a substantial market share in 1997, largely because of backward compatibility within the product line. Some analysts suggest that the DLT cartridge's strong market share is likely to persist for another decade or so (Bucholtz 1999). The same observations apply as well to 3480 tape, which was introduced in 1984. The Nordic Council Report to Preserve and Provide Access to Electronic Records notes that 3480 tapes "probably will be around for many years" (Nordic Council of Ministers 1996).

On the other hand, with the exception of CD-ROM media and drives, the market life of WORM and rewritable optical media and drives has been relatively brief and vendors have introduced new products with little concern for interoperability and backward compatibility.

Optical media vendors have created an organization called Optical Storage Technology Association (OSTA), which is promoting backward compatibility. OSTA recently announced its adoption of the Universal Disk Format (UDF), primarily for CD-Recordable optical media. UDF conforms to ISO 13346, which is the logical file-format standard for CD-Rs. In sharp contrast, DVD storage technology vendors have formed a DVD Forum that is promoting industrywide standards for interoperability and backward compatibility. However, until DVD becomes widely available, the use of optical storage media for long-term access to digital records remains problematic.

Two other considerations militate against the use of optical storage media for long-term access to digital records. Even though optical storage media may satisfy the criteria for affordability and suitability, the storage capacity and data-transfer rate of optical media are substantially less than those of most magnetic storage media. In the long run, therefore, the use of optical storage media for long-term storage could entail substantial costs when it becomes necessary to transfer them to new storage media.

Today, the mainstream of the off-line digital storage marketplace is magnetic recording technology; this situation appears likely to remain the case until DVD recording technology has multiple vendors and a substantial customer base. However, not all magnetic storage media are acceptable for long-term storage of digital records.

A prudent guideline to follow when selecting a specific magnetic storage medium is to avoid both established media on the verge of becoming obsolete and recently introduced media that are not yet firmly established in the marketplace. Implementation of this guideline would eliminate 6250 bpi tape and the 3590 Magstar tape from consideration; 4mm and 8mm tapes are excluded from consideration largely because they employ helical recording technology. In addition, the benefit of 4mm and 8mm media and drives being affordable is more than outweighed by their relatively low storage capacity and slow data-transfer rate vis-a-vis other magnetic storage media.

The magnetic storage technology that best satisfies the selection criteria reviewed earlier is DLT, followed closely by 3480 tape. Both enjoy a substantial marketplace presence that is likely to persist for some time. The commitment of vendors to backward compatibility, as enhancements are made in their product lines, along with high suitability and affordability factors, make them the preferred magnetic storage media. Of these two, DLT should be the storage medium of choice because of its high storage capacity and fast data-transfer rate.

Conclusion

This article has attempted to provide a framework within which archivists, records managers, and other information professionals can make informed decisions about the digital storage technology that best supports long-term access to digital records. Although the author recommends the use of DLT for the long-term storage of digital records, its use does involve a trade-off between increased storage capacity and the requirement to ensure full intellectual control of the content on one or more DLT. This problem can only be exacerbated as new storage media with greater storage capacities and data transfer rates enter the marketplace.

This circumstance is illustrative of a factor many information managers fail to take into account when thinking about long-term access to digital records: there is no free lunch. Whatever information managers do to extend the usability of digital records over time involves trade-offs between costs and benefits and varying degrees of risk. It is unlikely that a final or permanent solution to this problem will be achieved in the foreseeable future. Consequently, information managers need to remind themselves continually and also caution other decision makers to think long and hard about the consequences associated with the selection of storage media for long-term access to digital records.

REFERENCES

Bucholtz, Chris. "DLT: Tape's Future." 1999. Available at http://www.dynoteck.co.uk/pages/dhari.html.

Dollar, Charles M. Authentic electronic Records: Long-Term Access Strategies. 1999.

National Institute of Standards and Technology (NIST). Development or a Testing Methodology to Predict Optical Disk Life Expectancy. NIST Special Publication 500-200. 1991.

Nordic Council of Ministers. To Preserve and Provide Access to Electronic Records. 1996

Olsen, Florence. "USCS Juggles Data Requests." Government Computer News. 8 February 1999.

Pahwa, Ash. The CD-Recordable Bible, An Essential Guide for Any Business. 1994.

Ross, Seamus. "Preserving and Maintaining Electronic Resources in the Visual Arts for the Next Century?" Information Services & Use. 1995.

Silver, Bruce. DVD, CRD-R, CD-RW, and MO Media in Document Management Applications. 1999.

Taylor, Jim. DVD Demystified: The Guidebook for DVD-Video and DVD-ROM. 1997.

Van Bogart, John. Magnetic Tape Storage and Handling. 1995.

--. Electronic message to author Charles Dollar. 28 January 1998.

Charles M. Dollar, Ph.D., CA, heads Dollar Consulting. He is a consultant in archives and records management with 25 years' experience, and specializes in electronic records and archival education. He was previously with the U.S. National Archives and Records Administration and the University of British Columbia. He received a Ph.D. from the University of Kentucky. The author may be reached at [e-mail].