Digital Preservation: A Global Information Management Problem.

In an e-mail, Svanhildur Bogadottir, city archivist of the Reykjavik Municipal Archives, Reykjavik, Iceland, writes:

   At my archives, we are now receiving information in digital form almost
   every week. In endless sizes and formats, up to 15 to 20 years old. Most of
   it is impossible to read for us today and some of it has permanent value,
   but it is not available on paper, so it is really lost. It is a great and
   GRAVE problem how we can be sure to preserve our information today for the
   future. This is not a problem for the future but for today.

This brief statement underscores digital preservation as a global information management problem. If the city of Reykjavik experiences problems reading and processing older digital media, what about the large archival repositories operated by national and state/provincial governments throughout the world? And what about the untold thousands of electronic recordkeeping systems in business corporations that have long-term retention requirements but no formal policy or practices in place to assure their preservation?

For many years, archival institutions throughout the world have operated programs to preserve records possessing archival value that are worthy of permanent preservation. These programs have been organized around physical records -- paper, microfilm, and other visible record media. Now, however, archival repositories worldwide find themselves in a long transition from paper to electronic records as the predominant recordkeeping medium, a transition that began during the '70s, accelerated during the '80s and '90s, and will not come to fruition for another decade or more.

Archivists, records managers, and other information management specialists throughout the world face reinventing their professional practices to ensure permanent or long-term preservation of electronic records. It is no hyperbole to say that this is the biggest challenge the records and information community has yet confronted.

The essence of the digital preservation problem is how to support the ability to digitally process and read many years from now the electronic records being created today.

The digital preservation problem is of greater magnitude than problems associated with preserving physical records. With physical records, the information is contained on media that are durable. Moreover, physical records can be read by sight or with relatively simple viewing devices. Generally, deterioration is visibly apparent, and there is a time window -- usually measured in years, if not decades -- during which conservation measures can be undertaken if required.

Technically, there is nothing particularly complex about preserving physical records. Once appraised as archival, they are accessioned into the archives, finding aids are prepared for them, and they are housed in environmentally appropriate space. When these steps are completed, the records should retain their integrity and usability for many decades, even centuries, under proper environmental conditions. Finally, assuming the steps have been properly completed, the preservation of physical records is generally a once-and-done task. No further measures should be required.

With electronic records, however, the situation is drastically different: the recording media are not durable, and the records are not human-readable. The hardware and software required to read electronic records have even less life expectancy than the media on which they are recorded. Thus, the tasks required to ensure long-term preservation of digital records are technically difficult -- specialized expertise as well as the requisite hardware, software, and systems documentation are required. Moreover, the data preservation tasks must be performed periodically for as long as ongoing retention is required.

In short, there is not an ideal once-and-done solution to permanent preservation of electronic archives. However, archival institutions throughout the world do not have the luxury of doing nothing while they wait for such a solution to appear. Electronic records of archival value are getting older by the day, and they may be threatened with attacks on their integrity and retrievability. They may even be abandoned by their owners as old data. As a matter of top priority, archival institutions worldwide must have plans for dealing with this situation, and they must implement such plans as aggressively as possible.

Basic Requirements

The National Media Lab, located in St. Paul, Minnesota, has performed studies for many years on what it takes to preserve digital data. Dr. John W. C. Van Bogart, the lab's principal investigator for media stability studies, states: "Digital archives should be transcribed every 10 to 20 years." To realize lifetimes greater than this, one would be required to maintain all of the following:

* recording system and media

* system hardware and software

* operating system

* operations manuals

* ample spare parts

This requirement suggests that digital preservation facilities must adopt the museum approach, saving these processing components indefinitely. This approach may not be necessary.

Current state-of-the-art best practices in digital preservation employed by archival institutions throughout the world include:

* selecting storage media most appropriate for long-term data retention

* converting data to standard formats to facilitate its processing on a variety of computing platforms

* migrating data to new technology platforms when the computing environment is upgraded

* preserving systems documentation required to process the data

* copying or recopying the data onto new storage media at regular intervals

* taking steps to store and maintain these media properly

These measures are labor-intensive, error-prone, and costly. They must be performed periodically for as long as the data are retained. Thus, the long-term preservation of digital data is a substantial undertaking. Although new storage media in development have excellent potential for long-term data retention, they are still a long way from an ideal solution. These matters will have to be worked out over the next 10 years or so.

The Biggest Problem: Sustained Organizational Commitment

Nearly all commentators agree that technological obsolescence represents a far greater threat to the preservation of digital archives than does media longevity. In all hardware and software market sectors, service lives of less than five years are commonplace. Even the most fragile media's life will likely exceed the continued availability of readers for them. Thus, efforts to preserve physical media provide only a short-term, partial solution.

Technology obsolescence may not be the greatest problem compared to the lack of organizational commitment and the willingness to allocate sufficient resources to preservation efforts. The vicissitudes of organizational life make it highly problematic to sustain any management program in perpetuity. Organizations and the people who run them change frequently. Any organization that contemplates establishing a program for the permanent preservation of digital data must consider this reality.

One Example

Pfizer in the United Kingdom provides a specific example of a digital preservation initiative. Pfizer is a United States-based international pharmaceutical company employing some 46,000 people in 57 countries. The company's annual sales are about $14.7 billion. Fortune magazine ranks Pfizer the world's sixth largest pharmaceutical company.

As creator, user, and custodian of electronic records, Pfizer's needs include long-term, secure, managed storage of identified, business-critical records. In the pharmaceutical industry, new drug products may take 10 years or longer to develop. Products are subject to rigorous inspection by national regulatory bodies to establish their safety for human (or animal) use and their effectiveness for intended purposes. Records and data that support a marketed drug's development must be available for regulatory inspection during the compound's lifetime, a period generally assumed to be 40 years or longer.

In response to these digital preservation requirements, Pfizer's central research department at the company's Sandwich facility designed and implemented a central electronic archive (CEA) to provide long-term, secure, managed storage of the company's critical business records. The CEA is a product of close cooperation among several units: the computing services (IT) department, the pre-clinical information technology department, the records management unit (RMU), and several other interested departments. Under development for about four years, the CEA became operational in 1996.

Pfizer's CEA supports critical business records generated electronically that require long-term storage in a managed archival environment. Storage of these records in paper format is neither practical nor an alternative to storing them in their native electronic format. The CEA system, however, is not an electronic document management (EDM) system, since these software solutions typically support short-term retrieval and workflow needs rather than the myriad requirements for long-term management of electronic records.

The CEA is a custom-designed application. Pfizer conducted a global survey of the archival and records management systems market and determined that there was no single packaged software solution that would meet the company's requirements. To function, they needed to provide direct user access to online and near-line storage of records, supported by retention schedule functionality and by system and record migration plans. Several systems developers and integrators worked with Pfizer during various stages of the CEA's development.

The CEA's major features include:

Records management functionality -- The CEA is a managed archive with all the records management functions that this term implies. Each record must be indexed by various metadata elements and be properly designated for retention scheduling and archive management actions under rules established by the records management unit. These rules treat records archived to the CEA in the context of their value to the whole company, not just to the department that created them.

Data/document types management -- The CEA manages wide varieties of documents and data created on Pfizer's many computing platforms: word processing documents, spreadsheets, scanned images, raw data, processed data, and many others. Most records are created and managed while active on Pfizer's DEC VAX/Alpha servers.

User requirements -- To describe and store records, users must complete various metadata fields in a computer screen window. Users assign a records series type for every record or group of records. The records series type links the record to a departmental retention schedule maintained by the records management unit.

User-controlled archiving -- When records become inactive and are ready for storing, users copy them from the VAX to a format that permits their management by the CEA. Graphical user interface software, installed on desktop PCs, activates the process, which captures the metadata and archives the files selected. The software preserves their integrity (i.e., their evidential and informational value) so that they remain identifiable, accurate, and meaningful in the context of the purpose for which they were created.

File restoration -- The CEA's functional components identify file formats and the software version that created the file. Files can be restored to any server as long as security restrictions have been satisfied and the server can accommodate files of that type. Restored files can be read by the source software (the software that created them) or by any other software version that is able to read that particular file.

Records management unit responsibilities -- The RMU is responsible for overall management of the CEA's records retention aspects. This unit periodically reviews the CEA's holdings, applies the retention schedules, and provides ongoing user training and education.

Hardware and software components -- Three distinct device types support the CEA application: 1) desktop PC clients, which have the CEA client software installed; 2) the Open VMS system, which provides file services over the network to the PC clients; and 3) the CEA server, which stores all the archived files. The CEA application runs on a Sun UNIX system supported by an optical disk jukebox for bulk data storage. The file index (i.e., the metadata describing each file) resides in an Oracle database, which also runs on the Sun server.

Software for managing data migration -- Hierarchical storage management (HSM) automates the migration process and supports the CEA. The HSM software separates active and inactive data, automatically migrating inactive data from primary storage devices (online disk drives) to near-line or off-line secondary or tertiary storage devices. The HMS software is Epoch from EMC, a leading provider of data storage products.

Modular system design -- Since Pfizer's computing environment changes constantly, the CEA has a modular design to facilitate hardware and software upgrades. For example, an additional desktop platform (such as Windows NT Workstation) can be added by modifying the desktop client software. The CEA also incorporates a plan to migrate electronic records forward in time when new technology platforms are deployed.

Future plans include linking CEA record metadata with other corporate databases so that they can be validated or captured automatically. Pfizer is exploring intelligent software links, free text searches, or HTML as ways of adding metadata elements without additional user effort for archiving records. Other long-term preservation plans include migration of source software versions, migrating records to appropriate data standards, and purchasing "virtual" computers to run obsolete software via emulation protocols.

Subsequent columns will explore the digital preservation problem's other dimensions as well as global solutions to it. They will delve more deeply into best global practices for digital preservation in both the archival and IT communities. These columns will also examine the emergence of digital preservation service providers from the vendor community who may offer new solutions to this global information management problem.

REFERENCES

Beagrie, N. and D. Greenstein, "A Strategic Policy Framework for Creating and Preserving Digital Collections." British Library Research and Innovation Report 107. London: United Kingdom. 1998.

Bowen, David B. "Practical Issues in Implementing a Central Electronic Archive." Proceedings of the DLM Forum on Electronic Records. Brussels, Belgium: European Commission. 1996.

Bennett, J.C. "A Framework of Data Types and Formats, and Issues Affecting the Long-term Preservation of Digital Material." British Library Research and Innovation Report 50. 1997.

Electronic Records Programs: Report on the 1994/95 Survey. Committee on Electronic Records, International Council on Archives. 1996.

Feeney, Mary, ed. Digital Culture: Maximizing the Nation's Investment. London: National Preservation Office, The British Library, 1999.

Final Guidelines on the Management of Electronic Records from Office Systems. Public Record Office. 1998.

Guide to Depositing Data. The Data Archive, University of Essex.

Guide for Managing Electronic Records from an Archival Perspective. Committee on Electronic Records, International Council on Archives. 1996.

Guidelines on Best Practices for Using Electronic Information. Office for Official Publications, European Commission. 1997.

Haynes, D., D. Streatfield, T. Jowett, and M. Blake. "Responsibility for Digital Archiving and Long-term Access to Digital Data." British Library Research and Innovation Report 67. 1997.

Hendley. T. "Comparison of Methods and Costs of Digital Preservation." British Library Research and Innovation Report 106. 1998.

Lord, Philip. "Strategies and Tactics for Managing Electronic Data Records: A View From the Pharmaceutical Industry." Proceedings of the DLM Forum on Electronic Records. Brussels: European Commission, 1996.

Matthews, G., A. Poulter, and E. Blagg. "Preservation of Digital Materials Policy and Strategy Issues for the UK." British Library Research and Innovation Report 41. 1997.

Murdock, Alan. "Roles and Responsibilities in Managing an Electronic Archive." Proceedings of the DLM Forum on Electronic Records. Brussels: European Commission, 1996.

Ross, S. and A. Gow. "Digital Archaeology: The Recovery of Digital Materials at Risk." British Library Research and Innovation Report 108. 1999.

Statement of Principles: Preservation of and Long-term Access to Australian Digital Objects. National Library of Australia, National Preservation Office. 1997.

Waters, D. and J. Garrett. Preserving Digital Information. Washington, D.C.: Commission on Preservation and Access and Research Libraries Group, 1996.

David Stephens, CRM, CMC, FAI is vice president for the records management consulting firm of Zasio Enterprises Inc. He has been a consultant in the field of records management for more than 18 years and has published books and articles about information management in the United States and abroad. The author may be reached at [e-mail].