ARCHIVING VERSUS 'INFINITE STORAGE'.

The explosive growth of data and the difficulties in managing the increasing amount of data necessitate an examination of archive strategies. As the volume of data expands, data management becomes more and more of a challenge. An archive system can answer many of the challenges posed by this data avalanche.

A well-designed archiving solution is project-based and serves as a comprehensive media management system. Such a system should be flexible and allow users to make and distribute copies of archived data. And, if the archive strategy has been successfully implemented, archived data will be as easily accessed as live data.

There is a difference between archiving data and backing up data. Backups protect against accidental data loss and provide disaster recovery Backups cannot be tuned for data access and retrieval. In comparison, the goal of data archival is twofold:

1. Archival creates a "revision snapshot" for historical and audit purposes.

2. Archival provides cost-effective long-term storage.

Once a file is archived, the archived data becomes the primary retrieval file. Then, the original data can be revised or deleted as needed. The ability to find (and read) the data is crucial. For business, legal, and regulatory requirements, the data must be available for many years or even decades. Thus, longevity of the media, drive technology and data file systems must be weighed when designing the archive system.

And when you consider these factors, CD and DVD emerge as an excellent choice for archival. CD and DVD discs are exceptionally durable (up to 100 years or more without any special storage requirements), and there is near universal availability of drives and media based on ISO standards. These features, together with ISO file system standards for CD and DVD, protect your technology investment as well as your data.

ARCHIVING VERSUS "INFINITE STORAGE"

Placing data into a HSM or "infinite disk" system does not address the real world needs of archiving or backup. An infinitely large hard disk does not solve the basic problems that must be addressed in an effective data archiving system. As the volume of data expands, it becomes that much more difficult to manage.

Infinite storage or HSM (Hierarchical Storage Management) systems are defined by the traditional storage pyramid (Fig 1). Infinite storage transparently migrates less frequently used files to less expensive storage media with slower network access. This in turn preserves the appearance of the entire file system as a simple hard disk to network clients.

There are several problems with this approach, however:

* It does not provide data redundancy for files migrated from RAID.

* It exposes data to accidental and malicious edits.

* It lacks an integrated method to distribute data or to store data offline/offsite

* It does not provide any control over maintaining complete projects physically on the same media.

Solving the deficiencies of the HSM model requires a cohesive workflow strategy in which all data pertaining to an individual project is kept together as it is archived from primary storage (RAID) to lower cost storage media. Fig 2 shows such an implementation. It is easy to see that the true value of a nearline library is in the way that it enhances the value and accessibility of archived data. Increasing the functional size of RAID storage is NOT the goal.

ADVANTAGES INCLUDE:

* The entire archive process is automated.

* You choose which files to archive.

* You choose the access speed and retrieval process for your archives.

* You choose the level of redundancy required.

For disaster recovery a redundant copy of the data should be made for offsite or offline storage. This serves as additional protection for your archives, just like backup provides protection for live data.

The primary archive file can be stored nearline for network access or offline. By utilizing nearline access to archives, the archived data remains as accessible as live data, allowing true, seamless access and enhancing the value of the archived data. Offline data can be retrieved by placing the archive media on the server or on the desktop of the person who needs it. Once the archive system is set up, there should not be a need for a special user interface or network application for any network user to archive data to CD or DVD.

Since archive data is project-based, all associated files should be kept together. A single project can require either a single piece of media or a number of volumes. Note that only one project should be archived to a single piece of media; if multiple volumes are required, only one project should be contained in that set of media.

OTHER CONSIDERATIONS

* Automate the system.

To be truly effective, a data archival system must be automatic. Time and manpower requirements for a non-centralized system may render a manual archival system worthless.

* Create multiple copies.

You should be able to create multiple copies for nearline, offline, off-site storage, and distribution, as needed.

* Treat libraries as archives.

Nearline libraries should be treated as archives and not as live data. This greatly reduces your overall data management headaches, especially if you allow Internet access. Read-only access to archived data eliminates worries about overwriting data. And redundant offline copies provide yet another layer of protection for your data.

* Allow only read-only access to archives.

Write-once DVD and/or CD recording for archiving removes the possibility of unauthorized access to your data. All access to the media, including server access and network access, is read-only. Archived data cannot be overwritten, deleted, or changed.

* Migrate data as needed.

Many HSM systems are based on the use of an ever expanding nearline library. In contrast, an effective archive system lets you select the amount of nearline data to keep. Moving data offline should be easy, since the same CD/DVD discs used for nearline storage can be used for offline storage. There is no need to implement either a completely offline or nearline archival system. Since the data is project-based, you can determine which data can be migrated to offline storage and which data needs to remain accessible to the network.

* Use a widely available data distribution technology.

For data distribution, CD-R/CDROM technology has become the preferred method of many enterprises. It is certainly the most universal (second only to floppy diskette) of all portable data media types. DVD-ROM/DVD-RAM, CD technology's higher capacity offspring, are now beginning to ship in quantity.

Art Tolsma is the director of business development at Luminex Software Inc. (Riverside, CA) and Robert Levine is the owner of Canyon Communications (Yorba Linda, CA).