MIT's super archive. (Up front: news, trends & analysis).

Each year, Massachusetts Institute of Technology (MIT) researchers create at least 10,000 papers, data files, images, collections of field notes, and audio and video clips. Much of the material is published in professional journals, but the rest is ferreted away on personal computers, Web sites, and departmental servers, accessible only to a few.

Until recently, there has been no plan to archive or preserve such work. But in September 2002, MIT launched DSpace, a Web-based institutional repository where faculty and researchers can save their intellectual output and share it with colleagues anywhere for centuries to come. DSpace, the result of a two-year collaboration between MIT Libraries and Hewlett-Packard (HP), is built on open-source software and available to anyone free of charge. More important, many believe this digital repository will revolutionize the way research is shared and preserved and fundamentally change the way scholars disseminate their research.

The $1.8-million project is part of the five-year, $25 million MIT-HP Alliance, a research effort to develop digital information systems. In the spring of 2000, the project team that includes HP software developers, MIT administrators, and a faculty advisory committee began developing the system--a one-of-a-kind repository that can store all types of digital files and is accessible from any computer on campus. Every document stored in DSpace has a unique and permanent URL. Materials submitted to the repository are organized within a community--a school, department, lab, or center. Each community sets standards for DSpace content and determines who will be authorized to access documents.

When it first went online, DSpace could store almost a terabyte of data--enough to accommodate the information on about 1,500 CD-ROMs, but not enough to hold all the work MIT faculty have stored on their own hard drives and CD-ROMs. MIT plans to add storage capacity as necessary.

According to MacKenzie Smith, association director for technology for the MIT libraries and DSpace project manager, two types of digital repositories exist currently. One is for library holdings that happen to be in digital format, and the other is a preprint archive tailored to scholarly papers in a discipline that is a vehicle for getting them out quickly. These are not concerned with long-term preservation, whereas DSpace is committed to preserving published papers and their supporting documentation.

MIT and HP encountered several challenges along the way to building the super archive. Because applications change over time, a big challenge is the existence of different versions and operating systems that often can't talk to each other. To address the problem, the libraries developed a way to catalog formats for which MIT has the specifications and is, therefore, able to develop software that converts files to other formats as needed. The team also had to address the search function needs of DSpace users. It had to be easy for people to find their way through the millions of documents that will reside in DSpace. The developers selected Lucene, an open-source search engine that can index metadata as well as text and can be extended with additional sophisticated search capabilities. DSpace now uses Dublin Core, an established standard for creating the metadata that describe documents in DSpace, but the team is looking for a better solution. A three-year digital archiving project will explore how to provide metadata that is customized to specific disciplines but that also is searchable and manageable across the entire system.

To make DSpace inexpensive to upgrade, the team divided it into exchangeable modules that can be replaced as new versions become available. To ensure documents would be readable on computers of the future, the team developed a list of supported formats with the requirement that the libraries will keep them available and readable in the future. For unusual formats, the libraries guarantee bit preservation, storing the ones and zeroes of the original documents.

The majority of scholarly material at most universities goes unshared. But once DSpace is up and running, it will serve as a portal not only to MIT research, but also to research at partnering institutions. MIT has formed an alliance with other research institutions--Columbia University, Ohio State University, and the universities of Washington, Toronto, and Rochester--which will adopt DSpace. More than 30 other institutions are expected to install DSpace on their campuses as well.