Information Lifecycle Management and the government.

Information Lifecycle Management (ILM) is a Next Big Thing in technology and management circles. ILM isn't software, but rather a combination of processes and technologies to efficiently manage data throughout its lifecycle. Data lifecycles date from creation to modification to retirement to--sometimes--deletion. (This is where data and human lifecycles part company; some data really does seem to live forever.)

The ILM process involves matching data with the storage media it needs most at given points in the lifecycle. For example, a newly created piece of data would sit on its server's primary storage array. As the data ages, migration software automatically tracks its age and moves it onto secondary online storage. The secondary storage might be a cheaper online disk array or tape library. As the data ages even more, eventually the same migration software archives it onto a data warehouse for analysis, or onto tape and out the door to off-site vaults. It may or may not be eventually deleted ("deleted" is a dirty word in certain public sector circles).

Government agencies particularly need to store, manage and protect large amounts of historical data. There is nothing like a journalist invoking the Freedom of Information Act, only to find that the government can't "find" the data. Conspiracy theories and obstruction accusations ensue, but most of the time the government really can't find it. This is not a good thing. Military, meteorological, defense and space research are other examples of data that the government really needs to be able to protect and access for reports, information demands and analyses.

But no one--government, military, or their corporate cousins--can afford to keep everything on expensive primary storage. IT departments often make do by moving aging data off-line or into on-line silos (also called black holes.) Software management technologies like Hierarchical Storage Management (HSM) make that easier, but are a subset of ILM. HSM has been around for a decade in open systems (longer in mainframe) but it's only taken off the last couple of years. HSM migrates data around various storage targets as it ages--expensive primary storage at creation, and going slower and cheaper as the data becomes less active.

ILM uses HSM architecture, but its scope is much larger: it's focused on managing data throughout the entire enterprise, not just on organizing storage schemes. ILM stakeholders include IT but go beyond it: military and government IT professionals will still be intimately involved in carrying out ILM objectives, but they won't be leading the team that sets them. ILM project leaders should be senior people with legal responsibility for maintaining the data--the staffers and officers that legislative bodies and superior officers hold ultimately responsible for information. Government must adapt its information management practices to legal realities. Within this concept, teams decide how long given information types should stay accessible, and how quickly they can be retrieved.

The designation isn't always clear. For example, older records may require ironclad protection--users may not need to access them quickly, but they had better be able to get their hands on them when they have to. Other data becomes valuable historical data, still useful in analyses and reports. This kind of data doesn't need extremely fast access but it should be available in configurations like data warehouses, or secondary storage with query ability.

Within these guidelines, ILM strategies should enable teams to:

* Identify aging rates, access, protection levels and ongoing value of data.

* Assign appropriate storage targets and paths to fulfill mix of requirements.

* Track all data from its inception.

* Store newly created data on primary storage.

* Automatically migrate aging data onto optimum storage for its value and lifecycle stage.

* Retrieve any data within a reasonable timeframe.

* Store retired data off-line or record deletions (although DoD guidelines allows deletion, it's in fact becoming more rare in government and military IT).

There is no single software (including HSM) that does all of the above. According to Dr. Shuki Bruck, co-founder and chairman of Rainfinity, ILM requires four separate building blocks: transparent data movement, global name space, policy engines and discovery and classification. Transparent data movement migrates data to different storage targets without annoying users or interrupting applications. Global Name-space (foundational for some virtualization schemes) displays filenames in a way that looks perfectly logical to the user, while the actual file may be residing in many different places. It also lets administrators change file and directory locations, and users will never know the difference. Policy Engines let administrators automate data management across storage devices. Discovery and Classification groups files by classification and maps them to applications and users. This allows the policy engines and data movers to properly identify data types and characteristics.

StorageTek also prescribes four building blocks: storage, management, replication and integration.

Storage, ILM's foundation, includes a hierarchy of devices with different price and performance options. Online storage keeps highly valuable data immediately available, nearline keeps data networked on lower performance storage and off-line archival stores data that is only rarely used again.

Management software intelligently moves information along the storage hierarchy. It should help balance cost, performance and capacity throughout the entire process.

Replication copies data, and ranges from real-time remote replication to long-term archiving. Not all data is equally important, and protective strategies should change accordingly. Archiving less critical data to ATA disk and eventually to tape is a perfectly good choice, but IT should also continuously replicate critical records to local and remote sites.

Integration coordinates storage infrastructure and consolidates (hopefully) multi-vendor installations. This can be a good trick, but standards like SMI-S are making it easier.

What's Happening Now

Several upper echelon storage developers (EMC, HP, HDS, IBM and Veritas among them) are releasing ILM software and/or hardware that support it. However, completely integrated lifecycle management applications are a year-and-a-half to two years down the pike. Mark Lewis, EMC's chief technology officer, said that the technology is certainly in an early development stage. "We don't think anyone is doing it well," he said. "We don't think we're doing it well, at least not yet."

The sniping has already begun with HP taking potshots at storage developers EMC, Veritas and IBM. HP accuses EMC and Veritas of taking a storage-centric approach to ILM, and says that IBM is concentrating too heavily on HSM. Of course, each vendor is defining ILM in terms of its own core competence and products--EMC and Veritas are storage companies, IBM is big on HSM, and HP wants to sell more copies of its enterprise manager OpenView. However, HP's philosophy has some truth to it: it sees ILM as a collection of overall business processes instead of a particular application or technology.

ILM is here today. Its strategic framework provides a foundation for identifying data and lifecycles, and can guide technology purchases and deployments. However, analysts (and even some storage vendors) are cautioning customers that it will take some time to develop a comprehensive ILM product. Right now, government can only practice ILM as a strategic framework, using storage management products from different sources. As always, the tool itself is only as good as the plans behind it.

RELATED ARTICLE: Why Consultants on DoD Regs are Very Popular Right Now

A very short excerpt from the DoD's compelling new work, "Design Criteria Standard for Electronic Records, Management, Software Applications" follows:

"Mandatory record metadata components are shown in Table C2.T3. Mandatory in the Structure column indicates that the field shall be present and available to the user either as read/write or as read only depending upon the kind of data being stored. Mandatory in the Data Collection Required column indicates that RMAs shall ensure population of the associated data structure with non-null values ..."

When you are selling ILM components to the federal government and must observe DoD guidelines, strongly consider adding DoD expertise to your staff or partner with a company who is already familiar with the standards. They will lead your company through the minefield of DoD records management regulations.