ILM ... easier said than done.

The 21st century has brought us the Information Age and it is making most of the world realize that the value of digital data is steadily increasing during its useful life. In the past, businesses didn't pay too much attention to understanding or managing data throughout its lifecycle. Though ILM has become a popular storage topic and a widespread IT activity, it is not a new concept. The ILM concept is clear; the path to a successful ILM implementation is not. As a result, businesses are attempting to implement their ILM strategies with varying degrees of success.

ILM Originates on the Mainframe

Enterprises have probably already been exposed to ILM's concepts from their mainframe experiences when IBM's DFHSM (Data Facility Hierarchical Storage Management) combined with the successful introduction of StorageTek's Nearline automated tape libraries in 1988. ILM is clearly more than HSM, but an HSM capability is instrumental to ILM and implementing HSM is a simple way to get started managing data through its lifecycle. This convergence created the cornerstones of the ILM model by providing a policy engine, data mover via DFHSM, and the tiered hierarchy of disk and automated tape libraries. Seventeen years later, the ILM concept is attempting to move into a much larger universe: the Unix, Windows and Linux storage market.

The Value of Data Changes as It Ages

We are learning more about the behavior and usage patterns of data every day. The actual value of data may change based on unexpected circumstances having nothing to do with data age or activity levels. Consider medical records that haven't been used for long periods of time and suddenly become important in determining the cause of a hybrid disease such as SARS, West Nile Virus, or a new strain of flu. Seemingly unimportant historical data can instantly become extremely valuable in these cases. Some major healthcare providers indicate they will be required to manage and retain medical records for a person's lifetime plus seven years. This time period could often exceed 100 years for many patients. Similar retention periods are in effect for criminal and security related records. Over the next five years, it is estimated that a successful ILM implementation will generate an additional 10-25% annual digital storage demand depending on the application or business segment.

The Back End of the Data Lifecycle is Growing

The back end of the data lifecycle is growing, not shrinking, as was the case throughout the first 50 years of the storage industry. Lifetime retention policies are now being based on data value and legality issues in addition to the most recent reference activity. This change in data profiling requires new management policies based on the value of data. It also points to the need for a universal, standard classification data scheme (a policy engine) to emerge. In the past, businesses didn't pay too much attention to understanding or managing the data lifecycle. Since 2001, compliance requirements have changed much of the thinking regarding how long data should be managed and protected. As a result, compliance has become a catalyst for many ILM initiatives. With the exception of spam, businesses are saving more digital data for a longer time than ever before. Therefore, most of the storage providers are using ILM and compliance messages in their marketing efforts. In most cases, however, the implementation challenge is often more difficult than expected.

What is Required for ILM?

The ILM model includes policies to determine which characteristics the data requires to optimally select primary, secondary and archival storage as three storage tiers are the most common tiered storage deployed. If the data is critical and requires confidential, legal, or retention requirements from the application that generated it, the policy engine recognizes this and places the data on storage-infrastructure components that offer the appropriate management, security and data protection services. At this writing, encrypting data at rest is quickly gaining momentum. Similarly, if the data needs to be referenced or shared among a variety of applications and users, the ILM engine routes it to another set of storage components better suited to store data that is often accessed but seldom modified. Data that is written once and then rarely if ever accessed is sent to either an archive or deleted.

An ILM implementation requires several components to implement. In the simplest example, an ILM implementation requires at a minimum:

A policy engine: Required to map business requirements to the data and store the data at the optimal most cost-effective location(s). The uses of data and ultimately the value of data are usually determined in a combined effort with the IT department and the business. The policy engine links business and IT plans together and this alignment is critical to the success of ILM.

A data mover: The concept that the number of accesses declines to data as it ages has been a fundamental concept of HSM for over 25 years. HSM is becoming a key component for both compliance and lifecycle management implementations. For mainframe systems, an industrial-class HSM has been effective for years and runs 7X24X365. HSM serves the mainframe businesses as both a data mover and policy engine. However, finding a robust, policy-based HSM that works for Unix, Windows and Linux platforms that is anything close to being as effective as HSM on mainframe systems remains a distant goal. HSM does not specifically address the value of data.

A hierarchy of tiered-storage: This minimal requirement becomes more important and appealing as the size of the storage environment increases. For very small IT organizations, typically less than 5 terabytes, a tiered hierarchy may not be necessary. There are many levels of the storage hierarchy though three levels are most often used. Primary, secondary, and long-term data retention are the three generic levels and each level has increased economic value to the customer as data ages. However, vendors offer many product choices within each level.

Clearly, numerous software management tools can and should be added to improve the scope and effectiveness of any ILM strategy. However, without these three components, ILM becomes a brute-force and difficult implementation.

Fact or Fallacy?

ILM is not HSM. HSM is an outdated technology that has significant drawbacks.

Fallacy. HSM is old, but definitely not outdated. If this was true, why do most non-mainframe storage administrators want a mainframe-class HSM? Some believe HSM presumes that information always decreases in value as it ages and HSM has only one "policy" that can move data to less expensive storage as the data ages. In reality, HSM allows migration based on activity levels, age of data, or other user selected parameters such as file naming conventions. ILM takes policies to the next level as it recognizes that information's value can change throughout its lifetime. The ILM policy engine describes data value and requires the data movement capabilities of HSM, or an HSM-like functionality to migrate and recall data when the value changes.

It is more difficult to implement ILM on non-mainframe systems than on a mainframe.

Fact. The mainframe has components like DFSMS (Data Facility System Managed Storage) and DFHSM built into the z/OS operating system. DFSMS provides an effective policy engine that enables optimal data placement based on storage service requirements and business values. DFHSM is a highly effective data mover, which provides a consistent user interface that moves data between storage tiers based on user-defined policies. DFSMS along with DFHSM provide a consistent user interface, data mover, and policy engine across a storage hierarchy for all mainframe systems. Unfortunately, only about 5% of the world's digital data resides on mainframe computers. The DF-capabilities don't exist in any unified or mature state for Unix, Windows and Linux, which account for over 85% of the world's digital data. Without these policy and data movement capabilities in an easy to use form, ILM implementation remains difficult on non-mainframe systems.

ILM can be implemented for databases.

Fallacy. Until recently at least. ILM for databases is extremely important since an estimated 60% of disk data resides in a structured database of some type. Databases are growing at the rate of 20-30% per year and typically 60-80% of database data is low activity data. This forces businesses to use expensive disk storage to store the low activity or inactive portion of a database. As the database gets larger, the infrastructure expense & TCO increases. Since the database appears as a single file, specifically getting to its inactive data elements (rows, tables, etc.) is difficult. A few companies now offer software to migrate certain inactive database elements to lower cost storage. When the migrated data is requested, the software retrieves the archived data for access. At this point, the database archiving activity comes from non-database providers. The database providers have historically shown little interest in optimizing the storage infrastructure. The real issue here is that current database archiving is not a widespread activity since only certain databases or database applications are eligible for these archiving services and the savings may be less than expected. Always read the fine print.

Are there any new concepts to make database archiving more appealing?

Fact. A new database storage philosophy called active archiving is gaining some interest. It helps companies safely move infrequently used data, or "inactive reference data", from production databases to inexpensive storage such as automated tape or SATA disks. A pointer replaces the data that was moved so the application is unaware of the change. The database retains the most active or most important operational data and is relieved of the burden of storing large amounts of infrequently accessed data.

ILM can be implemented with a single piece of hardware or software.

Fallacy. No single (non-mainframe) vendor offers a complete single point solution for ILM implementation. Building an ILM strategy requires a stack of software and hardware solutions from multiple providers. Integrating these solutions can be very time consuming. The providers that can build the most optimal, easiest to use solution stack will be the near-term leaders in the ILM market.

Is data protection a layer of an ILM strategy?

Fact. In addition to storing data throughout its lifetime, it is also critical to protect data throughout its lifetime. Protecting data means implementing measures to address exposures against software, hardware, intrusion and theft. Protecting data against theft has recently gained momentum as savvy businesses are considering encrypting data at rest.

ILM providers don't usually mention how data moves from one storage tier to another tier; does this create a potential problem with an ILM implementation?

Fact. Most vendors don't address this issue. One of the key remaining challenges of a tiered storage hierarchy approach is the requirement to move data between different levels or devices in the tiered storage hierarchy. This process adds significant and unnecessary I/O overhead to the customer's application servers. This makes having too many tiers of storage impractical. During periods of heavy usage, server performance can degrade quickly. Leading-edge storage vendors are finally resolving this long-standing issue by borrowing a key architectural concept from mainframe virtual tape systems, which enables data to be moved directly from device to device without server involvement. Most of the data transfer activity consists of I/O and data movement from disk to tape, but the transfers may also be from tape to disk or disk to disk. This will manifest itself in the form of virtual tape for non-mainframe systems representing a significant and long-over-due I/O architectural advancement. Businesses should challenge the ILM storage providers to reveal their strategy for server transparent, device-to-device data movement.

Summary

ILM solidifies the concept of the value of data. An ILM strategy recognizes that not all data is created equal and its value will go up or down based on circumstances as it moves through its lifecycle. As the amount of data presently grows on the average of 30-40% annually for most businesses, the data should be managed and stored on different levels of increasingly cost-efficient technologies. ILM is more than a cost-effective storage hierarchy. As ILM advances to the next level, it will include:

* determining the value of data

* regulatory compliance

* security and data protection

* optimizing storage costs

* virtualization 

* re-mastering data

* business continuity

* disaster recovery

* encryption 

Businesses are now attempting to manage their data assets throughout its lifespan with varying degrees of success. Implementing a real ILM strategy requires the most effective software, hardware, and data protection capabilities at each stage of the data lifecycle. ILM takes a big step towards assuring that the right data is in the right place at the right time--and that it is secure.

Compliance has moved ILM to center stage as digital data becomes increasingly more valuable to the survival of any business. A recent study from AMR Research projects spending to comply with Sarbanes-Oxley will reach $6.1 billion and will spend nearly $15.5 billion on all compliance-related activities this year. ILM is becoming the preferred and over-arching storage management discipline of the future dealing with data from cradle to grave. The key words here are "the future". It is no longer sufficient for ILM providers to simply talk about the need for ILM; they now need to deliver workable solutions. Understanding what ILM means is easy; building a complete ILM strategy is not. This issue will be on our radar screens for the foreseeable future.

www.ibm.com

www.storagetek.com

RELATED ARTICLE: Moore's Maxims

* Hope is not a strategy.

* As storage becomes cheaper to buy, it becomes harder to manage.

* Optical storage is no longer a data center technology.

* Without electricity, there is no IT industry.

* Storing data is one thing; retrieving data is everything.

* Doing nothing is a form of management; just not a very good one.

* Don't let your process become your product.

* In the future, new computers may not have a delete key.

* Working hard is one thing; getting something done is another.

* The best way to predict the future is to create it.

* Everybody has problems.

It is what you do about your problems that makes a difference.

* Change occurs instantly. Expect it at any moment.

* The quickest way to create change is to create a massive disruption.

* Pain is the greatest motivator of all.

Excerpted from Storage: New Game New Rules, a paper by Fred Moore, president, Horison Information Strategies [c]2003.

By Fred Moore