Best practices for implementing data lifecycle management solutions.

What is the definition of high value data? One can argue that all data is mission critical at some point in its life. The idea behind data lifecycle management is that data has different values at different moments in time and should be managed differently in different phases of the life cycle. For instance, a law firm would consider the legal documents required for discovery and litigation in an active as very important before the final ruling is made, but afterwards these documents diminish in value over time. However, there are exceptions, such as the landmark case of Rowe vs. Wade, where the possibilities of re-use of the data are significantly higher.

Data lifecycle management solutions enable IT managers to assemble the appropriate combination of storage devices, media types, and network infrastructure to create a proper balance of performance, data accessibility, easy retrieval cost, and data reliability based on the relative value of two to the business. Consider how insurance companies store client forms on-line for two years to comply with government regulations and ensure responsive customer service; but after the retention period is passed, this data is migrated to off-line tape storage.

There are several components of a data lifecycle management solution. First, you need to have an inventory of your existing data and storage resources. You should classify your data and associate it with your business requirements and relative importance. Finally, you need to have data management policies that decide how to best match data with the appropriate storage resource. Should data be stored on-line, near-online, or off-line and when should data be replicated, migrated, or deleted? The ability to intelligently and dynamically automate these decisions and actions is the heart of a data lifecycle management solution. Following, are the best practices of implementing data lifecycle management solutions.

Everything Old is New Again

It is very easy to simply state that data lifecycle management is simply Hierarchical Storage Management (HSM) born again. Why invent this new category called data lifecycle management? Vendors first developed HSM products in the mid-1990s in the mainframe environment for distributed computing implementations. In HSM implementations, data automatically moves from expensive hard disks to less expensive optical media or to tape, according to specific policies.

HSM technologies have their shortcomings, though, which have slowed adoption. For instance, HSM products lack integration of data intelligence and scalability in a network environment, increasing management complexity in large disparate client/server environments. Before administrators can build HSM migration rules, they first need to understand the enterprise's data usage patterns and storage resource availability. What is your total capacity, available and used? What are your most active files and which files consume the most storage space? This type of information is typically not readily available unless a storage assessment was recently completed. Consequently, administrators using HSM products would often create migration rules in a vacuum. A rule such as migrate all *.doc files not accessed within the last 60 days may be appropriate fur one group of users, but may negatively impact productivity of another group or department.

Another issue stems from the fact that legacy HSM solutions were architected for static one-to-one relations between source and target volumes. Administrators were limited to manually configuring each dedicated HSM volume as a pre-defined source and/or destination device, and any migration rules could not be shared between multiple HSM servers. This has made HSM technologies impractical to implement for large distributed client/server environments. Finally, HSM technologies lack a global view of all storage resources in the network and instead require the administrator to constantly change the migration rules to adapt to the changes in the storage environment. For example, what if the original migration target is getting full? What if the administrator brings up a new storage device that can be used as migration target?

Data Lifecycle Management: The Need

Clearly, data lifecycle management at minimum represents a significant evolution of HSM techniques. IT managers are rethinking their migration strategies, fearing HSM was far too simplistic an approach. Today, many companies now want real-time access to their data for longer time periods. Consider how this requirement affects credit card transactions. In the past, credit card transactions were generally completed within a 120-day cycle. Because customers now have Web access to their credit card accounts, they now want the ability to review their transactions for the past year or perhaps longer. Even if customers don't use the data, it must be readily accessible or the value of the service is lost.

With the continuous reduction in the cost of DASD, IT administrators are storing more and more data on-line in order to speed access time and take advantage of new low-cost technologies such as ATA disk-based RAID. Customers want to implement tiered-storage solutions to automate data migration to these new low-cost storage devices. They want to migrate duplicate or inactive data from primary storage to secondary storage devices to reduce backup and recovery windows.

The key question for IT managers is how do they determine which data should reside on one storage device vs. another (i.e., highest performing, most highly available but costly storage vs. low-cost storage)? In fact, one of the toughest challenges with data lifecycle management is profiling the relative value or criticality of data and storage resources to the business. Only then can administrators match the right data on the right storage at the right time. By properly placing data on the appropriate storage according to business needs, IT can more effectively distribute data across multiple resources, which leads to improved storage utilization and reduced storage acquisition costs. Automate this process and IT can benefit from improved productivity levels as well.

Data Discovery and Identification

The right starting point when implementing a data lifecycle management solution is to inventory your data. What are the different types of data files users create in your organization? What file types are most or least common? Do data types vary by user or departments? How important are these files for the business? How much space does the individual data type consume on each volume? Can you predict their growth based on trends?

A data lifecycle solution should scan each server and volume and its contents in the network and summarize the information in various reports. Moreover, the reports should provide the information at various levels, ranging from a global network perspective down to the individual volumes; allowing the administrator to drill down on specific volumes when necessary. This enables administrators to quickly and non-intrusively determine total capacity (available and used) by servers, directories, volumes and groups. Administrators are also able to report on the parameters of the data, such as space consumption by top N files and directories, file age distribution and data usage patterns, stale file analysis, and more.

In the past, this data collection process was done manually using custom-developed scripts, pencil and paper, and custom spreadsheet that summarized the results. Traditional storage resource management (SRM) technologies requires agents to collect such information. However, newer SRM tools follow the agent-less approach. This new approach utilizes industry standard protocols such as CIFS and NFS, as well as other network services such as Microsoft Active Directory and NIS. This allows the administrators to quickly access their storage environment, especially in environments where storage servers are installed without IT knowledge.

Data Classification and Valuation

The next step is to organize and classify the data and storage resources--across multiple storage volumes--into logical resource groups. A data lifecycle management solution should have the intelligence to automatically group together certain file types, such as image files (i.e., jpeg or tiff) or audio/video files (i.e., mp3 or way). Administrators should also be able to create custom file groups using attributes such as file type, extension, age, size, last modified/accessed times, directory paths, or owners.

A major benefit of the file resource groups is the administrators' ability to categorize and prioritize their data according to organization-specific business rules, simplifying their data management policies without worrying about identifying and managing individual files or directories. For instance, an engineering company may group together all the CAD/CAM files not accessed within the last 180 days which are located in several project folders. Once the engineering team has completed the prototype of the new product design, the IT team may want to migrate this data from primary storage (e.g., SAN) to lower-cost storage (e.g., ATA-based RAID) in order to ensure only their most active projects utilized their high-performance storage. Another example might involve the administrator setting tip a different file group for all spreadsheets created by the Finance department and applying a more vigorous data replication strategy to this file group for business continuity purpose.

A data lifecycle management solution should also logically organize storage volumes into resource groups based on attributes such as storage type, cost her megabyte, capacity and make/model. These storage groups might logically bind together volumes that span multi-vendor storage systems, which greatly simplifies overall administration. For instance, an Administrator could create a storage group for all ATA-RAID devices and designate it as a migration target. This reduces the need to modify any existing policies when a new ATA-RAID device is brought online.

Policy Automation Engine

At the heart of a data lifecycle management solution is the policy engine, which incorporates these global resource groups into central migration policies. Even though the files and volumes reside in many different storage devices on the network, the policy engine executes policy actions on the data to achieve different storage objectives. If the objective is to optimize cost, then the policy engine will make the best data placement decisions based on the different levels of cost within the storage network, making the more cost-effective use of storage resources. As an illustration, the administrator would set a migration threshold (i.e., 70%) and if any volume within a high-cost storage group exceeds this threshold, the policy engine selects the less critical data and migrates it to another volume in the low-cost storage group, making more space available for additional high value data.

If the objective is to optimize utilization across multiple servers in a server farm, the policy engine selects and migrates data from over-utilized volumes to under-utilized volumes until capacity utilization is optimized across the network. Other policy actions might include replication of high value data on primary storage for disaster recovery, movement of data to automate storage consolidation, or deletion of inappropriate files.

Since no storage environment is static, a data lifecycle management solution cannot be static, which is a limitation for most custom storage management scripts. For example, a data lifecycle management solution needs to handle new files created by users or new storage purchased by IT without requiring administrator to reconfigure existing policies. The appropriate global resource groups should be automatically updated and any existing migration policies should be dynamically changed to apply to the new data and/or storage resources.

User/Application Impact

Ideally, the data lifecycle management solution should be completely transparent to applications and users who do not necessarily need to know where their data is stored as long as it is accessible. In a tiered-storage migration policy, data is typically moved from expensive hard disks to less expensive on-line storage, optical media, or to tape. Administrators should not have to inform the users that their files are in a new location, nor should they have to go to the client systems and change the file access paths. The users should not even know that their data has migrated to a less costly storage media.

A data lifecycle management solution needs to track where data is relocated to and must make the data available to the user and/or application as requested. One common technique is to separate the data from the file object. When the data is migrated, the file object in the local system still contains all the important attribute information about the file (e.g., file name, security information, etc.) but the data is now stored in another location. This file object now takes up much less space (1 disk allocation unit) instead of the full size the data, which may be multiple megabytes or more, When a user or application retrieves a file that has moved down the storage hierarchy, the data lifecycle management solution retrieves the data from the migrated target location in background and returns the data to the user application with minimal latency.

Backup Optimization

IT departments continually face the problem of expanding backup windows. This problem worsens as the total amount of data stored online increases. Administrators know that they do not need to backup all the online data all the time but lack the tools to implement such a strategy. Data lifecycle management solutions migrate less frequently accessed data from primary storage to secondary storage, so the administrator can now implement full-system backup policies only on the critical and active data remaining on the primary storage. This reduces the backup window and simplifies any required recovery.

Optimize Storage Utilization

Even the best IT departments struggle to achieve greater than 30-40% utilization across their entire network. Usually there are a couple of servers that are always near capacity limit while the others are significantly under-utilized. Instead of all the down-time, cost and overhead involved in expanding the storage capacity of the over-utilized storage (and storage expansion may not be feasible at all in some situations, requiring a new server replacement project), data lifecycle management solutions automate migration of data from over-utilized volumes into under-utilized volumes across DAS/NAS/SAN environments, which extends the life of primary storage and increases the utilization of under-utilized storage.

Tiered Online Storage

New storage devices are available that are pushing below $.02/MB--and prices are still coming down. However, most IT departments cannot take advantage of this cost savings because the manual steps involved with selecting and moving data to these devices makes the whole solution cost prohibitive. Data lifecycle management solutions enable these companies to automatically and transparently migrate less critical or inactive data to these lower cost devices; reducing the overall cost-per-terabyte and deferring future expansion requirements of high-cost primary storage.

In summary, next generation data lift, cycle management solutions, which offer a new approach to storage resource and automated data management, have arrived. By understanding the value of data and storage resources, data lifecycle management solutions match data to the most appropriate storage resources in order to reduce costs, consolidate and optimize resources, and improve operational efficiencies. A data lifecycle management solution should not involve a collection of ad-hoc tools. Instead, it should integrate the various components that include monitoring of storage utilization and data usage patterns, classification and valuation of resources, and policy-based automated data management actions. In the end, the policy engine plays a vital role by intelligently and adaptively making the best decisions about where to optimally place data on the storage network to ensure the right data is on the right storage at the right time.

www.arkivio.com

Albert Leung is CTO and Glenn Rhodes is director of product marketing at Arkivio, Inc. (Mountain View, CA)