Don't hesitate to automate: lower storage costs by automating storage resource and data management. (Automated Storage Management).

The tremendous growth of data and the accompanying growth of storage resources drive the need for a new breed of storage management tools that must go beyond just monitoring thresholds, generating reports, and executing scripts. Today's storage administrators are overwhelmed with the number of events they need to react to that they don't have the bandwidth to address their most critical problems. Among the typical alerts that administrators have to deal with are devices going off-line, capacity being exceeded, performance degradation due to applications competing for resources, and application servers requiring more storage space. Many of these disruptions can be avoided with storage management solutions that support capacity planning and powerful storage resource and data management automation that implements preventative as well as reactive actions.

Essentials of Storage Management Automation

End users only care about being able to save and access their data efficiently. The IT administrator's job is to provide the end users with the necessary storage resources to run their applications and ensure that the end users' data are always available to run their business. The CIO's and IT managers on the other hand, have to provide, the storage infrastructure within their given budget. The goal of storage management solutions, therefore, is to ensure that needed data is always available and efficiently accessible by the end users' business applications, as well as minimizing overall storage and administration cost. In order to achieve this ultimate goal, a storage management solution must understand how the data that are stored in different storage resources are being used. An effective storage management system, therefore, needs to integrate both storage resource and data management capabilities to optimize the utilization of storage resources.

An integrated storage resource and data management solution ties together the management of storage assets and data assets. These two types of assets are then correlated to match data storage devices with the way the data are used and their value to the organization. A storage management automation solution needs to identify unused storage resources, the opportunity for sharing such unused resource, and determine the best placement of frequently used vs. rarely used, old vs. recent, high-performance vs. low-performance, secure vs. non-secure, and high-valued vs. low-valued data.

Among the essential capabilities that .an automated storage resource and data management solution should provide include:

* Integrated data intelligence.

* Optimize storage resource utilization through optimal data placement.

* Automated management where the rules for automation reflects storage management goals.

* Capacity planning, forecasting, and storage procurement.

* Integrated data intelligence.

Storage management is only a means to an end since end users don't care about storage; they care only about their data. Storage is only important as a means of housing end users' application data. A storage management automation solution meets the end users' requirements if it is able to deliver their data efficiently, provide adequate protection and availability, while, at the same time, providing enough storage capacity as their data grows without losing access to their application servers. In order to achieve these goals, an automation solution must understand the value of various data to the organization and how they are used. It must allow administrators to classify data based on various attributes, such as group of users (i.e. marketing, engineering, accounting), application (i.e. CAD, CRM, ERP, Oracle), frequency of use, performance requirements, size, growth rate, business value, and availability requirements (i.e. must be available 24x7). At the same time, the storage management automation solution must continuously gather dynamic statistical information about who uses the data, frequency and period of usage, and space growth trends. Only with continuous intelligence about the data, can a storage management automation solution better utilize the available storage resources in the environment.

Storage Resource Optimization 

The dichotomy in the practice of storage management techniques in enterprises today is that administrators must continually react to end users and applications being out of capacity while trusted industry analysts are reporting to their executives that the storage in their enterprise is only 30-40% utilized. The rapidly declining price of storage devices allowed IT departments to add more storage to alleviate out of capacity problems, however, the increasing administrative burden of many more physical devices and the realities of today's economic environment demand that new storage management strategies and solutions be implemented. From the collected statistical and inventory information about the data, a storage management automation solution can recommend data location and optimize a company's storage utilization. A storage automation solution also gathers information about the available storage devices, such as published performance specifications, available capacity, reliability, and cost. Additional col lected attributes, including current configuration and storage allocations to different servers and users, as well as their dynamic characteristics such as observed performance, used capacity, and usage growth can then be used to better match the data access requirements with the best available storage.

To optimize storage resources and data placement, a storage automation solution must be able. to perform dynamic data migration. Optimal data placement changes with time. Recently created data tends to be used more frequently, whereas older data are used less often. Typical enterprises report that 60-80% of their online data is rarely accessed by their end users. These data can be migrated to less expensive, lower performance storage. Migration can be performed among storage with similar characteristics in order to balance capacity among those devices. An example where this is useful is in an environment where there are many servers with direct-attached storage and some servers' capacity is highly utilized while others have-low utilization. By migrating less-frequently used or less-valued data from the. highly utilized servers to those with less utilization, overall storage capacity utilization among the servers can be greatly improved. Migration may also be performed between storage devices with different c haracteristics, such as high-performance, expensive storage and low-performance, less-expensive storage. The storage devices can range from online (DAS, SAN, high-performance NAS), near-line (NAS, optical), to offline (tape) devices. Based on the selected optimization level, data migration can be completely automated or controlled by the administrator through high-level rules.

The migration performed by a storage management automation system must be transparent to the user. The user does not need to know the actual physical location of the data. For all practical purposes the data still appears to be located at its original location. If the user accesses data that has been migrated, then the automation system manages the recall of the data back to the original location. Data caching may also be employed to optimize data access after migration.

Automated Strategy-Based Management

A storage management automation solution must have the capability to automate many of the mundane administrative tasks that administrators must deal with every day. Automation does not mean simply alerting the administrator when an event occurs. An automation solution should be able to execute actions that would prevent detrimental events from occurring.

Today, administrators use scripts they write or new features in storage resource management software to define constraint rules in an attempt to protect the utilization of valuable storage capacity. Some examples of constraint rules include creating individual user or user group quotas, preventing certain types of files from being placed on high-end server volumes, and limiting the size that specific application data can grow on allocated storage resources. Although these types of constraint rules may be useful in preventing abusive utilization of highly valued resources, they tend to be inconvenient to intrusive for the end user and difficult to maintain and adjust with the company's continuously changing business needs.

Constraint rules cannot address the fast growth of business-application data that must be stored. In fact, for certain industries such as financial services and securities, the government requires companies to retain all electronic correspondences between their employees and clients. Storage-management solutions, therefore, must be capable of automating actions that administrators must perform to make additional storage capacity become available. Some examples of actions that can be automated by a storage-management-automation solution are:

* Once a month, migrate all CAD files that have not been accessed over 30 days from a group of CAD servers to a storage device with characteristic performance of 10,000 IOPS and $0.01/MB cost.

* Once a week, scan all data on all file servers, and delete all temporary files that have not been accessed over 72 hours.

* Once a week, scan all data on server A, and migrate all data that are only accessed once a week to a storage device with characteristic performance of 20,000 IOPS and $0.05/MB cost.

* On a daily basis, copy data from volume D of production server A where the data are collected and processed, to volume G of server B where the data is analyzed and visualized by an analytical application.

* On a daily basis move data on volume G of server A to volume H of backup server B.

* When allocated capacity to a server is 80% full, allocate an additional 72GB of capacity from a group of overflow NAS devices with minimum 80MB/sec throughput and RAID-5 protection.

Some of these automated actions are preventative, such as the migration of low-value or infrequently used files, or the deletion of temporary files. Others, such as the expansion of allocated storage space, are reactive. Automated tasks may also deal with maintenance, such as daily backup. Still another category of automation is the operational flow of data from one application server to another.

A storage-management solution must provide a high-level abstraction for defining automation. Administrators should not have to type command line interfaces or scripts to specify actions. Instead, they should be able to implement automation directly based on their storage-management goals. By defining storage management goals or strategies, automation rules to achieve those strategies are implicitly implemented. Examples of storage management strategies are to maximize access performance for specific applications, reduce performance bottlenecks, maximize average throughput, balance capacity among multiple servers and storage devices to improve utilization, or to optimize overall storage cost, while satisfying service levels and application requirements.

A storage management automation solution should also allow administrators to define strategies on similar groups of data or storage resources easily. Administrators should be able to create classifications of data based on attributes such as frequency of access, the owner and users of the data, their sizes, data type, applications, performance and availability requirements, business values, as well as their space growth trends. Similarly, storage resources can be classified based on their characteristics, such as available capacity, reliability, performance, manufacturer, model, device type, and cost. By creating classifications of data and storage resources, administrators can define the same strategy-based automation across similar groups of resources.

Effective strategy-based automation can only be defined after enough intelligence has been collected about the data and storage being managed. Therefore, a closed loop monitoring and reporting capability must be provided to correlate the behavior and performance of the managed system With currently active automated strategies. Logs of automation events are instrumental for evaluating the effects of specific strategies. As part of strategy-based management, a mechanism must also be provicled for verifying the consistency of strategies and automated tasks.

Analytical processing, visualization, and data-mining tools are useful for analyzing automation configurations. Taking strategy evaluation to the next step, the storage automation system should be able to recommend best-practices strategies for automation based on historical performance, inventory, and growth information. The administrator would then have a choice to accept the suggested strategies or create a custom one.

Capacity Planning, Forecasting, and Procurement

Based on historical growth trends, future storage growth can be predicted, for each server, owner, group, device, application, and data type. Storage growth analysis should be done for each class of storage: DAS, NAS, SAN, tape, online, near-line, and offline. Storage growth also should be planned for different types of data, such as temporary, replicated, and backup. Near-term planning for growth can be done through defining the appropriate strategies and automated tasks to migrate, move, or permanently delete data to free up space, as well as expand server and user allocation of storage. Long-term planning for growth determines when additional storage purchases need to be made.

Storage procurement needs to be tightly integrated with capacity planning and forecasting. Capacity planning analyzes when data growth will exceed current storage capacity, and what types of data, what applications, for what servers and users, and how large the growth will be. The IT manager, however, also needs to know what kind of storage to purchase in order to satisfy data growth. This is where storage procurement comes in. Where storage allocation and expansion deals with capacity growth within already existing storage, storage procurement projects future storage growth and recommends the best storage device(s) to purchase that would optimize cost and deliver the required performance. On a per-application, per-user, and per-server basis, storage procurement determines the resource growth requirement and determines what type of storage, in terms of cost, performance, reliability, and availability, would match the data growth requirement.

Future Directions of Storage Management Automation

Storage management automation solutions will start to go beyond monitoring, reporting, alerting, and simple rules-based constraints to support automating the control of resources such as storage configuration, allocation, and transparent data movement and placement. More intelligent storage-management automation systems will begin to appear in 2002 and 2003 to analyze the collected inventory and statistical information about data and storage resources and to understand which data is more valuable and should therefore be placed in more expensive high-performance storage. The junk files that are rarely accessed, if at all, can be stored in any cheap JBOD. Since the value of data is dependent on an organization's line of business, storage management software will allow the administrator to define storage management strategies by logical business groupings (e.g. business units, functional departments, work groups).

Automation of daily storage resource management operations, such as expanding allocated storage, and migration of infrequently used data from branch to corporate offices are areas that can significantly reduce the administrator's burden. Managing LUN and volume configurations, security, allocation, integration with virtualization, and storage procurement are capabilities that will become part of a storage automation solution.

As technology in this area matures, storage management automation will be tailored to specific applications such as databases, CRM, ERP, document imaging, or content management. Each has applicationspecific resources and data types that must be handled differently and where the utilization of those resources can be optimized from detailed analysis of data usage.

Storage management automation is directly aimed at lowering the cost of storage utilization and procurement and improving the productivity of storage administrators by allowing them to automate rudimentary tasks and focus on critical issues requiring human intervention. With an automation system that integrates storage resource and data management, the administrator will have a self-managing storage environment to optimize data placement and dramatically increase overall storage utilization. IT managers will also be able to reduce overall storage acquisition cost by identifying the most cost-effective storage resources to purchase in a just-in-time manner.

With better intelligence about the data, integration of storage resource and data management capabilities, such as automated data migration, and powerful strategy-based automation, storage management automation solutions will enable storage administrators and IT management to deliver to their customers the level of service demanded by the organization within the budget defined by their current business and economic conditions.

Dr. Claudia Chandra is director of product management at Arkivic (Mountain View, Calif.)

www.arkivio.com