Hierarchical storage management.

Hierarchical Storage Management (HSM) is an archiving approach that migrates less active data off primary servers and moves it to less expensive storage, while keeping it available to users. George Symons, Legato's vice president of product management and development, defined HSM as "a way to use multiple storage environments for a conceptual single file system. Admin-istrators want to be able to set policy and move files from primary storage to a secondary or tertiary storage based on those policies."

It's no surprise that IT administrators need this sort of approach. Users are creating monster data files and IT is buying storage to match, but in fact most storage is seriously underutilized: common estimates for Windows systems are at 30%, and 40% for UNIX environments. Not only is storage hard to use to capacity, organizations also need the performance of disk-based systems. Caught in this cycle of under provisioning and over-spending, companies need approaches that will keep files readily available for users, yet will help them stop spending money willy-nilly on expensive disk-based storage systems.

HSM can help alleviate over-provisioning and performance problems by automatically migrating files along a hierarchy of storage devices from most expensive to least--top tier, middle tier, and bottom tier. The top tier usually corresponds to fast, primary disk subsystems like RAID arrays. A secondary or middle tier can be less expensive disk, optical, or even fast tape, while tertiary storage is usually tape or optical. In this case, HSM is an addition to backup and recovery software--instead of making copies of data and storing them elsewhere, it moves the data from more expensive primary storage to secondary or tertiary storage. The move decisions are largely automated, set in advance according to the company's business rules and by the type of data.

HSM is different from backup. Instead of moving files, which requires users to request a restore. HSM keeps migrated files available by leaving stubs at their original locations on the primary storage. When a user issues a call for a file, the HSM application locates it on secondary storage and returns it transparently to the user. The HSM application can base migration thresholds and policies on several factors and combinations of factors including age, date last used, size, user, and application. HSM improves utilization rates because administrators can set capacity watermarks. When data amounts reach capacity, the HSM application selects data by policy and migrates it to assigned secondary or tertiary storage, leaving a file pointer or stub in place.

Algorithms usually control these threshold events, but organizations must first hammer out the data management policies to define the events. For example, what is "active" versus "inactive" data? (This is trickier than it looks.) Is an inactive file one that hasn't been used for three months? Six months? A year? Perhaps an Excel file is only used once a year at annual report time, but when that time rolls around that file had better be available. The HSM application should also have the ability to migrate data without significant manual intervention, which requires using predictive algorithms on file-based and/or block-based data. It must also make secondary data locations user-accessible, ensuring that the data is transparently, and immediately, available to the user. This is even more challenging for databases than for files, since HSM applications must separate and migrate database objects to secondary storage, while keeping them completely accessible to queries.

First developed in the mainframe world, HSM is quite successful in that environment. Nick Tabellion, CTO of Fujitsu Softech, was with IBM in the '90s working with the heavyweight MVS operating system and DFSMS, which provides extensive data management services for IBM mainframes. One of the DFSMS tools is DFSMShsm, a hierarchical storage manager providing backup, recovery, migration, and space management functions as well as disaster recovery. Two factors helped mainframe customers to achieve a 70-80% device utilization rate:

1 The HSM component was fully integrated into the MVS operating system. Tabellion believes that "you have to have your hooks in the OS to make an HSM product go." When the MVS OS was updated, so were the HSM hooks.

2 MVS enables storage pooling. This is the infamous "virtualization" concept in open systems--in the mainframe world, this actually works. Storage pooling allowed the HSM tool to treat multiple storage devices as single volumes, so it did not matter to a user where a file actually was. If the user needed the file, HSM automatically located and restored it.

By the mid-'90s, vendors began developing HSM on open systems. However, implementations proved complex and only partially successful, which gave a perfectly good concept a very bad name. Tivoli and other HSM developers were faced with the challenge of needing to write HSM applications to multiple operating systems. Quite often those file systems had multiple versions and were subject to fast-changing environments. When customers added HSM to these operating systems, they added a layer of complexity at the machine level and the management level. In addition, open systems did not have workable storage pools since their storage was direct-attached and subject to disk space and distance limitations. HSM activity could also bog down the network as it sent migrates and restores across the LAN. Customers threw up their collective hands, and developers went back to the drawing board.

Symons acknowledged the previous difficulties in open systems HSM implementation but added, "I do believe very strongly that it's changing." The reason is cost: storage boasts a phenomenal growth rate and the budgets to match, but organizations are trying to rein in storage purchases and the amount of data they must manage. HSM offers a way to leverage lower cost storage devices while maintaining heavily used data on primary servers, and relieves the storage management burden by automating migrates and restores.

For example, Network Appliance (which prefers the term "data migration" to HSM) provides ATA disk-based secondary storage to clients such as energy companies and chip manufacturers. Both types of companies have extensive data stores and must access them quickly, but the sheer amount of their data has made high cost production systems prohibitive. Chris Bennett, NetApp's Director of Platforms and Systems, Product Marketing pointed out that the key to affordable, disk-based secondary storage isn't just inexpensive disks but the data migration management piece. "The key to making this stuff practical is not just slapping ATAs together. Anybody can do a JBOD, put together a boatload of drives. If you just keep adding drives they rapidly become unmanageable. Keeping track of them and maintaining them becomes difficult. The key thing is to enable rational people to employ tens of hundreds of terabytes of storage without adding appreciably to their value structure."

Both Tabellion and Symons believe that HSM increases in value with virtualization, so administrators can pool secondary storage. Tabellion said, "SANs greatly improve the environment where an HSM product might be successful. I think with virtualization you'll start seeing average utilization of NT and UNIX files dramatically improve. I'm firmly convinced we can double NT and go up to 100% on UNIX for utilization."

With SANs and virtualization, HSM on open systems may finally catch up with decades-old technology on the mainframe side. Said Symons, "Seems like we rediscover everything in the open systems world."