ICM: beyond just storage.

According to Forrester Research, enterprise data growth is relentless, as much as 150% CAGR in raw storage capacity with some companies seeing a doubling of data or more each year. The growth is fueled by data intensive applications, new data archiving requirements and rapidly falling storage costs that encourages customers to see capacity as a solution.

It's no secret that most businesses today are faced with explosive data growth, which they can no longer afford to keep on high-end storage platforms or employ the large numbers of staff that is required to manage their information. In fact, overall storage management software is expected to reach $9.5 billion in three years time.

At the data layer, a myriad of applications exist for data protecting, migrating and archiving to loosely address this growth. Unfortunately, most of those applications are designed for older storage architectures or for specific data types. The result is a proliferation of "data silos" where one application is used for email, another for file systems, and yet another for databases. And despite the ongoing hype around the popular concept of Information Lifecycle Management (ILM), which looks at managing data and associated metadata from its creation and initial storage based upon its importance to the businesses to the time when it becomes obsolete and is deleted, it will not live up to its lofty expectations without true automation and management at the information and value assignment layer.

Management of data from an information perspective is not a new idea. Many companies have dedicated staffs whose responsibilities are the creation of policy for managing the retention of business critical and/or regulated information. What is new is the emergence of a new class of software aimed at automating the policy creation and information management objectives at an enterprise level. This new class of software has been called Information Classification and Management (ICM). ICM essentially starts with the ability to discover and catalog data at a previously unavailable level of detail at the metadata and content level.

Described as the "missing brain of ILM," by industry analyst firm the Taneja Group, ICM goes beyond storage management, backup and recovery to analyze, classify, and move data based on policies that meet each company's needs across a wide range of industries. Without an underlying classification system provided by ICM, critical ILM initiatives including data tiering, archiving for compliance and electronic legal discovery are difficult if not impossible to implement.

ICM, the first official product solutions category within the growing ILM strategy, is not only required and necessary to achieving the ILM objectives; it will lead to higher service levels and the ability to reduce costs by storing data storage appropriately. While ILM is still a concept and a theory, ICM is very much real with actual solutions being offered on the market today, and will emerge from relative obscurity to becoming a necessary component for advanced storage solutions in 2006.

Traditional solutions are often costly and almost impossible to implement and manage. However, a data classification project doesn't have to be complex or difficult to accomplish, but it can easily escalate if the proper classification is not taken.

Classification

Why is classification important? If businesses do not know what information they have or how valuable it is, it's very hard to decide what data should be placed where and which metadata tags are assigned to it. After all, what is being stored is just as important as the architecture itself and understanding content is key to determining data's business value.

IDC attributes storage growth to three applications: data warehousing, email, and data mining. However, this neglects the problem of files from desktop applications spreading across the enterprise, which increases amounts of storage and presents a continual backup and compliance problem.

In response, the industry has produced several point products that work for a particular application or data type. The development of point solutions is a byproduct of conventional thinking about the data management problem which starts by breaking data down in three sub-types: structured, semi-structured and unstructured, and more importantly, to assign its value.

Structured -- typically is described as data stored in relational databases like Oracle or Microsoft SQL Server. While this class typically a small part of the amount of the data stored, its importance is very high because of the type of applications it supports (i.e. order entry, billing, inventory etc.)

Semi-structured -- refers to data held in email systems like Microsoft Exchange or Lotus Notes. The amount of data held in email systems is growing rapidly for two reasons, increasing email volume and the tendency of many users to use the email system as a primary storage location for their work.

Unstructured -- is everything not covered by structured and semi-structured. In practice this includes all the files generated by office applications, etc. As such, unstructured data is the least managed and fastest growing piece of the data management problem.

More than likely you will need to define a number of data classes, depending on the environment. A large enterprise might end up defining a dozen data sets to classify. Classifying and defining your data will provide a logical framework for assigning each new application to a data class based on its characteristics, and mapping it to the storage tier that accurately supports it, thus making it easier to locate and manage your data and its different levels later on.

The best ICM solutions support data stored in multiple formats, and thus can classify all three point solutions. But sorting the millions of emails and hundreds of thousands of memos, letters, reports, spreadsheets, and contracts, all of which exist in dozens if not hundreds of formats, is a daunting task. ICM and its automated policies help manage this problem, because although point products are useful in addressing the short term needs of a specific application, they have serious limitations as a solution to meet data management requirements. Practical limitations of point solutions include scalability, usability, efficiency and flexibility.

Business and IT need to manage data based on its value; they don't care whether the data is in a database, email application, file, or all three. In general, data that is stored in the enterprises falls into three high-level categories regardless of how or where it is produced:

Relevant Data -- Every organization has data and applications that are absolutely critical to business operations. Without them, the company may fail. In addition to the critical data there is also data that still has value to the business operations, but no longer requires the same level of protection or performance as the truly critical data. All of this "relevant" data must be protected and provided appropriate levels of performance for as long as it is needed.

Regulated Data -- In recent years the number of government regulations affecting how corporations manage data have increased dramatically; examples include HIPAA and SOX. The issue of government regulation is unlikely to go away and is likely to increase as governments respond to issues like identity theft. As a result certain data has to be managed with very specific requirements for retention, protection and security. Frequently this applies to data that is no longer required for day-to-day operations and is simply kept to satisfy regulations.

Redundant Data -- Typically, there comes a time when data is no longer relevant to core business operations and is no longer required to meet regulations. At this point, the data can be considered redundant. Data falling into this category can be from many sources, such as old versions of documents, old email or database records that are no longer required. All of this redundant data is consuming disk space, straining backup windows, posing a potential legal liability and generally wasting resources.

The problem faced by customers is determining what data goes in which category and how/when it moves to another category. For example, today's "Relevant Data" becomes tomorrow's "Regulated Data" and eventually becomes "Redundant Data" when it is no longer needed for regulatory compliance.

ICM and data-classification tools help IT understand the relevance of the data to the business needs. Enterprises are generating a lot more data today than any time in the history. At the same time unmanaged data is becoming a legal liability in this world especially from the perspective of confidentiality and privacy laws. Data intensive industries such as medical, financial and media industries are in need of these new management tools. After all, implementing ICM and understanding your information exchange allows managers of any enterprise ability to easily build tiered storage systems and perform data archival efficiently. Leveraging ICM solutions in a daily routine saves lot of precious capital for enterprises, while understanding business value of the data allows managers to align data management practices and operating costs. Emerging ICM solutions will provide a solid foundation for the needs of the future.

Hemant Kurande is the chief technology officer (CTO) of Scentric (Alpharetta, GA).

www.scentric.com