The impact of compliance on storage: will you benefit from increased demand?

After years of research and analysis, the expected lifetime of people, animals, plants and most other objects has been documented and reasonably well understood. For data, this process is just beginning. Most businesses still don't know the value or the required lifetime of their data. Increased concerns about terrorism, anti-money laundering, corporate corruption, and increased privacy have moved the compliance issue to center stage. As custodians of most of this increasingly valuable data, the storage industry has a new demand driver in compliance. Numerous new government regulations have made us change the amount of time we preserve data and the way we manage data as it ages.

For example, SEC rule 17a-4(t) mandates several new digital archiving requirements. These requirements include what type of storage format should be used, how long data must be retained, and where and how long duplicate copies of data must be stored, as well as specifying countless security policies. The back-end of the data lifecycle is swelling, not shrinking as was the case previously, and retention policies are now being based on data value and legality issues, not just reference activity. This change in the storage landscape calls for new management policies based on the value of data and means that a universal, standard classification scheme for data needs to emerge. All data is not created equal and many storage management vendors are working on advanced data classification schemes to generate a value factor for specific data. Is compliance with regulatory agencies worth the expense?

Increasing regulatory pressure to comply with federal and global mandates for e-mail, medical/insurance, legal, financial/government classified data is forcing many businesses to fortify any potential weak points in their long-term storage systems. New applications and a variety of legal and business requirements are driving the need for many businesses to reexamine or finally create their security, long-term storage and archival policies. One of the most visible examples of the emphasis on the increasingly critical value of archival data lies with the HIPAA (Health Insurance Portability and Accounting Act) requirements. Not only does HIPAA require health providers to preserve data for a yet to be determined time period, but the failure to protect critical patient data carries penalties presently ranging up to or exceeding $25,000 per violation.

The threat of the fines and other forms of non-compliance are encouraging storage administrators to examine the increasing amount of archival data required to be kept indefinitely for future reference. For example, the PACS (Picture Archiving and Communications System) application that captures and stores radiology information and other medical images is a primary component of the HIPAA requirement. Data used to be retained for one year, and then three years, then seven years, now infinite retention seems inevitable for some applications. Some health care businesses are planning to retain digital records for patients' lifetimes plus seven years (which could be over 100 years). At that point, the data may never be deleted. The growing list of regulations is becoming increasingly important to storage administrators' data management strategy and includes:

* The Sarbanes-Oxley Act: Defines rules for falsification of records and e-mail with retention and deletion guidelines requiring data to be kept 4 years after audit

* HIPAA: Health Insurance Portability and Accountability Act for medical images and records, possible lifetime or longer retention periods

* Telecommunications: Title 47, Part 42

* Banking: OCC and FDIC regulations

* Defense: DOD 5015.2 regulation

* Osha: Records of individuals exposed to toxic substances retained for 30 years

* Pharmaceutical and Life Sciences: Records for food kept for 2 years after availability

* Electronic transactions: The Uniform Electronic Transactions Act 

* Brokerage Business: SEC Rule 17a-3 and 17a-4, for the life of the business entity

* The U.S. Patriot Act 

* Numerous other regulations are under review: Estimates suggest as many as 10,000 total regulations may exist

Lingering Compliance Questions

Understanding what happens to digital data throughout its lifetime is becoming an increasingly important aspect of effective data management. Compliance has fueled the concept of Information Lifecycle Management, but keep in mind that ILM is more than compliance. What happens to data as it ages? Does usage decline as data ages? Does the value of data increase or decrease as it ages? Why are we keeping more data longer than ever before? What conditions indicate when data should be retired? Do storage management requirements change as data goes through its lifecycle? If data is the most valuable asset of so many businesses, why do we know so little about it? Why don't any storage vendors include the non-digital assets of a business such as film and paper as part of their ILM strategy or offerings?

Impact of Compliance on Storage

These questions have become increasingly important and are in need of answers in order to understand where data should ideally reside and how it should be managed during its existence. In particular, the probability of reuse of data has become one of the most meaningful metrics for understanding optimal data placement and it is important for HSM (Hierarchical Storage Management) to be more effective. For much digital data, the axiom of "90 days on disk and 90 years on tape" applies for lifetime management. For most all data types, the number of references to data significantly declines as it ages. This basic observation provides deeper insight into more cost-effective storage management as it enables the movement of less active data to lower-cost levels of storage. The lower frequency of access as data ages has been a fundamental concept of the HSM concept for over 25 years, and HSM is becoming a key component of both compliance and ILM implementations. However, finding a single, robust, policy-based HSM as effective as the HSM on mainframe systems that works for Unix, NT, and Linux platforms remains a distant goal.

Managing Compliance Data

A better way to manage data throughout its lifetime is paramount. Note that I said managing data, not devices. The lifetime of data now exceeds the lifetime of the devices containing the data. Data will not reside on the same piece of media for the duration of its digital lifetime for several reasons. Therefore, the ability to move data to newer technologies throughout its lifetime is required. Most likely, it won't become possible without some major enhancements to the existing levels management capability. As we continue to observe, data is growing faster than our ability to manage it. As SAN deployment continues to evolve, optimal data placement within the tiered storage hierarchy will begin to occur automatically without human involvement but will initially be host based. Later, these functions will move outboard of the servers and will be implemented as either an in-band or out-of-band function in the storage network itself. Advanced policy-driven SRM (Storage Resource Management) software is positioning to measure reference patterns and trigger management actions that result in moving data to the most optimal storage locations throughout its lifetime.

Final Thoughts

The value of data is increasing, irrespective of economic and other pressing global issues. As the storage management requirements for data change over time, storage management has become a lifetime activity. Where data is initially stored is often not the same place where it will finally be stored. The value of data will change during its lifecycle based on unforeseen conditions. Begin building your compliance strategy now. The months that lie ahead promise that even more compliance concerns are headed your way. Today, compliance may seem like an option. Tomorrow, compliance will not be an option.