Finding a solution to the data growth problem: Warren Daniels, Princeton Softech.

Enterprises today face a major challenge to their operational efficiency and the quality of their customer service in the shape of increasingly rapid data growth rates. In recent research conducted by the International Oracle User Group (IOUG), 25 per cent of storage professionals said their data growth rates would increase between 11-25 per cent and a further 13 per cent said that they would grow by more than 25 per cent over the next 12 months.

Enterprise ERP, CRM and custom applications are driving strategic initiatives and generating revenue opportunities. Processing more transactions and collecting more customer information may be great for business but unmanaged data growth can negatively impact the ability of businesses to provide superior service and support. The effects can slow application performance, strain financial and technical resources and jeopardise completing business-critical processes on time.

A Thirst for Data

One of the key reasons for this growth is that commercial organisations are carrying out far more electronic transactions than they have ever done before. This development is being fuelled by the growth of ebusiness and increasing reliance on the Internet. In addition, initiatives like retailers' card loyalty schemes are driving a real thirst to store data on individuals.

The proliferation of new devices and systems across enterprises and employee populations is also driving data growth.

The third key factor is that there is a raft of legislation globally from Sarbanes-Oxley to Basel II and from the UK's Data Protection Act to the Financial Service Authority's (FSA) Policy on data retention, which is driving the necessity to store data for longer periods.

This enterprise data must remain easily accessible so that companies can respond to audit or discovery requests. Although much of the historical data retained for compliance reasons may never be needed again, non-compliance carries the risk of severe penalties. 23 per cent of respondents to the IOUG survey identified additional transaction volumes as a key reason for this growth. 15 per cent highlighted new systems as important and a further 11 per cent stated compliance was key. Interestingly, 16 per cent did not know where the additional data was coming from. Today, many database professionals are struggling to predict with any degree of accuracy the volume to which data within their organisation is likely to grow.

Issues to Resolve

So what are the implications of these exponential rates of data growth?

Application degradation is becoming an increasingly important concern. The more a business's databases expand and the more data it stores, the more the performance of the applications it supports will degrade. 60 per cent of respondents to the IOUG survey reported that lack of available storage impacted on the performance of their databases.

Consequently, enterprise applications run more slowly. 46 per cent of survey respondents recorded that the issue had affected their database availability. And this reduction in performance inevitably has a knock-on effect on employee efficiency and customer service.

Data management also becomes much more complex as a result of rapid data growth. It is often difficult for organisations to anticipate where surges in demand are likely to come from. As a result, enterprises that traditionally would have bought more storage to solve their data growth problem can no longer budget accurately for additional IT infrastructure and hardware to satisfy their storage needs.

They no longer know what their data growth is likely to be over a given period. This issue is becoming an increasingly significant management challenge, particularly as many businesses react by pulling in 'rogue' budget from elsewhere within the organisation.

In addition, most large corporates are running enterprise applications, which typically require frequent systems upgrades and migrations. Large enterprises often suffer severely as a result of prolonged downtime of these business-critical applications.

Applications may even be out-of-service for extended periods of time until the installation of the upgrade or new release is completed. Often, this is because of the huge volumes of data that have to be ported from one application to another as part of the process. Extended downtime can cost millions of pounds and lost business cannot be recovered.

Storage no panacea 

Inefficiently managed data growth also has significant cost implications. The traditional approach has been to increase spending on expensive software, servers, databases and storage. This common response may resolve constraints temporarily. However, even frequent performance tuning and upgrades do not address the underlying problems of an overburdened IT infrastructure.

So what alternatives can enterprises explore is addressing data management issues? Historically, businesses have worried that by archiving data they are effectively deleting information from their databases.

In fact, the best data archiving applications allow users to remove historical or little used data from their production database and then transfer it to a separate archive file before compressing it.

Reducing the amount of information in the production database means that less disk space is required for application data, cutting the costs of storage. And because there is lessinformation to sift through, applications process faster, operations run more efficiently and organisations derive more business value from mission-critical applications.

Reducing pressure on the production database enables users to speed their applications and compression allows them to reduce their data footprint. This, in turn, allows businesses to significantly reduce their spend on costly nearside storage. Data within the archive remains visible to the application but because it uses less expensive storage, it is much cheaper to retain than it would be if retained within the production database.

This type of enterprise data management strategy can be applied right across an organisation independent of application, database or infrastructure. And typically, large organisations can achieve a payback on their investment within six months. At the same time, employee, supply chain efficiency and customer service levels can be dramatically improved.

There are also significant benefits from a compliance perspective. Data within a live production environment can still be accessed modified and changed. However, once it has been archived, it cannot be altered or modifed. In compliance terms, it is audit-ready.

A well-executed enterprise data management strategy ensures that all referentially integral information is maintained within a given archive file.

Reducing the Multiplier Effect 

Enterprise data management can also help to address the 'multiplier effect'. The rapid growth of application data in production systems is replicated across all "cloned" environments, typically including the development, quality assurance and staging functions. Adding to the complexity, it is not uncommon for organisations to maintain several backup copies of critical data or to implement mirrored databases that provide assurance against data loss.

As data is duplicated, storage and maintenance costs increase proportionally. The "multiplier effect" increases capacity requirements and limits the amount of space available for further database growth or emergencies. Unless companies can find ways to streamline the source copy of the data, the expense will quickly become prohibitive. Enterprise data management allows companies to take a proactive approach that addresses the challenges of the multiplier effect, without diverting additional resources from IT business initiatives. By implementing archiving to segregate current from historical transactions and moving the data to a secure archive, businesses can effectively maintain application databases at a manageable size.

Using subsetting capabilities creates "right-sized" test and development databases, reducing the multiplier effect even further. Once a business establishes data management policies for production and testing environments, much of the required processing is automated. Routine archiving provides consistent results for managing the impact of the multiplier effect across the enterprise requirement levels on an ongoing basis. For example, if a business wanted to archive all historical financial data over two years of age, it could implement an archiving strategy to archive this data automatically.

In addition, by maintaining a set amount of data in its production database over a given period, a business can budget efficiently for its data management requirements.

Looking to the Future

Today, more organisations, particularly large financial services companies, major telcos, utilities and government and public sector organisations and major retailers, are appreciating the benefits of enterprise data management. In the UK, we are increasingly seeing businesses archive a specific application database within the business, before strategically rolling out the strategy across the rest of the organisation.

Typically large businesses will stand to benefit most from adopting an enterprise data management strategy. However, database archiving as a best practice for data management can be successfully applied whatever the size of the organisation.

Including routine archiving as part of scheduled application and database maintenance procedures provides ongoing control for managing data growth, enabling businesses to meet performance targets consistently while providing superior customer service.

www.princetonsoftech.com