Building the first industry-standard storage benchmark.

This article is the first in an ongoing series exploring the evolution of performance analysis and benchmarking in the enterprise storage industry as well as the Storage Performance Council's launch of the first industry-standard benchmark for network storage systems, the SPC Benchmark-1. In the next 18 months, this industry first will establish a level playing field that will fuel a revolutionary landscape of comparison which will ultimately aid users, integrators, resellers, and vendors alike. This article series will explore the foundation of this revolution, the detailed design of the SPC-1 benchmark and how the benchmark can be used to produce more informed purchasing, configuration, and tuning decisions.

Today, there are no rigorously agreed upon standards for the benchmarking and subsequent comparison of enterprise storage subsystems. A wealth of performance analysis tools, like the Intel developed IOMeter, exist that allow vendors and integrators to stress test storage configurations across a modest set of workload parameters such as read-to-write ratio and I/O request size. But until recently, consensus has not been achieved within the body of manufacturers who sell storage products on a "fair" benchmark to facilitate the "apples-to-apples" comparison of different products, technologies, and configurations. Perhaps the best-known de facto standard for the comparison of high end storage is the proprietary work done by Dr. H. Pat Artis of Performance Associates Inc. in the IBM OS390 benchmarking tool called the PAIO driver. It should be noted that there are a wealth of CPU, system, database, OLTP, and application-based benchmarks in the computer industry peddled by a variety of organizations from Standard Performance Evaluation Corporation (SPEC) to the Transaction Processing Performance Council (TPC), but the storage egg has never reached the frying pan until now. It should also be noted that while several "stress testing" tools exist for storage, none tackle the problem of producing scalable and correlated results in the presence of storage networks like today's Storage Area Networks (SANs).

The Industry Dilemma

Enterprise applications must be online 24x7 and be online with uniform/satisfactory response times, independent of minor variations in imposed load.

These response times, characteristic of a block engine, must be upheld independent of the presence of various data protection schemes, disaster tolerant configuration options, designs for sustaining fault-tolerance, cache sizes/options, and minor imposed application loads. In addition, with the advent of storage networks:

* Processors and storage are increasingly becoming independent purchasing decisions.

* Storage subsystems are becoming vastly more complex.

* An increasing variety of technology is available for building solutions (Fibre Channel, SCSI, iSCSI, adaptive RAID subsystems, etc.).

Despite these market conditions, the new "zero latency enterprise" lacks "apples-to-apples" performance and price/performance data to accurately compare products and technologies and configure storage solutions.

The Consumer Morass

An IT director faced with the acquisition of a storage subsystem for his/her open system is potentially faced with a list of performance facts and statistics that rival the tome of figures listed in the financial section of a modern financial newspaper. These include, to name only a few: disk RPM, disk seek time, disk data-rate, back-end bus data bandwidth, front-end bus data bandwidth, the number of front-end and back-end buses, cache size, protocol time, IOPS rate (for cache hits), as well as vendor unique and incomparable benchmark or stress testing results that are utterly incomparable to any other vendor's stress test results. In mass, these statistics represent as much help for an enterprise storage purchasing decision as the statistics in a modern financial newspaper represent for an attempt to beat the S&P 500. Individually, each of these statistics are as relevant to assessing the bottom-line performance of an enterprise class SAN-based storage solution in a real world environment as using a butterfly net to catch a hippopotamus.

The Epiphany

Five years ago, a phone call from a former college friend (of InfoSizing Corp.'s Francois Raab, who is the primary author of the now legendary TPC-C benchmark) produced the epiphany that started the Storage Performance Council (SPC). His proposal was simple: Apply the incredibly rigorous principles tot benchmarking developed at the Transaction Processing Performance Council (TPC) to storage systems. The idea was genius, less one dimension. Building a TPC-class benchmark that measured storage instead of applications/systems and produced comparable results across vendors and network storage configurations was a design problem tantamount to suicide. Fortunately, none of us who agreed to build the first industry-standard storage benchmark knew that we were in for five years of working nights and weekends. It's now clear why the effort to build SPC-1 took five years and why no other industry standards organization (despite prior attempts) has succeeded.

The Forum

Executing on the epiphany required four critical elements for success:

1. Assemble most of the major manufacturers in the storage industry (the endorsement of the bulk of the industry was required for credibility) and get a modicum of blood (i.e., funding) from them.

2. Legislate the development practices of these manufacturers under the impressive quality standards established by a decade of methodical trial-and-error at the TPC.

3. Find and enlist the sheltered/reclusive storage performance experts at each of the manufacturers and induce an atmosphere of productivity independent of competitive parochialism.

4. Secure a formal administration service for the effort that was expert at TPC development practices and that could stitch together the fabric of the benchmark as the individual participants supplied by this group of manufacturers morphed over time (Walter Baker and Jack Stephens of Gradient Systems were selected).

These hurdles were initially conquered in early 1997, and the Storage Performance Council was formed with the mission to stimulate the industry to build better products and stimulate the IT community to more rapidly trust and deploy multi-vendor storage networking technology. In support, the SPC is a non-profit corporation that defines, standardizes, and promotes storage subsystem benchmarks as well as disseminates objective, relevant, and verifiable performance data and related test tools to the computer industry and its customers.

The SPC's tactical objectives are to provide to IT consumers and integrators the first accurate database of performance and price/performance results spanning manufacturers, configurations, and products. Additionally, the SPC provides new powerful tools that anyone can use to, for the first time, thoroughly analyze and tune storage networks.

At the time of the printing of this article, the SPC is the only industry standards body exclusively building benchmarks for the computer data storage industry and is composed of performance experts for virtually every major vendor in the storage industry. The SPC has developed a rich infrastructure to ensure that benchmark results will be easily produced and that the results will be credible and widely disseminated. Members of the SPC currently include: Adaptec, Compaq, Dell, The Evaluator Group, Hitachi Data Systems, IBM, Ideas International, LSI Logic Storage Systems, NEC, Sun MicroSystems, Unisys, and VERITAS Software.

SPC Roadmap

The SPC's first objective was to build a block server-based benchmark that would have the broadest possible market appeal in the online enterprise storage market and be capable of scaling to very high I/O throughput levels in storage network topologies. In support of this objective, a large number of detailed I/O traces from different classes of real-world applications were collected and applications with similar I/O profiles were organized into groups. The group comprising the largest market segment (database/OLTP servers and mail servers) was targeted for detailed workload analysis and fed into the benchmark development production line. In the future, the SPC will build benchmarks addressing other classes of applications including sequential I/O processing and, thus, support applications such as backup/restore and video on demand. Eventually, the SPC seeks to have a suite of block server-based benchmarks representing a sufficient breadth of applications that virtually any IT shop will be able to estimate the performance of a manufacturer's product in its environment. The SPC will also target a benchmark suited for Network Attached Storage (NAS) subsystems.

The First Benchmark

SPC Benchmark-1 is the first industry standard storage benchmark and the first standard benchmark for SANs. SPC-1 uses a highly efficient multi-platform and multithreaded workload generator to emulate the workload characteristics of sophisticated enterprise-class multi-user I/O applications, such as database/OLTP and mail servers. The SPC-1 benchmark enables companies to rapidly produce valid performance and price/performance results using a variety of host platforms and storage network topologies. The SPC has sought an implementation for the first benchmark that:

1. Provides a level playing field for diverse manufacturers.

2. Gives manufacturers, consumers, the analyst community, and the press results that are powerful, yet simple to use.

3. Provides value throughout the lifecycle of a storage subsystem (i.e. the development of product requirements, product implementation, performance tuning, market positioning, and purchasing evaluations).

4. Is easy to run, easy to audit/verify, and easy to use to report official and widely publicized results.

Roger Reich is the founder of the Storage Performance Council and a member of the steering committee. He is also senior technical director at VERITAS Software (Mountain View, CA).

www.storageperformance.org

Next month: exploring the metrics, tests and intended use of the SPC-1 benchmark.