Leveling The Playing Field for "Open" Providers Of Storage.

Introducing SPC-1: the quiet performance riot

Systems and storage are increasingly becoming independent purchasing decisions. Customers in this "multi-vendor" world, however, lack apples-to-apples data to assess the performance of competing products. Furthermore, the performance of enterprise storage subsystems can vary dramatically. Stories ring through the industry of solutions (and vendors) being replaced when they fail to provide adequate performance to mission critical applications. In walks the Storage Performance Council or SPC.

SPC who? For the past three years, an unsung team of performance experts from virtually every major storage manufacturer have been busy building the first industry standard benchmark for storage. SPC members include Agilent, Compaq, Dell, Hewlett-Packard, Hitachi, IBM, Ideas International, LSI Logic, NEC, Sun Microsystems, Unisys, and VERITAS Software. The tactical objective of the SPC is to provide a single web site that purchasers can browse to obtain up-to-date and relevant performance and price/performance data on products. This web site will fuel the SPC's strategic objectives of allowing purchasers to make informed product selections, driving vendors to build better products, and stimulating the industry toward "multi-vendor" mix-and-match utopia.

The Council's first benchmark (SPC-1) is designed to have broad market appeal and provide a level playing field between "open" providers of storage (either direct attach or storage networks). As such, SPC-1 is characterized as predominantly a random access workload for server class computer systems that is applicable to most enterprise class storage environments today. This random 110 environment is generally modeled after one of the most ubiquitous applications in the market--a mail server. In addition, the SPC has designed its first benchmark to be vendor-neutral and platform independent. The benchmark was developed leveraging experience gained at other standards bodies (like the TPC) to assure that it provides value throughout the product lifecycle (e.g., development of product requirements, product implementation, performance tuning, capacity planning, market positioning, and purchase evaluations).

The draft SPC-1 standard is currently in public review phase and can be obtained at www.StoragePerformance.org. The Council's objectives in 2000 are to collect public comment on SPC-1, polish it into a final standard, and build a set of test kits that sponsors can license to efficiently produce results. The intent is to make SPC-l easy to run, easy to audit, and easy to use to produce results. Unlike performance benchmarks in other areas of the computer industry, ease-of-use is being designed into the specification and testing process to cajole storage vendors into reporting a plethora of results across their product lines. SPC-1 tests and resulting metrics are designed with an understanding that there are two classes of environments that are critically dependent on storage subsystem performance (see Table).

The first environment is comprised of systems, which have many users or many simultaneous application execution threads that can saturate the total 110 request processing potential (i.e., throughput) of a storage subsystem. An example of such an environment would be an online transaction processing system handling airline reservations. In this case, the success of the system rests on the ability of the storage subsystem to process large numbers of 110 requests while maintaining acceptable response times to the application(s) it supports. The maximum 110 request throughput capability of a storage subsystem in this environment will be documented by the SPC-1 IOPS (I/Os Per Second) result.

The second environment is comprised of business critical applications whose success is dependent upon minimizing their wall clock completion time, since they must issue thousands of synchronous 110 requests (one after the completion of another) to finish. An example of this environment would be a relational database rebuild operation. In this case, the total 110 request throughput on the storage subsystem is kept small in an effort to drive to bare minimum the time required to complete each of these synchronous 110 requests and, thus, achieve significantly reduced wall clock completion time for the application. The ultimate strength of a storage subsystem in this environment is documented by the SPC-1 LRT (the Lightly-loaded Response Time) result.

Also key to the benchmark is the process of reporting a SPC-1 benchmark result, as test sponsors must publicly disclose the price (including three years of service) of the configuration being tested. This reporting practice allows the computation and comparison not only of performance, but also price/performance. To date, the price/performance of enterprise class storage products has been carefully shrouded, leaving consumers, in many cases, penny-wise and pound-foolish. Thanks to the SPC, the storage industry will not only end up with a level playing field for performance comparison, but will also gain new financial perspective across the different classes of storage solutions flooding the market.

Why so long and so quiet? The delay in building the industry's first storage benchmark is due to three reasons. One, the Council wanted to ensure that the benchmark would be embraced by the industry as a whole and chose to follow a formal, and sometimes glacial, standards development practices. Two, the resulting benchmark standard prohibits cheating (for example, carefully prohibited are products that are inappropriately engineered to exclusively favor the benchmark). Three, the benchmark has defined metrics that will provide developers, consumers, the analyst community, and the press with results that are powerful and yet simple to understand. To these ambitious ends, the council has virtually "locked itself in the basement" to facilitate productivity. Now, that effort is poised to pay off.

The SPC is a non-profit corporation that defines, standardizes, and promotes storage subsystem benchmarks, as well as disseminates objective, verifiable performance data to the computer industry and its customers. SPC membership is open to all companies that manufacture, integrate, or distribute storage products, as well as interested parties, including members of the press and academia. Contact the SPC Administrator through www.StoragePerformance.org for additional information.

Roger Reich is the technical director at VERITAS Software (Colorado Springs, CO) and the founder of the SPC.