The workload driving the first industry-standard storage benchmark.

This article is the third in an ongoing series exploring the evolution of performance analysis and benchmarking in the enterprise storage industry as well as the launch of the first industry standard benchmark for storage developed by the Storage Performance Council (SPC). This industry first will, in the next 18 months, establish a level playing field that will fuel a revolutionary landscape-of-comparison that will ultimately aid users, integrators, resellers, and vendors alike. This series of articles explores the foundation of this revolution, the detailed design of the SPC-1 benchmark, and how the benchmark can be used to affect more informed purchasing, integration, configuration, and tuning decisions.

The First Benchmark: SPC Benchmark- 1 (SPC-1) is the first industry standard storage benchmark and the first standard benchmark for Storage Area Networks (SANs). SPC-1 uses a highly efficient multi-platform and multi-threaded workload to emulate the precise characteristics of sophisticated enterprise class multi-user I/O applications. The SPC-1 benchmark enables companies to rapidly produce valid performance and price/ performance results using a variety of host platforms and storage network topologies.

Selecting the SPC-1 Workload

One of the principal challenges facing the SPC in building an industry benchmark standard was segmenting the enterprise storage market in order for an I/O profile (or workload) to be selected. Another challenge was to select a profile that represents a sufficiently broad segment of the market so end-users would find results relevant and useful in evaluating products. The SPC acknowledged that providing relevant storage benchmark data covering a breadth of enterprise applications (from seismic data process to banking) would no doubt require a set of benchmarks, but the question was, how many benchmarks? And could the set of benchmarks be minimized?

The solution rested with data collection and analysis by a number of SPC member companies. The needed elements of an I/O profile were agreed upon and a variety of application classes such as back-up, OLTP, print servers, email servers, etc. were targeted for data collection. Elements of the I/O profile included: read/write ratio, I/O size, locality, re-reference probability, sequentiality, interarrival time, and so on. This process took years, but ultimately the SPC collected a body of data that facilitated the accurate segmentation and targeting of the market for benchmarking as illustrated in Figure 1. This body of data pointed to the need for a small set of benchmarks that could in fact satisfy end-users' requirements, as outlined in the mission detailed for the SPC.

The SPC-1 Workload

The most prominent group of applications that displayed a "core" of common I/O characteristics was represented by OLTP systems, database systems, or mail server applications. This "core" of applications was characterized by predominantly random I/O operations requiring queries as well as update operations and simultaneously had threads of sequential I/O processing interspersed with random I/O operations. Because the focus of SPC-1 is on the commonalities of these applications, it was necessary to develop a model that would simplify the workload to the point that it highlighted the similarities of these business segments while removing any conflicts and details that weren't central to the narrow task of performance evaluation. In support, the model used in SPC-1 has two central scaling components: Business Scaling Units and Application Storage Units.

Application Storage Units (ASUs)

The SPC-1 benchmark synthesizes a community of users running against storage that is organized into the three logically separate Application Storage Units (ASUs) as would be encountered on a real-world application. The assignment of each ASU in SPC-1 is as follows:

* The Data Store (ASU-1) holds system-level data and represents the area where the server initially stores, for example, incoming mail messages or transactions. As users read their mail, it may remain on the data store, be transferred to the user store or be deleted. Forty-five percent of the capacity of the total capacity of the bench mark is contained in ASU-1. The Data Store has four parallel I/O streams associated with it. There is a read and write stream that is uniformly distributed over the entire address space, as well as some highly localized I/O to specific areas of the ASU. Additionally, there is a sequential read stream present. The I/O intensity for ASU-1 represents 59.6% of the total SPC-1 I/O command traffic.

* The User Store (ASU-2) holds user-level data and represents the area where, for example, users store their personal email, or a transactions manager stores information persistently in a database. Forty-five percent of the capacity of the total capacity of the benchmark is contained in ASU-2. There are three parallel I/O streams associated with ASU-2 User Store. Similar to the ASU-1 Data Store, the User Store also has read write streams that are randomly distributed across the entire address space of the ASU. There are also localized I/O streams, although there are fewer of these than are present on the Data Store. The I/O intensity for ASU-2 represents 12.3% of the total SPC-1 I/O command traffic.

* The Log (ASU-3) contains the log files written by the server in the course of normal operations, providing durability in the event of system failure. Ten percent of the capacity of the total capacity of the benchmark is contained in ASU-3. This stream represents logging and other sequential write activity. The I/O intensity for ASU-3 accounts for 28.1% of the total SPC-1 I/O command traffic.

ASUs are distributed within a manufacturers' storage subsystem, as in Figure 2.

For an SPC-1 result, a Test Sponsor must report the mapping of ASUs in Logical Volumes, Addressable Storage, Configured Storage, and Physical Storage.

* Physical Storage: the formatted capacity of all storage devices that are physically present in the Tested Storage Configuration.

* Configured Storage: includes all addressable storage capacity and any other storage devices or components of storage devices necessary to implement the Addressable Storage Capacity described in Clause 2.4 (example: hot spares, parity disks, journal disks, log disks, etc.)

* Addressable Storage Capacity: represents the total storage that can be read and written by application programs on Host Systems and is directly available for use by application programs that implement this benchmark (such as the Workload Generator).

Business Scaling Units (BSUs)

BSUs represent the I/O load imposed by the application's user population against ASUs. Each BSU represents the aggregate I/O load created by a specified number of users. In the synthetic framework of the SPC-1 benchmark, one SPC-1 BSU corresponds to a community of users who collectively generate up to 50 I/Os per second. The throughput of 50 I/Os per second for one SPC-1 BSU represents a best case, in which no significant contention occurs for storage resources. The actual throughput obtained for any given number of SPC-1 BSUs depends upon the unique characteristics of each TSC. Thus, to obtain greater I/O Request Throughput, the test sponsor will increase the number of BSUs presented by the workload Generator to the TSC.

In SPC-1 test sponsors scale BSU load and ASU capacity independently, although a test sponsor will benefit in the market from driving both scaling factors to maximum.

www.storageperformance.org

Next article: Reviewing the SPC-1 system and storage configuration; and the SPC-1 Workload Generator.

Roger Reich is the founder of the Storage Performance Council and a member of the steering committee. He is also senior technical director at VERITAS Software (Mountain View, CA).