Characterization of a high-density data center.

ABSTRACT

This paper shows the results of a characterization of an 11,490 [ft.sup.2] (1067 [m.sup.2]) high-density data center, with focus on a zone with heat dissipation greater than 8 kW and up to 26 kW per frame. To gain insight into the operational health, a data center survey is conducted. The purpose of the survey is to measure and collect data on power consumption, airflow, and temperature throughout the data center. The results are used to profile and validate the data center via a computational fluid dynamics (CFD) model for detailed analysis. The methodology used in the study is similar to that of the National Center for Environmental Prediction (NCEP) data center case study (Schmidt 2004).

INTRODUCTION

High-density installations are planned based on published best practices, consultant advice, guidance from industry associations and organizations, as well as equipment manufacturer recommendations. Because of the choices available in the marketplace, every data center is different, from computer room air-conditioning (CRAC) unit selection and placement to raised-floor height and perforated panel locations. While publications such as ASHRAE's Thermal Guidelines for Data Processing Environments (ASHRAE 2004) provide a common set of guidelines for equipment placement and layout configurations, there is not a cookie cutter way to plan a high-density installation. Airflow in a high-density data center is complex and requires engineering analysis of the environment. However, the end goal is to provide the information technology (IT) equipment operating in the data center with the proper temperature and humidity at the air intake.

Datacom Equipment Power Trends and Cooling Applications (ASHRAE 2005) provides power trends at the equipment level and tells how to use the trends when making infrastructure decisions. Many leading datacom equipment manufacturers offer equipment that tracks the heat load density trend. As needs for processing and storage increase to handle application growth and the footprint decreases due to technology compaction, air cooling is becoming increasingly challenging. The questions of when liquid cooling becomes necessary and what implementation should be used (zone, frame, rack-mount drawer, or chip level) are not easily answered. The existing conditions of a data center must be studied and coupled with the IT purchasing and planning strategies to try and predict the future. The existing heat load, airflow, and temperature conditions of a high-density data center, studied in November 2005, are presented in this paper along with analysis.

DATA CENTER GENERAL INFORMATION

The data center profiled is San Diego Super Computer (SDSC) located on the University of California San Diego campus. The data center area based on the drawing is 88 x 140 ft (26.8 x 42.7 m). However, the area usable for CRAC units, power distribution panels, and IT equipment is 11,490 [ft.sup.2] (1067 [m.sup.2]). Figure 1 shows an overall plan view with the zone considered for further investigation outlined. The IT equipment installed in the outlined zone is 7039-651 IBM eServer pSeries 655 Server Model 651, IBM eServer pSeries 690, and 7045-SW4 IBM eServer pSeries High Performance Switch (HPS) Model SW4. The data center slab-to-slab height is 16 ft (4.9 m) broken down as follows: 2 ft (0.6 m) concrete subfloor to raised floor, 10 ft (3.0 m) raised floor to suspended ceiling, and 4 ft (1.2 m) above the suspended ceiling. Selected ceiling panels are removed in the hot aisles to facilitate IT equipment exhaust air to the CRAC units with return duct extensions, as shown in Figure 2. The CRAC units also have turning vanes installed.

[FIGURE 1 OMITTED]

[FIGURE 2 OMITTED]

A pseudo arrangement of hot aisle/cold aisle is used throughout the data center with varying aisle pitch (distance from the center of one cold aisle to the center of the next cold aisle), ranging from 7 to 10 raised floor panels. All panels are 2 x 2 ft (0.6 x 0.6 m). The perforated raised floor panels are estimated to be 22% open based on the diameter of each hole and the hole pattern. There are several underfloor blockages from insulated chilled-water supply and return pipes as well as from cable bundles and cable trays (see Figures 3 and 4).

MEASUREMENT TOOLS

Airflow through the perforated panels is measured with an Alnor Balometer Capture Hood. Because the capture hood obstructs the airflow from the perforated panel, the readings must be adjusted by a correction factor. However, the Alnor capture hood has a built-in back pressure compensation feature, via a flap, that accounts for the flow impedance of the capture hood. The feature is used for every measured perforated panel. Measurement accuracy based on the manufacturer's specification sheet is [+ or -]3% of the airflow reading. Cable cutouts are measured with an Extech wind vane with anemometer. Measurement accuracy based on the manufacturer's data sheet is [+ or -]2% of the velocity reading. Temperature measurements of the IT equipment air intakes are made with an Omega Model HH23 Digital Thermometer with type-T thermocouple. Measurement accuracy based on the manufacturer's product specification is [+ or -]0.1% of the temperature reading. All test equipment cited is calibrated on a yearly basis to ISO/IEC Standard 17025 (ISO/IEC 2005).

All voltage and current measurements are provided by the staff at SDSC. The amperage data by circuit supplied by SDSC is measured with a Fluke T5-600 electrical tester and Fluke 33 clamp meter. The instruments are not calibrated, but the input power to several frames is checked. A laptop with custom communications software is connected to an interface on the frame and the percent difference is less than 5%.

[FIGURE 3 OMITTED]

[FIGURE 4 OMITTED]

MEASUREMENT METHODOLOGY AND RESULTS

Power Measurements

The heat load of the data center, including the IT equipment, CRAC units, and lighting, is collected on site with the help of SDSC personnel. The IT equipment in the data center is supplied by 208 V AC, either three-phase or line-to-line, and 120 V AC line-to-neutral. The CRAC units are supplied with 480 V AC line-to-line. Table 1 shows a breakdown of the power dissipation. The CRAC units have a range of heat output that seems to vary significantly and requires further investigation. The heat output of the IT equipment is calculated from the sum of the amperages multiplied by the associated mains connection. The result is volt-amps (VA), but a power factor of 0.95 is provided by SDSC to determine the watts, as most IT equipment has active mitigation to comply with the harmonics emissions standard 61000-3-2 (ISO/IEC 2005). However, it is important to note that all the IBM p690 and p655 three-phase servers, which are a significant load in the data center and are not within the scope of 61000-3-2, include active mitigation and have a power factor very close to one. Table 2 shows a breakdown of the zone with the high-density servers. The numbers in parentheses are the pieces of IBM hardware. The maximum error in the total heat load is estimated to be [+ or -]5%.

Several p655s are measured directly at the frame through a communications interface on the bulk power system. The measurements compare favorably to the data from the power panels, although the power panels are consistently higher by less than 5%. This result is most likely due to measurement error with noncalibrated instruments and distribution losses.

Airflow Measurements and Modeling

There is a total of 12 Liebert Deluxe System/3 chilled-water CRAC units installed on the raised floor. Table 3 provides the manufacturer's specifications on each of the individual CRAC units. The sensible cooling is calculated based on a return air temperature of 73[degrees]F (23[degrees]C) dry bulb, a return air relative humidity of 37%, a supply water temperature of 41[degrees]F (5[degrees]C), and a return water temperature to the chiller of 53[degrees]F (12[degrees]C). The return air data are collected from the Liebert monitoring system and averaged, as the temperature and humidity sensors are located in various locations as shown in Table 3. The method used to measure airflow through each CRAC unit for the NCEP data center could not be used because of the return duct extensions, and there was not enough time during our study to do a rectangular duct velocity traverse via the equal area or the log Tchebycheff method (Richardson 2001). However, the results from a computational fluid dynamics (CFD) simulation tool for airflow and temperature distribution in raised-floor environments indicate that the actual airflow is within the measurement tolerance to the manufacturer-specified airflow. The CFD results will be discussed at the end of this section.

The data center has a total of 233 perforated panels. The total open area is 200 [ft.sup.2] (18.6 [m.sup.2]) based on 22% open perforated panels. Each perforated panel is measured via the Alnor capture hood with back pressure compensation, and the airflow is recorded. The total volumetric airflow from the 233 perforated panels is 99,947 cfm (2830 [m.sup.3]/min). The data center also has a total of 306 cutouts used for power and communications cable routing. The open area of each opening is measured and adjusted for the estimated cable blockage. The total open area is 92 [ft.sup.2] (8.5 [m.sup.2]) Because of time constraints, only 17 openings are measured using the anemometer with wind vane.

The data collected on the CRAC units, perforated panels, and cutouts during the study are used to build a CFD model and to compare the results to the measurements. In addition, significant underfloor blockages from cable trays and cable bundles as well as chilled-water supply and return lines are added to the model. Also, the data center is spot checked for perimeter openings in the raised-floor cavity. None are revealed, but an exhaustive review is not done. Finally, the distributed leakage area of air between perforated panels is estimated to be 0.2% based on the average width of the gaps between floor panels. Typical percent leakage area values can range from 0.1% to 0.2%, but it can be as high as 0.35% (Radmehr et al. 2005). The model is run several times with different distributed leakage values to arrive at an acceptable comparison between measured and predicted values. However, the leakage may be larger than assumed since the data center has been in existence for some time and there are areas of high static pressure, as high as 0.094 inch of water gauge (w.g.) (0.023 kPa). All in all, there is only a 1.7% difference in total cfm between the measured and simulated airflow. Figure 5 shows the measured versus modeled results on a perforated panel-by-panel basis. The average static pressure for the entire raised floor is 0.048 in. w.g. (0.012 kPa).

Initial runs of the model showed some wide excursions because of dampers that are used on some of the perforated panels. Since the measured airflow rates are available and the pressure drop usually varies as the square of the airflow rate, it is possible to refine the model. The additional airflow resistance is entered in the polynomial expression for a particular perforated panel and the model is rerun. The airflow from the perforated panels in row AA, closest to the CRAC units CCU8-12, shows the largest discrepancy between measured and modeled values, but perforated panels in adjacent rows show good correlation.

A comparison of the cutouts shows good correlation between measured and modeled openings less than 25 [in..sup.2] (161 [cm.sup.2]). The percent difference is less than 15%. The larger cutout comparison shows that the predicted volumetric airflow is significantly higher than the measured airflow. The reason for the discrepancy most likely has to do with the measurement technique. Regardless of the size of the cutout, three measurements are taken to obtain data in the shortest amount of time. Because the wind vane area is 5 [in..sup.2] (32 [cm.sup.2]), it is very difficult to obtain a representative linear airflow with three readings of a large opening, say 100 [in..sup.2] (645 [cm.sup.2]), before integrating over the area.

[FIGURE 5 OMITTED]

Temperature Measurements

Temperatures are logged at the air intakes of the frames in the outlined zone shown in Figure 1. A single temperature reading is captured for each frame at a height of 68.9 in. (1750 mm) and in accordance with ASHRAE (2004) Thermal Guidelines for Data Processing Environments guidelines of 2 in. (50 mm) in front of the covers. The temperature of the air exiting each perforated panel is measured by the capture hood and recorded. The return air to the CRAC units could not be easily measured because of the return duct extensions. Instead, the CRAC sensor data are recorded.

HIGH-DENSITY ANALYSIS

The gross density of air-conditioning capacity is 138 W/[ft.sup.2] (1485 W/[m.sup.2]) and the current heat load density is 132 W/[ft.sup.2] (1421 W/[m.sup.2]). Although this information is not particularly useful in the overall operational health evaluation, as it does not indicate airflow distribution problems, hot spots, etc., gross W/[ft.sup.2] (W/[m.sup.2]) is commonly used by real estate operation personnel in figuring data center costs. Table 4 gives a view of the total airflow in the data center with estimated accuracy.

Table 4 shows that approximately half of the airflow in the data center is from cutouts and leakage. Although there may be some benefit to cooling of IT equipment, prior studies (Schmidt and Cruz 2002) show that the cutout air is heated by the IT exhaust air before returning to the IT air intakes.

The frame power, airflow rates, and inlet temperatures for the zone outlined in Figure 1 are examined similarly to the NCEP data center profile. The zone is divided into three sections for further study. The environmental characteristics are charted in Figures 6-10, which show that the air intake temperatures are within or below the ASHRAE Class 1 recommended dry-bulb temperature range of 68[degrees]F-77[degrees]F (20[degrees]C-25[degrees]C) regardless of frame power consumption and airflow rate through the IT equipment.

Table 5 shows a correlation between average perforated panel supply airflow, cutout airflow, frame airflow, and frame temperature rise for a section and subsection. The average perforated panel supply airflow comes from the panels directly in front of the frames for this analysis, even though a cold aisle may be 4 ft (101.6 mm) wide. For comparison to the average frame airflow, the average perforated panel airflow is adjusted for the frame width of 1.25 panels. The frame airflows are as follows: 2960 cfm (84 [m.sup.3]/min) for the p655, 1100 cfm (31 [m.sup.3]/min) for the p690, and 800 cfm (23 [m.sup.3]/min) for the switch frames. The frame temperature rise is calculated based on the difference between the average temperature of the air exiting a given row of perforated panels and the aggregate return air temperature of the CRAC sensors within the vicinity of that row.

Table 5 shows that the airflow rates from the perforated panels in front of the frames are much less than the frame airflow rates. Despite the imbalance in airflow, air intake temperatures are within or below ASHRAE Class 1 as shown in Figures 6-10. If the perforated panel and cutout adjacent to a frame are combined to provide the airflow, the calculated temperature rise based on average frame power is as shown in Table 6.

The calculated temperature rise in Table 6 is higher than the actual temperature rise in Table 5 for each section. Therefore, the conclusion is the same as that for the NCEP data center--the migration of chilled air within the data center from zones of high density to zones of low density is occurring as the local chilled-airflow rates for the high-density frames are much lower than what might be considered adequate.

AIRFLOW COMPARISON

An airflow comparison is made between the actual airflow and a calculated airflow from the data center measured heat load and temperature difference. The actual airflow--based on perforated panel measurements, limited cable cutout measurements, and modeling--is 185,700 cfm (5258 [m.sup.3]/min) with an estimated accuracy of 10% derived from various simulation results. The actual airflow range is 167,130-204,270 cfm (4733-5784 [m.sup.3]/min). The average temperature difference from perforated panel temperature measurements and CRAC sensors is 59[degrees]F (15[degrees]C) with an estimated error of 10%. The total heat load accuracy is estimated to be [+ or -]5%. The calculated airflow is 177,664 cfm (5031 [m.sup.3]/min) with a range of 153,437-207,205 cfm (4345-5867 [m.sup.3]/min). There is good overlap in the actual and calculated airflow ranges; therefore, the comparison of the data is validated.

SUMMARY

The paper presents a detailed characterization of a high-density data center. On-site measurements of heat load, airflow, and temperature are measured and collected to study the data center. These parameters are used to build a CFD model and run simulations to provide detail on parameters, such as cutout and leakage airflow, that could not be captured during the study either because of time or physical constraints. The model is validated based on comparison between perforated panel measurements and the results of the model, as the total airflow percent difference is only 2%. An airflow comparison also confirms that the actual and calculated airflow are in agreement.

The high-density area of the data center with IT equipment is studied. The key IT equipment health indicator is the inlet temperature, which is within or below the ASHRAE Class 1 guideline even though the sum of the perforated panel and cutout airflow rates is up to two-thirds less than the frame airflow rates. While the local conditions do not seem adequate to satisfy and maintain the air intake temperatures of the high-powered frames, the overall data center flow rate can handle the total data center heat load. However, as more high-density equipment is installed, there is a risk of locally elevated inlet temperatures even though there may be sufficient cooling capacity. The conclusion for SDSC and NCEP data centers are similar.

ACKNOWLEDGMENTS

The author would like to thank Dr. Roger Schmidt, Dr. Hendrik Hamann, Dane Miller, and Harald Zettl for their help with collection and interpretation of the data. The characterization and paper would not have been possible without their contribution. The author also thanks the staff at SDSC, especially Mike Datte and Jeff Filliez, for their full cooperation in allowing IBM to study the data center and publish the results.

REFERENCES

ASHRAE. 2004. Thermal Guidelines for Data Processing Environments. Atlanta: American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc.

ASHRAE. 2005. Datacom Equipment Power Trends and Cooling Applications. Atlanta: American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc.

[FIGURE 6 OMITTED]

[FIGURE 7 OMITTED]

[FIGURE 8 OMITTED]

[FIGURE 9 OMITTED]

[FIGURE 10 OMITTED]

IEC. 2005. Electromagnetic Compatibility (EMC), Part 3-2: Limits--Limits for harmonic current emissions (equipment input current [left arrow] 16 A per phase). Geneva, Switzerland: International Electrotechnical Commission.

ISO/IEC. 2005. General Requirements for the Competence of Testing and Calibration Laboratories. Geneva, Switzerland: International Organization for Standardization.

Radmehr, A., R. Schmidt, K. Karki, and S. Patankar. 2005. Distributed leakage flow in raised-floor data centers. Advances in Electronic Packaging 2005, Proceedings of IPACK 2005-73273, The ASME/Pacific Rim Technical Conference and Exhibition on Integration and Packaging of MEMS, NEMS, and Electronic Systems, pp. 401-08.

Richardson, G. 2001. Traversing for accuracy in a rectangular duct. Associated Air Balance Council TAB Journal, summer 2001 Issue, pp. 20-27.

Schmidt, R. 2004. Thermal profile of a high-density data center--Methodology to thermally characterize a data center. ASHRAE Transactions 110(2):635-42.

Schmidt, R., and E. Cruz. 2002. Raised-floor computer data center: Effect on rack inlet temperatures of exiting both the hot and cold aisle. Thermomechanical Phenomena in Electronic Systems, Proceedings of the Intersociety Conference, 2002, pp. 580-94.

Joseph F. Prisco, PE

Joseph F. Prisco is a senior engineer with IBM, Rochester, MN.

Table 1. Total Data Center Power Dissipation

CRAC                     kW      kW

CCU1                      15.8
CCU2                      23.3
CCU3                       7.5
CCU4                       7.3
CCU5                       8.1
CCU6                      14.1
CCU7                      10.0
CCU8                      10.8
CCU9                      14.1
CCU10                     10.8
CCU11                     14.1
CCU12                     19.1
CRAC Power                       155.0
Lights                            15.0

Power Panels

CL1U                     426.2
U3                        55.5
RDC                      595.7
U4                       175.7
SATA                      53.1
GG                        39.5
                                1345.7
Total Power Consumption         1515.8

Table 2. Select IT Power Dissipation

IT Equipment             kW

p690 (11)                 91.4
p655 (18)                420.9
HPS (6)                   46.3

Total Power Consumption  558.6

Table 3. CRAC Unit Detail

                           Sensible
CRAC Unit Number  cfm      Cooling (kW)  Sensor Location

MHB-CCU01          16,500   142          Power Pillar 56
MHB-CCU02          16,500   142          Power Pillar 56
MHB-CCU03          12,400   102          Return Duct
                                         Extension
MHB-CCU04          12,400   102          Return Duct
                                         Extension
MHB-CCU05          12,400   102          Return Duct
                                         Extension
MHB-CCU06          16,500   142          Power Pillar 20
MHB-CCU07          16,500   142          Backside Power
                                         Pillar 54
MHB-CCU08          16,500   142          Backside Power
                                         Pillar 54
MHB-CCU09          16,500   142          Return Duct
                                         Extension
MHB-CCU010         16,500   142          Backside Power
                                         Pillar 54
MHB-CCU011         16,500   142          Return Duct
                                         Extension
MHB-CCU012         16,500   142          Power Pillar 20
Total             185,700  1584

Table 4. Data Center Airflow

Airflow                cfm      Notes              Accuracy

Total volumetric flow  185,700  Liebert            10%
                                specifications
Perforated panels       99,947  Measured            5%
Cut-outs                75,156  Calculated with    15%
                                approx. blockages
Leakage                 10,597  0.2% distributed
                                area in the model   5%

Table 5. Section Analysis of Airflow and Temperature Rise

                 Average         Average   Average
                 Perforated      Cut-Out   Frame     Average Frame
                 Panel Airflow,  Airflow,  Airflow,  Temperature
Section Number   cfm             cfm       cfm       Rise, [degrees]C

1                645             762       2420      12.9
2(1)             672             386       2077      12.0
2(2)             816             189       2960      12.4
3(1)             390             182       1100      11.4
3(2)             537             584       1025      11.7

Table 6. Calculation of Average Frame Temperature Rise

                                                     Calculated
                Average Perforated  Average Frame    Average Frame
                Panel and Cut-Out   Power            Temperature
Section Number  Airflow, cfm        Consumption, kW  Rise, [degrees]C

1               1408                19.0             23.8
2(1)            1057                17.8             29.6
2(2)            1005                19.7             34.5
3(1)             571                 8.7             26.7
3(2)            1121                 8.6             13.5