Coarse-grid CFD: the effect of grid size on data center modeling.INTRODUCTION Data centers house IT equipment in racks typically arranged in rows which face one another. Alternating cold and hot aisles are formed as this pattern is repeated across the data center. Cool ing airflow is often supplied by Computer Room Air Conditioners (CRACs) located around the perimeter of the room via a raised floor plenum and perforated floor tiles located in front of the racks. Warm rack exhaust is most often returned, un-ducted, "through the room." One alternative to the raised floor is the use of local coolers placed directly within the rows of IT racks. The local coolers considered here draw warm air from the hot aisle and supply cool air to the cold aisle. (See ASHRAE [2005a] for more details and a discussion of other cooling options.) Regard less of cooling architecture, the primary data center cooling objective is to maintain IT equipment inlet temperatures within a prescribed range, for example, that specified in ASHRAE [2004]. The secondary cooling consideration, which has been rising rapidly in importance, is energy consumption. (See US EPA [2007] for more information.) CFD analysis is now commonly used to validate a new design or assess an existing facility and there are at least three commercially-available CFD packages specifically for data center applications. While there have been great improvements in usability and solution speed, the latter due mainly to computer hardware, CFD is still sufficiently expensive and time consuming that its use for data center applications is restricted to equipment vendors, engineering consultants, and universities. Further, since plans often change between design and construction of a new facility and equipment is constantly added and removed in an existing facility, the few simulations that are performed are often out of synch with the real data center. Substantially faster cooling prediction tools, ideally approaching real time, would help drive the use of such technology down to the data center operator level where it can guide choices associated with IT equipment deployment, capacity planning, cooling system operation, etc. Some progress has been made in the area of near-real-time cooling performance prediction for data centers. VanGilder and Shrivastava [2006], VanGilder et al [2007], and Shrivastava et al [2007] used three different techniques to compute the cooling performance of racks in real (or near-real) time for simple clusters of racks arranged in two approximately equal-length rows around a common cold or hot aisle. The techniques include 1) performing CFD in advance "offline" and then computing airflow patterns in the cold aisle "on the fly" based on superposition, 2) using a CFD engine to compute the airflow and temperatures in either a cold or hot Partially Decoupled Aisle (PDA), and 3) the use of Neural Networks to directly predict cooling performance metrics for each rack. Rambo and Joshi [2005] used Proper Orthogonal Decomposition (POD) in a 2-dimensional analysis of servers inside of an equipment rack. CFD analyses were performed "offline" for many different levels of inlet velocity or server position. The POD technique was then used to create a "low dimensional description" of the flow inside the rack which could reproduce the entire flow field inside the rack with good fidelity with the full CFD model for any value of inlet velocity or server position. In a somewhat different application, Bash et al [2006] proposed a system for controlling CRACs based on physical measurements by a distributed network of sensors. The control system algorithm may be based on CFD analyses performed "offline" prior to equipment installation. To date, there is nothing in the literature on the subject of near-real-time cooling prediction for arbitrary full-scale data center applications. However, techniques have been used to model other indoor environment applications in near-real time. Multizone or network modeling is used extensively for predicting airflows, temperatures and contaminant transport; however, these models do not provide resolution of results below the room level. (See ASHRAE [2005b] for more details). Huang and Haghighat [2005] created an Integrated Zonal Model (IZM) in which a jet model was integrated into a multizone model using multiple sub-zones in a single room. Similarly, Ren and Stewart [2003] modified a popular zonal model program to handle sub-zones. Elhadidi and Khalifa [2005] used the POD technique to model the airflow and temperature distribution in a small office space. While all of these tools can predict airflow and temperature distributions fairly quickly with meaningful accuracy for simple examples, it is unclear how the techniques would be scaled to handle an arbitrary data center application. The primary limitation with the sub-zonal models is that semi-empirical correlations are used to represent different driving forces such as ventilation jets, heat sources, and warm or cold surfaces. A typical data center has many such complex and interacting jet patterns and heat sources which would be very difficult to characterize a priori. The primary limitation with the POD technique is the vast number of "offline" CFD simulations that would be required to construct a low-dimensional model general enough to handle practical applications. Mora et al [2002] and [2003] compared both fine and coarse-grid CFD using the k-[EPSILON] turbulence model and sub-zonal models to experimental measurements of indoor airflows; the focus of the latter study was a two-dimensional mechanically ventilated isothermal room. They concluded that coarse-grid CFD is better suited to predict airflows in large indoor spaces than are sub-zonal models when airflow details are desired. Further, it was found that moving from what the authors considered a "coarse" to a "fine" grid increased solution time by a factor of 13 with only a modest improvement in accuracy. Considering the research summarized above, coarse-grid CFD may represent the most practical option for substantially reducing solution time relative to traditional CFD for general data applications. This study is an initial investigation into this opportunity. METRICS USED IN THIS STUDY The cooling performance metrics used in this study are the average rack inlet temperature ([T.sub.ave]), the maximum rack inlet temperature ([T.sub.max]) and the Capture Index. The Capture Index (CI) is based solely on airflow patterns (not temperatures) and measures the effectiveness of supplying cooling air to rack inlets or scavenging heated air from rack exhausts. There are both cold and hot-aisle versions of the capture index. The cold-aisle capture index (CACI) is defined as the fraction of air ingested by a rack which originates from local cooling resources (e.g. perforated floor tiles or local coolers). The hot-aisle capture index (HACI) is defined as the fraction of air exhausted by a rack which is captured by local extracts (e.g. local coolers or return vents). Values of 100% and 0% represent the extremes of the "good" and "bad" on the cooling-performance spectrum. While rack inlet temperatures (along with relative humidity) are the ultimate cooling performance metric from the point of view of the IT equip ment, inlet temperatures may be considered only a symptom of airflow patterns within the data center. For example, imagine a cluster of racks bounding a cold aisle in a typical raised-floor environment. Now imagine an airflow pattern such that several of the racks draw most of their airflow not from the perforated tiles but from the surrounding room environment. Depending on the room environment, the rack inlet temperatures may be acceptable; however, intuitively, the cooling system is not performing well relative to the design intent. If computed, low CACI values would quantify our intuition. The capture index is easily computed in a CFD analysis by tracking the airflow which is ingested (CACI) or exhausted (HACI) by each rack. With raised-floor layouts, rack exhaust air is typically returned to the CRACs "through the room" rather than captured locally. In this case, the cooling strategy must be to ensure sufficient cooling airflow is delivered to the rack inlets and CACI is therefore the obvious capture index metric. With local cooler layouts, it is reasonable to compute either or both CACI and HACI. However, in this case, a "room neutral" cooling strategy may be pursued in which rack exhaust air is captured before it escapes the hot aisle thereby leaving the rest of the data center at a consistent "room" temperature regardless of airflow patterns near the rack inlets. Thus, HACI is generally the preferred capture index metric when local coolers are used. For more details on the Capture Index, see VanGilder and Shrivastava [2007]. The average inlet temperature, maximum inlet tempera ture, and (either cold or hot-aisle) capture index are computed for each rack (of the more than 500 total racks) at several different grid sizes. In order to consolidate this data and conveniently assess the effect of grid size on solution time and accuracy, for purposes of this study, "accuracy" is defined relative to the percentage of difference between the computed cooling performance metric at any grid size and the same metric computed at a benchmark grid size. For both cold and hot-aisle capture indices, the accuracy of CI at rack i is defined as [Accuracy.sub.CI] = 1 - |[CI.sup.i] - [CI.sub.benchmark.sup.i]|/[CI.sub.benchmark.sup.i]. (1) Similarly, the accuracy for both average and maximum rack inlet temperatures is defined as: [Accuracy.sub.T] = 1 - |[T.sup.i] - [T.sub.benchmark.sup.i]|/[DELTA] [T.sub.ref] (2) where [DELTA][T.sub.ref] is a reference temperature-difference scale, always taken as 20[degrees]F (11[degrees]C) in this study. Note, the denominator in Equation 2 requires a temperature difference in order for the accuracy definition to be independent of the choice of temperature units. An alternative reasonable choice for [DELTA][T.sub.ref] would be the typical maximum temperature difference for each layout studied. The fixed value was selected here because the typical temperature difference across many racks, CRACs, and local coolers is about 20[degrees]F. A fixed value also allows for consistent comparison across all cases and the relatively small value selected tends to exaggerate differences relative to the benchmark. Finally, note that with a fixed [DELTA][T.sub.ref], |[T.sup.i]-[T.sub.benchmark.sup.i]| values of 0[degrees]F and 20[degrees]F imply an accuracy of 100% and 0% respectively with a linear variation between these end points. A difference of more that 20[degrees]F with the benchmark produces a negative accuracy. SCOPE OF APPLICATIONS AND CFD MODELING CONSIDERATIONS The scope of this study is restricted to quantifying the effect of grid size on accuracy and solution time for typical data center applications. A practical assessment is made on the basis of metrics including rack inlet temperatures and capture indices (CI) which are often the key products of a data center simulation. Ten data center layouts, eight of which are based on actual facilities, are used in an attempt to cover the velocity and temperature (absolute value and gradient) ranges found in typical facilities. The conclusions of this study are valid for the specific (commercially-available) CFD code used here and, possibly, others utilizing similar physical and numerical models. For present purposes, accuracy is defined relative to a benchmark grid size as discussed above; it is not the absolute accuracy of the CFD analysis relative to the performance of an actual data center. Regardless, to have practical value, CFD analyses should be verified and validated (ASHRAE 2005b). Verification is the process by which the CFD code capabilities are assessed relative to the intended application while validation refers to the combined capability of the CFD code and user to accurately simulate the application and is typically demonstrated by comparing simulations to experimental data for similar, representative applications. The commercially-available CFD software used in this study was developed specifically for the purpose of modeling airflow in buildings and includes models of physical phenomenon relevant to data center modeling. The code uses a first-order differencing scheme, the SIMPLE (Patankar [1980]) pressure-velocity coupling algorithm, and a choice of turbulence models including the proprietary version of the k-[EPSILON] turbulence model selected for this study. The convergence criterion is based on the sum of the absolute error over all grid cells for each primitive variable, for example, x-velocity, temperature, etc. The solution is considered converged when the error of each primitive variable is less than half of one percent of the appropriate characteristic scale. For example, the characteristic scale for temperature is the total heat dissipation specified in the model. While a validation study was not repeated exclusively for the present study, several comparisons of experimental data to CFD simulations (of data center applications using the same CFD software and modeling techniques used here) are available in the literature (Patel et al. 2001; VanGilder and Schmidt 2005; Shrivastava et al. 2006; Iyengar et al. 2007). (Also see Rambo, J. and Joshi [2007] for a recent literature review of data center studies which include experimental measurements.) A Cartesian grid system is used with effort to make the grid as uniform (cell aspect ratios close to one) as possible. This provides a high-quality grid and allows conclusions to be based on grid size rather than on grid distribution or cell shape. (See ASHRAE [2005b] for more information on grid distribution and quality.) Both "nominal" and "average" grid sizes are discussed. The former is based on the "maximum overall grid size" allowed by the CFD code; since there will also be, in general, grid cells aligned with the edges of all objects, the nominal size is always greater than or equal to the average size. The average size is the length every side of every grid cell would have if the entire volume were broken up into uniform cuboidal cells based on the actual number of total cells. For cases in which the maximum grid spacing is naturally aligned with all equipment sizes and locations, nominal and average grid sizes are equal. For the primary focus of this study, which includes the analysis of ten realistic data center layouts, a structured Cartesian grid system is used; grid lines extend all the way across the solution domain in each direction. The use of a "localized" grid is also studied using one example layout. With a localized grid, the grid density can be controlled independently from the global grid and the (localized) grid lines do not extend across the entire domain. Raised-floor plenums are not explicitly modeled and leakage airflow is ignored; however, non-uniform tile airflow distributions are prescribed in cases for which this information was available from previous CFD studies or experimental measurements. Racks, CRACs, and local coolers are modeled as simple "black boxes" with airflow spread uniformly over the entire front and rear faces of the equipment and, for clarity, rack airflow and power are restricted to three different combinations. Other equipment, including Power Distribution Units (PDUs), Uninterruptable Power Supplies (UPSs), Cooling Distribution Units (CDUs), etc. are modeled simply as zero-airflow and zero-power blocks. In cases, where equipment sizes differ by less than two inches, dimensions are modified to obtain a common size. The geometric simplicity achieved by many of the above assumptions makes it possible to define the reasonably uniform grid distributions consistent with the scope of this study. Solution time is reported based on 500 iterations of the CFD solver. While most cases required fewer or more iterations to converge, 500 iterations is sufficient to reach a reason able level of convergence after some manual optimizing of solution-control parameters. Furthermore, by focusing on the time per iteration rather than total solution time, timings could be conducted consistently and economically on one machine while many other machines could be used (simultaneously) to actually solve cases to the point of convergence. Solution time is reported based on a Pentium 4 3.0 GHz machine with 1 GB of RAM. Four of the 60 core layouts, each larger than 4 million grid cells, were too large to be analyzed on this machine so reported times were extrapolated. Furthermore, 3 of the 60 core layouts required more than 2GB of RAM and were, there fore, solved on a machine with a 64-bit OS which could accommodate the required RAM. GRID INDEPENDENCE STUDY Although this entire paper may be considered a grid independence study, in this section we start with the two simple layouts shown in Figure 1. CFD analyses of these two layouts include features common to the ten realistic layouts discussed below including the magnitude of temperature and velocity absolute values and gradients. To achieve grid independence, the grid must be sufficiently fine to resolve these gradients. In practice, a non-uniform grid is often used to efficiently locate grid where it is needed; a uniform grid is used here to facilitate a consistent definition of grid size among layouts while avoiding the time and effort required to tailor a non-uniform grid to each layout. [FIGURE 1 OMITTED] The layouts of Figure 1 are small and geometrically regular in that all object boundaries are located only at multiples of six inches in all three coordinate directions. Consequently, it is practical to consider grid sizes down to one inch and several grid sizes (1, 2, 3, and 6 inches) can be created with rigorously uniform (aspect ratio equal to one) grid distributions so as to avoid effects due to grid distribution rather than grid size. Analyses of these simple layouts assist in selecting the "benchmark grid size" against which all coarser grid results are compared for the realistic data center layouts. In both cases, the ceiling height is 12 ft, horizontal dimensions may be scaled based on the 2 ft x 2 ft floor tiles, and the total cooling airflow is relatively low in order to provide meaningful CI variations. In the raised floor layout, the plenum is not modeled; however, a non-uniform distribution of 60[degrees]F airflow is prescribed as shown in Figure 1a. The 3000 cfm of total cooling airflow leaves the room uniformly over the ceiling area. Since this is a grouping around a cold aisle, CACI is the appropriate version of capture index. In the local cooler layout, Figure 1b, two local coolers attempt to capture the hot rack exhaust from the hot aisle and HACI is the appropriate version of capture index. The coolers supply airflow at 68[degrees]F. Figure 2 shows the effect of grid size on the capture index for each rack and the temperature distribution along a vertical line (gradient) centered on the inlet of rack A1. Although seven grid sizes were analyzed for each layout, for clarity, only a limited set of results are shown in Figures 2aii and 2bii. Note that in both the raised-floor and local-cooler environments, there are sharp temperature gradients at the top of the rack (6.5 ft height) where there is strong mixing of airflow from the cold and hot aisles. Otherwise, the temperature gradients at the inlet of the rack are generally steeper in the raised-floor environment - where there is generally more pronounced vertical temperature stratification. Figures 2aii and 2bii also show the total number of grid cells in each direction. The finest grids (1 inch) for the raised floor and local cooler layouts contain 3.1 and 2.9 million grid cells respectively. [FIGURE 2 OMITTED] Figures 2ai and 2bi show that below a grid size of about 16 inches, there is a little absolute change in CI although values do not appear to asymptotically approach a "true" value as the grid size is reduced. The temperature distribution at the inlet of rack A1 does; however, appear to exhibit a well-behaved approach to grid independence. Still, even the very fine 1-inch grid size is not quite truly grid independent. The "better" convergence of the temperature distribution relative to CI's may be explained as follows. Temperature (at any point in the domain) is a primitive variable computed directly by CFD while capture index (like average rack inlet temperature) is a derived quantity computed by integrating primitive variables over the rack or cooler inlet faces. While the primitive variables at any point do converge to a "true" value at every point as grid size is reduced, the convergence of derived quantities is more complex; the number of primitive variables included in the derived quantities changes with grid size and the primitive variables may converge at different rates at different points over the rack or cooler inlet faces. With Figure 2 in mind, the 2-inch grid is selected as the benchmark level against which results of coarser-grid analyses are compared for the realistic layouts discussed below. Two-inch grid results are fairly grid independent for practical purposes. Furthermore, for several of the layouts considered, a 2-inch grid is the finest grid which is practical from a computing-hardware and solution-time perspective. EFFECT OF GRID SIZE ON ACCURACY AND SOLUTION TIME FOR TEN REALISTIC DATA CENTER LAYOUTS The ten layouts at the core of this study are summarized in Figure 3 and Table 1. All layouts feature alternating cold and hot aisles; hot aisles are labeled in Figure 3 for cases in which there are no perforated tiles to make the distinction obvious. Further more, a 2 ft x 2 ft floor grid is shown for all layouts, even for those which do not have a raised floor, so that horizontal dimensions may be scaled. In all layouts, CRACs and local coolers supply airflow at 60[degrees]F and 68[degrees]F respectively. Table 1 shows that the power density, based on the total floor area shown in Figure 4, ranges from 75 to 185 W/[ft.sup.2] which is fairly representative of newer facilities. Layouts a through e are cooled by CRACs placed around the perimeter of each room. Layouts a through d utilize a raised floor plenum with airflow supplied to the racks through perforated floor tiles. Layout e uses up flow CRACs which draw warm return air near the (hard) floor and supply cool air horizontally down the cold aisles from near the top of the units. Layouts f through i are cooled exclusively by local coolers while layout j is a hybrid utilizing a raised floor with CRACs and local coolers. Layouts a and g are hypothetical layouts which represent the typical generic cold/hot aisle layout for raised floor and local-cooler architectures respectively. All layouts except a and g are based on actual facilities which have been previously modeled in detail with CFD. For clarity, only three different combinations of rack power and airflow are used; effort was made to match these levels as closely as possible to the original facility data. All local coolers are 1 ft wide with 2900 cfm of airflow. While detailed tile airflow distributions are not shown, the minimum, maximum, and average per-tile values are summarized in Table 1. Layouts c and d include tiles through which airflow travels from the room back down into the floor plenum. In all cases, the global air ratio, the ratio of total cooling to total rack airflow, is fairly low in order to obtain non-trivial CIs (less than 100%) and inlet temperatures (above the supply temperature) for the majority of racks. Only one capture index, either CACI or HACI, is computed for each rack. For racks bounding a common hot aisle with local coolers, HACI is computed. In all other cases, CACI is computed. [FIGURE 3 OMITTED]
Table 1. Summary Data for Ten Data Center Layouts
Layout Total Power Ceiling Number Total Total
Floor Density Height of CRAC Local
Area (W/f[t.sup.2]) (ft) Racks Airflow Cooler
(f[t.sup.2]) Airflow Airflow
(cfm) (cfm)
A 896 161 12 40 16,800 N/A
B 3,588 114 12 85 65,400 N/A
C 1,081 130 8.5 32 23,800 N/A
D 1,202 75 8 42 18,500 N/A
E 1,802 78 13 70 30,000 N/A
F 622 180 12 17 N/A 14,500
G 840 171 12 40 N/A 23,200
H 1,441 185 12 47 N/A 31,900
I 1,040 149 12 24 N/A 17,400
J 2,856 180 12 98 20,400 37,700
Tile Airflow (cfm)
Layout Min. Max. Ave. Global Air Ratio
A 420 420 420 0.89
B 0 600 499 1.26
C -900 2,700 850 1.30
D -100 2,200 578 1.31
E N/A N/A N/A 1.34
F N/A N/A N/A 1.14
G N/A N/A N/A 1.22
H N/A N/A N/A 1.01
I N/A N/A N/A 0.98
J 400 400 400 0.95
Each of the 10 layouts was analyzed at 6 different grid levels for a total of 60 scenarios. As discussed above, stated solution times are relative to the same Pentium 4 3.0 GHz machine, however, the scenarios were actually solved to the point of convergence using several machines. Total computer time for all 60 scenarios was approximately 22 days. The number of iterations required to achieve convergence varied greatly from scenario to scenario (even for different grid sizes for the same layout) and manual adjustments of convergence parameters such as "false time steps" were often required to improve convergence speed. However, most scenarios can be made to achieve reason able convergence within about 500 iterations. Figure 4 shows the average effect of grid size on solution time and accuracy relative to the 2-inch benchmark solution over all scenarios studied. Data points are plotted at each of the six grid sizes analyzed though, for clarity, only five grid sizes are labeled. There are two plots for each of the four metrics [T.sub.ave], [T.sub.max], CACI, and HACI. The "average" plot is simply the aver age difference in accuracy of each of the four metrics over all racks as computed by Equation 1 or 2. For example, at an aver age grid size of 5.8 inches, the solution time for 500 iterations is about 25 minutes and maximum temperature predictions are, on average, about 91% accurate. The "percentage of racks within 80%" plot indicates the percentage of racks which are within 80% accuracy as computed by equation 1 or 2 for each metric. Continuing the same example, at an average grid size of 5.8 inches, 94% of all racks for which CACI was computed are within 80% accuracy. Note that solution time is plotted on a log scale; very small improvements in accuracy are associated with significant increases in solution time. [FIGURE 4 OMITTED] Taking a broad view of Figure 4, we get our best return in accuracy for computing time at a grid size of about 10 inches with an average solution time of only 4 minutes. On average, results are 89% accurate and 87% of all racks are within 80% accuracy for all metrics. Of course, the best compromise between accuracy and solution time depends upon accuracy goals. But, for an initial layout design of a data center, for which only vague equipment details and rough summary data are known, there is little justification for a finer grid. Moving up to the approximately 6 inch average grid, results are better - 92% average accuracy and 92% of the racks are within 80% accuracy for all metrics - but solution time has increased to 25 minutes. This is substantially slower but still excellent for most data center applications. As the grid is made smaller than about 6 inches, there is some modest additional improvement in accuracy but at a steep cost in solution time which increases by about a factor of 50 as the grid size is reduced from 6 to 2 inches. In general, solution time increases approximately as [N.sup.1.2] where N is the total number of grid cells. And, since N is proportional to 1/[[DELTA]x.sup.3] (for a perfectly uniform grid in three dimensions) where [DELTA]x is the cell size, solution time is approximately proportional to 1/[DELTA][x.sup.3.6]. Figure 5 shows similar plots broken down by individual layout. Most layouts follow the overall average trends and the above comments again apply. Notably, the average accuracy of [T.sub.max] is low for several layouts for grid sizes smaller than about 6 and especially 10 inches. The maximum inlet temperature for any rack may occur at any location over the rack inlet; it is therefore, reasonable to expect that [T.sub.max] is more sensitive to modeling changes than the rack-averaged metrics. Also, notable is the lack of any clear trends associated with raised floor versus local cooler layouts, global air ratio, etc. [FIGURE 5 OMITTED] All of the data shown in Figures 4 and 5 are tabulated in Appendix A along with an additional metric, the "minimum accuracy", for each scenario. The minimum accuracy shows that while the cooling performance of most racks may be computed with acceptable accuracy, there may be one or more racks for which predictions are poor. A closer look at the underlying data shows that most such cases are associated with very small benchmark CACI or HACI predictions. For example, consider the minimum CACI accuracy of -127% computed for case c with a nominal grid size of 6 inches. Although, not shown in the summary data, this mini mum accuracy value is associated with a rack which has a 2-inch-grid benchmark solution CACI value of 7.5% and a 6-inch-grid value of 24.5%. While this combination does yield a minimum accuracy of -127% per Equation 1, in either case, the computed value is intolerably low and the proposed data center layout would need modification. It is only at larger CI values, say 75% or greater, for which there would be a risk of mischaracterizing an unacceptable design as acceptable or vice versa. In summary, under the present assumptions of CFD solution type, modeling detail, and a fairly uniform grid distribution, there is little accuracy benefit but large solution time penalty for average grid sizes smaller than 6 to 10 inches. And, since a 6 inch grid naturally maps to the size and placement of most objects in a typical data center, this choice will often yield a marginally more uniform and higher-quality grid. LOCALIZED GRID EXAMPLE To this point, analyses have been restricted to fairly uniform grid distributions. Although grid quality may be somewhat reduced, a non-uniform grid distribution may improve calculation efficiency by concentrating grid cells only where they are most needed-generally in regions of sharp gradients - and using a fairly coarse grid in other areas. (See ASHRAE [2005b] for more information.) For example, consider a raised-floor data center with CRACs located around the perimeter of the room (e.g., layouts a-d of Figure 3) with an 18 ft high ceiling. It is unlikely, that there are sharp temperature or velocity gradients in the empty space above the racks. As this volume (about 11 ft by floor plan area) is potentially large, a slight increase in cell size in this area can create an enormous savings in solution time. In this section, we take this line of reasoning one step farther and consider a "localized", also referred to as a "block-structured", grid as shown in Figure 6. The grid is still Cartesian but is unstructured in that (localized) grid lines no longer extend through the entire computational domain. A detailed study of non-uniform or localized grid distributions is not provided and the scope of this section is restricted to considering only the hypothetical layout shown in Figure 6. There are two identical rows of racks and local coolers oriented perpendicularly to one another in a small room of overall dimensions of 20 ft by 25 ft with a 16 ft ceiling. This layout was selected precisely because it is not a "good" layout; the local coolers in the left-hand row create complex airflow patterns around the racks of the right-hand row. For the present example, we focus on predicting the capturing of hot rack exhaust from the 5 racks by the two coolers in the right-hand row. Consequently, a localized grid region 3 ft wide with length and height equal to the row of equipment is defined as shown. Rigorously speaking, we should also consider refining the grid down stream of each cooler as these regions represent areas of steep velocity gradients - although only the upper cooler in the left-hand row directly impacts the hot-exhaust capture which is our present focus. [FIGURE 6 OMITTED] Every combination of nominal global and local grid sizes of 3, 6, 12, 18, and 24 inches for which the global grid size was greater than or equal to the local grid size was analyzed. Figure 7 shows the minimum (out of the 5 racks) HACI accuracy, again computed per Equation 1, and the total solution time for all scenarios. The local grid size clearly has much more effect on the results than does the global grid size; in fact, for a fixed local grid size, the global grid size makes very little difference. Otherwise, earlier conclusions from the uniform-grid study apply now to the local values; a local grid cell size of about 6 or even 12 inches (even with a global grid size as large as 24 inches) provides the best compromise between solution time and accuracy. The 6-inch-local/24-inch-global grid is 87% accurate relative to the (uniform) 3-inch benchmark grid and requires a solution time of only 1.4 minutes per 500 iterations. By comparison the use of a uniform 6-inch grid (6-inch-local/6-inch-global) improves accuracy to 92% but solution time increases nearly 7 times to 9.6 min. [FIGURE 7 OMITTED] While more work is required before results can be generalized, this example suggests that the use of localized grid has great potential in reducing solution time relative to a uniform grid distribution and a fixed accuracy goal. Furthermore, it may be possible to concentrate grid cells based primarily on calculation goal, e.g. in the hot aisles for HACI predictions or in the cold aisles for CACI and inlet temperature predictions, while using a very coarse grid elsewhere. SUMMARY AND CONCLUSIONS The effect of grid size on relative accuracy and solution time was studied with a commercially available CFD package using a Cartesian grid, a first-order differencing scheme, the SIMPLE pressure-velocity coupling algorithm, and the k-e turbulence model. The conclusions of this study are valid for the CFD code used here and, possibly, others utilizing similar physical and numerical models. Ten data center layouts, eight of which were based on actual facilities, were analyzed at six grid levels each. Effort was made to make the grid distribution as uniform (aspect ratio close to one) as possible. The layouts included four traditional raised-floor layouts, one solid-floor layout with perimeter up-flow CRACs, four local-cooler layouts, and one hybrid layout which included a raised floor and local coolers. All equipment was modeled as simple black boxes with airflow spread uniformly across rack, CRAC, and local cooler faces. Additional analyses included a detailed grid-independence study of two simple layouts and a study of one layout in which the grid size at the rear of the racks was varied independently from the surrounding global grid. Solution times are based on a Pentium 4 3.0 GHz machine. The main findings are: Results are not rigorously grid independent even for a grid size as small as 1 inch, although, for practical purposes, results do not change significantly below a grid size of about 6 inches. Relative to a 2-inch-grid benchmark, 6 inch grid results are, on average, 92% accurate while 92% of all racks are within 80% accuracy. Average solution time is 23 minutes. Relative to a 2-inch-grid benchmark, 10 inch grid results are, on average, 89% accurate while 87% of all racks are within 80% accuracy. Average solution time is 4 minutes. Solution time increases approximately as [N.sup.1.2] where N is the total number of grid cells. And, since N is proportional to 1 / [[DELTA]x.sup.3] (for a perfectly uniform grid) where [DELTA]x is the cell size, solution time is approximately proportional to 1 / [DELTA][x.sup.3.6]. Consistent with this observation, solution time increases by a factor of about 50 as grid size is reduced from 6 to 2 inches. For the localized grid example, the 6-inch-local/24-inch-global grid results are 87% accurate and require a solution time of 1.4 minutes relative to 135 minutes for the 3-inch-grid benchmark. While the best grid choice depends, among other factors, on the goals of the analysis and the completeness/quality of available input data, an average grid size of 6-10 inches may represent the best trade-off between accuracy and solution time for typical data center simulations. Further, by using localized grid regions in areas of primary interest (i.e., near the racks), it may be possible to use a grid size as large as 24 inches or even larger elsewhere. More research is needed, particularly in the areas of non-uniform grid distributions, alternative CFD solution algorithms and grid types, and the effect of the fidelity with which the CFD model represents reality. NOMENCLATURE CI = capture index CACI = cold aisle capture index HACI = cold aisle capture index N = total number of grid cells [T.sub.ave] = average rack inlet temperature [T.sub.max] = maximum rack inlet temperature [DELTA]x = grid cell size REFERENCES ASHRAE. 2004. Thermal guidelines for data processing environments. Atlanta: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc. ASHRAE. 2005a. Design Considerations for Datacom Equipment Centers. Atlanta: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc. ASHRAE. 2005b. Fundamentals. Chapter 34. Atlanta: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc. Bash, C.E., Patel, C.D., and Sharma, R.K. 2006. Dynamic thermal management of air cooled data centers. Proceedings of Intersociety Conference on Thermal and Thermomechanical Phenomena and Emerging Technologies in Electronic Systems (ITherm), San Diego, California, May 30-June 2. Elhadidi, B. and Khalifa, H.E. 2005. Application of proper orthogonal decomposition to indoor airflows. ASHRAE Transactions, Vol. 111, Part 1, pp. 625-634. Huang, H. and Haghighat, F. 2005. An integrated zonal model for predicting indoor airflow, temperature, and VOC distribution. ASHRAE Transactions, Vol. 111, Part 1, pp. 601-611. Iyengar, M., Schmidt, R.R., Hamann, H., VanGilder, J.W. 2007. Comparison between numerical and experimental temperature distributions in a small data center test cell. Proceedings of Pacific Rim ASME International Electronic Packaging Technical Conference and Exhibition (InterPack), Vancouver, British Columbia, Canada, July 8-12. Mora, L., Gadgil, A.J., Wurtz, E., and Inard, C. 2002. Com paring zonal and CFD model predications of indoor air flows under mixed convection conditions to experimental data. Proceedings of the Third European Conference on Energy Performance and indoor Climate in Buildings (EPIC), Lyon, France, October 23-26. Mora, L., Gadgil, A.J., Wurtz, E. 2003. Comparing zonal and CFD model predictions of isothermal indoor airflows to experimental data. Indoor Air 13 (2), pp. 77-85. Patankar, S.V. 1980. Numerical heat transfer and fluid flow. Washington, D.C.: Hemisphere Publishing Corporation. Patel, C., Bash, C., and Belady, C. 2001. Computational fluid dynamics modeling of high compute density data centers to assure system inlet air specifications. Proceedings of IPACK' 01, July 8-13, Kauai, Hawaii, USA. Rambo, J. and Joshi, Y., 2005. Reduced order modeling of steady turbulent flows using the POD. Proceedings of ASME Summer Heat Transfer Conference, San Fran cisco, California, July 17-22. Rambo, J. and Joshi, Y., 2007. Modeling of Data Center Air flow and Heat Transfer: State of the Art and Future Trends. Distributed and Parallel Databases, Volume 21, Numbers 2-3 / June, 2007, pp. 193-225. Ren, Z. and Stewart, J. 2003. Simulating air flow and temperature distribution inside buildings using a modified version of COMIS with sub-zonal divisions. Energy and Buildings, Vol. 35, Issue 3, March 2003, pp. 257-271. Shrivastava, S., Iyengar, M., Sammakia, B., Schmidt, R., and VanGilder, J.W. 2006. Experimental-numerical comparison for a high-density data center: hot spot heat fluxes in excess of 500 W/[ft.sup.2]. Proceedings of the Inter Society Conference on Thermal Phenomena (ITherm), San Diego. Shrivastava, S.K., VanGilder, J.W. and Sammakia, B.G. 2007. Use of artificial neural network in data center cooling prediction. Proceedings of Pacific Rim ASME International Electronic Packaging Technical Conference and Exhibition (InterPack), Vancouver, British Columbia, Canada, July 8-12. U.S. Environmental Protection Agency ENERGY STAR Program. 2007. Report to Congress on Server and Data Center Energy Efficiency. Public Law 109-431. http://www.energystar.gov/ia/partners/prod_development/downloads/EPA_Datacenter_Report_Congress_Final1.pdf VanGilder, J.W. and Schmidt, R. 2005. Airflow Uniformity Through Perforated Tiles in a Raised Floor Data Center, Proceedings of IPACK' 05, Jul 17-22, San Francisco, USA. VanGilder, J.W. and Shrivastava, S.K 2006. Real-time pre diction of rack-cooling performance. ASHRAE Transactions, Vol. 112, Part 2, pp. 151-162. VanGilder, J.W. and Shrivastava, S.K. 2007. Capture index: an airflow-based rack cooling performance metric. ASHRAE Transactions, Vol. 113, Part 1, pp. 126-136. VanGilder J.W., Zhang, X., and Shrivastava, S.K. 2007. Partially decoupled aisle method for estimating rack cooling performance in near-real-time. Proceedings of Pacific Rim ASME International Electronic Packaging Technical Conference and Exhibition (InterPack), Van couver, British Columbia, Canada, July 8-12.
Appendix-Addition Data for Ten Layouts Studied
Grid
Layout Nominal Actual Total Cells Time Per 500
Size (in) Average Iterations (min) *
Size (in)
24 17.8 3,276 1.1
18 13.8 7,056 1.1
12 11.0 14,144 1.7
A 6 6.0 86,016 7.5
3 3.0 688,128 85.9
2 2.0 2,322,432 551.7
24 17.2 19,278 3.0
18 12.9 45,981 5.6
12 10.3 90,090 11.3
B 6 5.6 569,184 70.6
3 3.0 3,714,240 808.9
2 2.0 12,494,016 3865.0
24 15.5 4,560 1.3
18 12.5 8,680 1.4
C 12 10.0 17,199 1.9
6 5.8 88,111 8.3
3 3.0 637,602 71.3
2 2.0 2,141,439 440.8
24 15.2 5,890 1.1
18 12.7 9,996 1.2
12 10.0 20,709 2.0
D 6 5.8 106,656 9.8
3 3.0 780,288 85.0
2 2.0 2,639,952 559.2
24 15.2 14,664 2.2
18 11.7 31,800 3.1
12 10.0 52,080 4.9
E 6 5.6 296,604 31.2
3 2.8 2,333,610 500.0
2 2.0 6,702,360 1983.1
24 15.0 4,480 1.2
18 13.3 6,426 1.1
12 9.8 15,960 2.0
F 6 5.8 76,700 11.0
3 3.0 583,982 132.3
2 2.0 1,938,150 773.3
24 15.5 4,704 1.3
18 14.2 6,048 1.3
12 11.4 11,700 1.9
G 6 6.0 80,640 14.1
3 3.0 645,120 155.2
2 2.0 2,177,280 778.3
24 13.3 14,455 2.6
18 12.0 19,980 3.0
12 9.3 43,056 6.0
H 6 5.6 197,736 39.7
3 2.9 1,337,952 308.6
2 2.0 4,440,744 1163.7
24 17.7 3,920 1.4
18 14.5 7,056 1.2
12 10.9 16,653 2.2
I 6 6.0 99,840 16.0
3 3.0 798,720 240.1
2 2.0 2,695,680 557.4
24 16.4 13,986 2.9
18 12.8 29,610 4.1
12 10.6 51,376 6.8
J 6 6.0 288,000 38.5
3 3.0 2,285,568 454.5
2 2.0 7,713,792 2,018.7
24 15.8 8,359 1.7
18 13.1 15,891 2.1
Average/Summary 12 10.3 31,288 3.8
6 5.8 177,943 23.1
3 3.0 1,279,960 265.3
2 2.0 4,172,450 1,185.8
Accuracy Relative to 2-Inch-Grid Bench mark
Average Accuracy
[T.sub.ave] [T.sub.max] CACI HACI
88% 66% 91% N/A
88% 79% 90% N/A
89% 86% 91% N/A
A 89% 90% 91% N/A
92% 91% 92% N/A
100% 100% 100% N/A
98% 74% 99% N/A
98% 69% 99% N/A
99% 92% 99% N/A
B 99% 97% 99% N/A
100% 99% 100% N/A
100% 100% 100% N/A
80% 58% 55% N/A
89% 73% 77% N/A
C 90% 79% 74% N/A
94% 89% 86% N/A
98% 96% 97% N/A
100% 100% 100% N/A
92% 88% 92% N/A
92% 86% 92% N/A
95% 89% 95% N/A
D 98% 92% 97% N/A
99% 97% 99% N/A
100% 100% 100% N/A
90% 88% 91% N/A
91% 91% 93% N/A
92% 92% 94% N/A
E 95% 94% 96% N/A
96% 95% 97% N/A
100% 100% 100% N/A
92% 86% N/A 89%
94% 87% N/A 88%
95% 90% N/A 95%
F 96% 91% N/A 93%
98% 96% N/A 98%
100% 100% N/A 100%
88% 76% N/A 86%
88% 75% N/A 86%
87% 73% N/A 83%
G 93% 86% N/A 83%
98% 95% N/A 96%
100% 100% N/A 100%
94% 84% N/A 89%
94% 86% N/A 89%
95% 86% N/A 90%
H 96% 90% N/A 94%
97% 93% N/A 95%
100% 100% N/A 100%
94% 85% N/A 88%
92% 82% N/A 87%
90% 83% N/A 91%
I 90% 85% N/A 93%
91% 87% N/A 93%
100% 100% N/A 100%
92% 83% 84% 85%
94% 85% 86% 83%
94% 88% 86% 87%
J 96% 92% 91% 93%
98% 96% 97% 98%
100% 100% 100% 100%
91% 78% 85% 87%
92% 81% 90% 87%
Average/Summary 92% 85% 91% 89%
94% 91% 94% 91%
97% 94% 97% 96%
100% 100% 100% 100%
Accuracy Relative to 2-Inch-Grid Bench mark
Minimum Accuracy
[T.sub.ave] [T.sub.max] CACI HACI
71% 33% 74% N/A
70% 47% 73% N/A
70% 65% 73% N/A
A 64% 46% 67% N/A
68% 74% 69% N/A
100% 100% 100% N/A
92% 35% 94% N/A
95% 27% 96% N/A
96% 61% 97% N/A
B 98% 87% 98% N/A
99% 95% 99% N/A
100% 100% 100% N/A
-10% -42% -554% N/A
36% 17% -237% N/A
C 35% 17% -300% N/A
72% 65% -127% N/A
93% 83% 64% N/A
100% 100% 100% N/A
61% 56% 61% N/A
75% 61% 71% N/A
81% 69% 80% N/A
D 91% 62% 88% N/A
97% 84% 96% N/A
100% 100% 100% N/A
75% 32% 69% N/A
78% 59% 82% N/A
79% 62% 86% N/A
E 85% 80% 91% N/A
87% 78% 90% N/A
100% 100% 100% N/A
79% 57% N/A 69%
83% 62% N/A 72%
85% 69% N/A 85%
F 89% 72% N/A 82%
95% 89% N/A 91%
100% 100% N/A 100%
66% 31% N/A 21%
65% 33% N/A 20%
65% 34% N/A 20%
G 73% 57% N/A 6%
92% 72% N/A 72%
100% 100% N/A 100%
80% 52% N/A 38%
79% 57% N/A 36%
76% 56% N/A 44%
H 87% 69% N/A 61%
88% 72% N/A 69%
100% 100% N/A 100%
72% 51% N/A 29%
69% 32% N/A 27%
70% 37% N/A 41%
I 68% 56% N/A 70%
76% 70% N/A 72%
100% 100% N/A 100%
59% 36% 23% -52%
70% 33% 41% -40%
70% 29% 37% -33%
J 78% 54% 57% 29%
86% 75% 85% 91%
100% 100% 100% 100%
-10% -42% -554% -52%
36% 17% -237% -40%
Average/Summary 35% 17% -300% -33%
64% 46% -127% 6%
68% 70% 64% 69%
100% 100% 100% 100%
Accuracy Relative to 2-Inch-Grid Bench mark
Percentage of Racks with in 80% Accuracy
[T.sub.ave] [T.sub.max] CACI HACI
80% 25% 88% N/A
83% 55% 90% N/A
83% 85% 90% N/A
A 80% 93% 85% N/A
88% 98% 90% N/A
100% 100% 100% N/A
100% 40% 100% N/A
100% 32% 100% N/A
100% 94% 100% N/A
B 100% 100% 100% N/A
100% 100% 100% N/A
100% 100% 100% N/A
66% 38% 75% N/A
78% 41% 81% N/A
C 84% 56% 88% N/A
88% 81% 94% N/A
100% 100% 97% N/A
100% 100% 100% N/A
86% 81% 90% N/A
88% 76% 90% N/A
100% 76% 100% N/A
D 100% 88% 100% N/A
100% 100% 100% N/A
100% 100% 100% N/A
97% 84% 97% N/A
91% 89% 100% N/A
96% 91% 100% N/A
E 100% 100% 100% N/A
100% 99% 100% N/A
100% 100% 100% N/A
94% 76% N/A 92%
100% 76% N/A 85%
100% 88% N/A 100%
F 100% 88% N/A 100%
100% 100% N/A 100%
100% 100% N/A 100%
80% 50% N/A 80%
80% 45% N/A 80%
80% 48% N/A 70%
G 90% 75% N/A 75%
100% 95% N/A 98%
100% 100% N/A 100%
98% 64% N/A 83%
98% 79% N/A 83%
96% 74% N/A 85%
H 100% 83% N/A 89%
100% 94% N/A 98%
100% 100% N/A 100%
92% 75% N/A 79%
92% 58% N/A 83%
88% 63% N/A 96%
I 83% 75% N/A 96%
92% 83% N/A 96%
100% 100% N/A 100%
91% 69% 73% 74%
95% 70% 71% 74%
95% 81% 71% 83%
J 99% 89% 79% 88%
100% 97% 100% 100%
100% 100% 100% 100%
90% 60% 90% 80%
92% 62% 91% 80%
Average/Summary 93% 79% 93% 84%
96% 90% 94% 87%
99% 97% 98% 98%
100% 100% 100% 100%
* Based on a Pentium 4 3.0 GHz, 1 GB Ram
James W. VanGilder, PE Member ASHRAE Xuanhang (Simon) Zhang Associate Member ASHRAE James W. VanGilder is a principal engineer and Xuanhang (Simon) Zhang is an engineer with the Cooling Simulation Group, American Power Conversion Corporation, Billerica, MA. DISCUSSION H. Akbari, Senior Scientist, Lawrence Berkeley National Laboratory, Berkeley, CA: (1) Do you have measured data to support (and validate) the CFD simulations? (2) Can you comment on the application of CFD to cases where natural convection is dominant? James W. VanGilder: (1) Experimental validation was not part of this study; however, the paper references several studies that compare similar modeling efforts (i.e., modeling techniques and specific CFD code) to measured data. (2) This study was restricted to typical data center applications, which are forced-convection dominated. |
|
||||||||||||||
Printer friendly
Cite/link
Email
Feedback
Reader Opinion