Cooling System with Low Power Usage Effectiveness below 1.02x for Server Rooms.
In order to drastically reduce the power for cooling information and communication technology (ICT) equipment such as servers, research is ongoing for the development of cooling elements and cooling systems for Internet data centers (IDCs) (Ministry of the Environment, 2016). In particular, the objective of the current project is to realize a power usage effectiveness (PUE = total power of the IDC / power of ICT equipment) of less than 1.02.
In recent times, server power utilization in IDCs has considerably increased, owing to high-power computer infrastructure and high-end GPU computing. For IDCs with high-load servers, a liquid immersion cooling system and a conventional air-cooling system for low-load hard disk drive storage are used together; however, in doing so, energy efficiency is not considered. Though energy conservation through the use of outside air and natural energy has progressed for the latest large-scale IDCs (over 1000 racks), leading to a PUE of around 1.1 (Aizawa et al. 2013-2015). The better energy conservation approaches and energy-saving technologies that are compatible with small- and medium-scale server rooms (less than several 100 racks) are desired. It should be noted that improving energy efficiency in small-to-medium-scale data centers is more difficult than in the case of large-scale IDCs, owing to the difficulty in installing such cooling systems as well as the associated high costs or low suitability of particular energy conservation systems at such centers. Considering this, our proposed system is suitable for both small- to medium- as well as large-scale server rooms. In particular, we intend to develop a cooling system that leads to the reduction in power consumption using general-purpose auxiliary peripherals (e.g., pumps and fans) to nearly zero.
In a previous study (Aizawa 2018), we proposed a server room with zones based on heat densities (high, medium, or low) and a cooling system that utilized outside air. In our previous study, cooling water was provided by a cooling tower, and the amount of time for which the cooling tower is employed increases with increasing temperature conditions of server as much as possible. We constructed a physical large-scale cooling system using cooling devices such as general-purpose chillers, cooling towers, pumps, and fans; based on our experimental results and energy simulations, we confirmed that the proposed approach leads to an annual average PUE of less than 1.04.
In this study, however, we change the configuration of the previously proposed cooling system based on the results of our experiment and last fiscal year's annual energy consumption simulation. In particular, we changed the cooling tower and piping from an open-type to closed-type system, removed some heat exchangers and heat storage tank, and included a chiller. The chiller is used as infrequently as possible, being used only in the cases when sufficient cooling cannot be maintained owing to high wet-bulb temperature of the outside air. In addition, we measured the power consumption and PUE of the modified cooling system under variable flow control and variable fan speed control. Based on the experimental results and annual energy simulation, we confirmed the stability of this modified cooling system in the zoned server room, achieving an annual average PUE of approximately 1.02.
Figure 1 shows an overview of the zoned server room and proposed cooling system. The server room can be zoned according to the heat load densities as high, medium, and low. When a server room with mixed heat load densities is cooled, the cooling efficiency reduces because the cooling is dictated by the high heat load density. In particular, the heat load density is said to be high, medium, or low if it is 16 kW (54.6 kBtu/h) or more, 9-16 kW (30.7-54.6 kBtu/h), or 8 kW (27.3 kBtu/h) or less per rack, respectively; in the case of a high heat load density, air cooling is not effective, whereas in the case of low heat load density, liquid immersion is not. For example, hard disk storage has a low heat load and can only be air-cooled. In this project, a low PUE was realized by using different types of cooling approaches for different heat load densities: liquid immersion cooling by natural convection of a refrigerant in a liquid tank for high heat loads (Matsuoka et al. 2018) (Matsuoka et al. 2017), drop coolant for medium heat loads (Matsuda et al. 2017), and air cooling for racks with low heat load (Impress 2017).
To achieve an annual PUE of 1.02 or less, which is the primary objective of this study, it is necessary to reduce the power of the cooling system by more than half from that of the system with an annual PUE of 1.04 that was presented in the previous study. Thus, we study the countermeasures and configuration of the cooling system using annual energy simulation. We set the upper limit for the temperature of the cooling water to 40[degrees]C (104[degrees]F) to ensure stable operation of the ICT equipment based on the individual experiments in the previous study at each load.
Figure 2 shows the outline of the cooling system proposed in our previous study before the abovementioned changes were implemented. In particular, the previous cooling system was composed of an open-type cooling tower, heat storage tank, primary pump, heat exchanger, and secondary pump; in addition, the cooling tower and each load were connected via heat exchangers. The secondary pump and heat exchangers were included to adjust the temperature and flow rate at each load; however, the ratio of the energy consumed by the secondary pump to the total power of the cooling system was high. Furthermore, the temperature exchange efficiency in the heat exchanger was around 0.6. Although our approach involved storing the cooled water produced at night in a heat storage tank for use during the summer day peak temperature, the difference between the outside air wet-bulb temperature during the day and night during the summer peak in Japan was small, and therefore, a sufficient cooling water temperature could not be obtained. Compared with an IDC that circulates air in the server room, the proposed system can reduce the power of heat sources as well as that of the fan used to circulate indoor air.
Table 1 lists the comparative conditions of the annual energy simulation and annual PUE results. The annual PUE values for different cooling water set temperatures with the open-type and closed-type cooling tower (including the presence or absence of the cooling tower heat exchanger), chiller, primary pumps, secondary pumps, and heat exchangers, were compared. For outside air conditions, the 2010 standard weather data (Tokyo) was used. For equipment characteristics, we used the experimental results from the previous study, new equipment basic characteristics as well as manufacturer specification values for new equipment. The electric power consumption and coefficient of performance (COP) of the chiller were measured during the winter and summer peaks. As can be seen from Table 1, the annual PUE is around 1.02 in Cases 2, 3a, and 3b for the closed-type cooling tower (without the cooling tower heat exchanger), with the primary pumps (capacity division) and secondary pumps, but without the heat exchangers.
Figure 3 shows the graphical representation of the cooling system power consumption, which is listed in Table 1. In Case 1, the power consumed by the chiller and secondary pump is larger than in other cases. The reason for the large amount of electric power consumed by the chiller is the longer operation time because of the effect of temperature loss in the heat exchanger. In Case 3b wherein the cooling water set temperature was 35.2[degrees]C (95.4[degrees]F) (with a margin to upper limit temperature for stable operation of the ICT equipment), the annual PUE was 1.023, suggesting the feasibility of our objective. In particular, the primary changes to the configuration of the cooling system are as follows: 1) The open-type cooling tower was changed to the closed-type one (with removing the cooling tower heat exchanger), 2) The secondary pump and heat exchanger were removed, and the capacity division in the primary pump was changed, 3) The heat storage tank was changed to a chiller during the peak summer time.
Figure 4 shows the system diagram of the cooling system after the configuration changes specified above were made. In Figure 4, the measurement points such as temperature, flow rate, and electrical power consumption, among others, are shown. In addition, the PUE calculation process is depicted on the right side of the figure. The cooling system consists of the closed-type cooling tower (with a fan and pump), primary pumps, and chiller; it allows indirect outside air cooling (free cooling). In this cooling system, the cooling water produced by the cooling tower is supplied to each load; the ICT equipment is cooled by exchanging heat with the heat medium (refrigerant or air) at heat exchangers of each load after the headers. Temperature and flow rate as well as other properties were measured around the headers of each load and at the inlet and outlet of the equipment, and the operation state of the cooling system was confirmed. Electrical power consumption was measured using a power meter on the power panel; furthermore, the PUE is calculated based on the power consumption of the cooling tower fans and pumps, primary pump, and chiller. In the measurement and calculation of PUE, a stable average value for 10 min during operation was used. Moreover, power for lighting and loss is not included in the PUE calculation. In our experiment, to simulate heat loads for the cooling system, pipe insertion-type electric heaters were added to the downstream of the return header. The thermal load was set at around 16, 15, and 8 kW (54.6, 51.2, 27.3 kBtu/h) for high, medium, and low loads, totaling around 39 kW (133 kBtu/h).
Figure 5 shows the installation conditions of the cooling tower and chiller on the roof of the building where the experiment was conducted. The pump unit of the experimental apparatus was installed below the rooftop and the headers supplied cooling water to each load. We installed two cooling towers and two chillers, because, as future work, we plan to examine the influence of two driving in summer.
RESULTS AND DISCUSSION
Experiments were performed to characterize the auxiliary machines and measure the power consumption and PUE as well as the operation status of the system. Furthermore, the annual PUE was calculated based on the experimental results.
Basic characteristics of the equipment. As the operating characteristics of the pump, electrical power consumption and differential pressure with varying flow rates of the cooling water were experimentally measured. Figure 6 shows the measurement results for power consumption and differential pressure of the primary pump (small pump) with different flow rates of cooling water. Figure 7 shows the relationship between the inverter frequency and power consumption of the fan as well as spraying water pump in the cooling tower. Figure 8 shows the relationship between the inverter frequency of the cooling tower fan and spraying water pump and the approach (i.e., cooling water outlet temperature--outside air wet-bulb temperature) as the operating characteristics of the cooling tower, when the outside air wet-bulb temperature is around 8 [degrees]C (46.4 [degrees]F). The results were, the approach was around 4 [degrees]C (7.2 [degrees]F) in the rated frequency of the cooling tower fan, the approach was around 4 [degrees]C to 6 [degrees]C (7.2 [degrees]F to 10.8 [degrees]F) when the cooling tower fan inverter frequency was 30 Hz to 50 Hz, the approach was around 4 [degrees]C (7.2 [degrees]F) when the water spray pump inverter frequency was 30 Hz to 50 Hz, and when the water spray pump inverter frequency was 20 Hz, the approach increased to around 16 [degrees]C (28.8 [degrees]F).
Operation status of the cooling system. Figure 9 shows the transition of the driving situation when the outside air wet-bulb temperature was 17-18[degrees]C (62.6-64.4[degrees]F). At the time 18:00 in Figure 9, the return cooling water temperature was around 37[degrees]C (98.6[degrees]F) and power consumption of the cooling system was 1.2 kW (4.1kBtu/h) (PUE=1.031) with the cooling tower fan frequency, cooling tower spray pump frequency, and primary pump (small pump) frequency set at 50 Hz, 30 Hz, and 23 Hz, respectively. Then, the cooling tower fan frequency was set at 15 Hz, cooling tower pump frequency was set at 25 Hz, and small pump frequency was raised such that the inlet water temperature of 35[degrees]C (95[degrees]F) at the cooling tower is achieved. After 19:00, the small pump frequency was further increased to 35 Hz; consequently, the inlet water temperature at the cooling tower was almost stable at about 36[degrees]C (96.8[degrees]F). At that time, the cooling system power consumption was around 0.52 kW (1.8kBtu/h) (PUE=1.013). The approach increased as the cooling tower fan and spraying pump were throttled; in addition, the flow rate of the small pump was increased to handle the same heat load, but the electrical power consumption of the cooling system decreased owing to power consumption characteristics of the element equipment. Even when the return water temperature target is the same, the cooling system's electrical power consumption could be lowered by selecting an operating method based on the electrical power characteristics of the element equipment. The final inlet water temperature target of 40[degrees]C (104[degrees]F) indicated the upper limit temperature of the return water for stable operation of the ICT equipment. Then, when the small pump frequency was lowered to 25 Hz, the inlet water temperature at the cooling tower stabilized at around 40[degrees]C (104[degrees]F). The stable cooling operation was confirmed based on the transition of the cooling water temperature, when the cooling system power consumption was around 0.42 kW (1.4kBtu/h) (PUE=1.011).
PUE of the cooling system. Figure 10 shows the outside air wet-bulb temperature and PUE trend for the cooling system after the configuration change. The horizontal axis in the figure represents the outside air wet-bulb temperature. In particular, the subfigure on the top shows the annual occurrence frequency (hour of occurrence) of each wet-bulb temperature in the standard meteorological data (Tokyo), while the one below shows the PUE against the outside air wet-bulb temperature. The dashed line in the subfigure below shows the case when the annual PUE is 1.02 in the energy simulation for the cooling system after the configuration change. When the outside air wet-bulb temperature is around 20[degrees]C (68[degrees]F) WB or less, the PUE is low, approximately 1.01, and when the outside air wet-bulb temperature is higher than 20[degrees]C (68[degrees]F) WB, the PUE increases; this is due to the increase in cooling system power consumption because of the cooling tower, primary pump, and chiller.
The experimental result when the outside air wet-bulb temperature was 17[degrees]C (62.6[degrees]F) WB is also shown in Figure 9; the energy simulation values were in agreement with our experimental results. When the outside air wet-bulb temperature is around 8[degrees]C (46.4[degrees]F) WB, the estimated value includes assumed countermeasures against electrical power reduction in the experimental results owing to the equipment characteristics in the cooling system after the configuration change. Furthermore, when the outside air wet-bulb temperature is around 27[degrees]C (80.6[degrees]F) WB, the estimated value is considered to be the manufacturer specification value of the chiller and the electrical power consumption of the cooling tower and primary pump based on the experimental equipment characteristics. Based on these results, the feasibility of obtaining an annual PUE of less than 1.02 for the modified cooling system can be confirmed.
In our study, the configuration of a previously proposed cooling system that utilized outside air for a zoned server room based on heat density was modified to improve its PUE based on the annual energy simulation results.
An experiment was conducted using a real-scale modified cooling system after the configuration change and simulated load devices, including a real server machine and cooling auxiliary devices. Then, the annual average PUE was estimated. Our results not only confirmed stable operation of the cooling system, but also the feasibility of an annual average PUE of approximately 1.02. Our future work will involve confirming the estimated PUE through actual measurements during the summer season.
This study was supported by the development and demonstration projects for CO2 emission reduction of the Ministry of the Environment in Japan. Liquid immersion- and refrigerant sprinkle-type ICT equipment cooling elements were developed by Osaka University, EEC Research Institute, and Fujitsu Limited. We give sincere thanks to the members of these institutions for their helpful comments. In particular, we would like to thank Mr. Matsuoka and Mr. Matsuda of Osaka University and EEC Research Institute; Mr. Fujimaki, Mr. Yamamoto, and Mr. Kubo of Fujitsu Limited; and Mr. Shibata, Mr. Ikeda, and Mr. Murata of Takasago Thermal Engineering Co., Ltd. for the fruitful discussions.
Ministry of the Environment. 2016. The development and demonstration projects for C[O.sub.2] emission reduction of the Ministry of the Environment in Japan. Tokyo: Ministry of the Environment of Japan. (http://www.env.go.jp/press/102545.html)
Aizawa, N., Shibata, K., Ikeda, M., Matsuoka, M. et al. 2013-2015. Development of air conditioning system for data center toward radical low-carbon (1st - 3rd). Tokyo: The Society of Heating, Air-conditioning and Sanitary Engineers of Japan.
Aizawa, N. 2018. Cooling System with Low PUE Cooling Power for Server Rooms. ASHRAE Wiinter Conference 2018, Conference Paper.
Matsuoka, M., Matsuda, K., and Kubo, H. 2017. Liquid immersion cooling technology with natural convection in data center. IEEE 6th International Conference on Cloud Networking. Plague: IEEE.
Matsuoka, M., Matsuda, K., and Kubo, H. 2018. Effective Cooling of Server Boards in Data Centers by Liquid Immersion Based on Natural Convection Demonstrating PUE below 1.04. ASHRAE Wiinter Conference 2018, Conference Paper.
Matsuda, K., Matsuoka, M., and Miyake, Y. 2018. Proposal of Cooling System for High Performance Computing by Drip-Feeding Cooling. ASHRAE Wiinter Conference 2018, Conference Paper.
Impress. 2017. Demonstration project of next-generation data center supporting IoT society starts. http://sgforum.impress.co.jp/article/3634?page=0%2C0
Table 1 Comparative conditions of the annual energy simulation and annual PUE. Set point temp. Case Cooling Air outlet water temp. temp. for low load [degrees]C [degrees]F [degrees]C [degrees]F Case 1 37.7 99.9 40.0 104 Case 2 37.7 99.9 40.0 104 Case 3a 37.7 99.9 40.0 104 Case 3b 35.2 95.4 37.5 99.5 Case Calc. Cooling Cooling Chiller Large annual water tower/Heat pump PUE flow rate exchanger (kW) l/min [ft.sup.3]/s Case 1 1.106 168 0.099 Open/ With 2.2 With Case 2 1.021 257 0.151 Close/Without Without Case 3a 1.022 234 0.138 With 7.5 Case 3b 1.023 157 0.092 High load Mid. load Low load Case Small 2nd. 2nd. 2nd. 2nd. 2nd. 2nd. pump Hex Pump Hex Pump Hex Pump (kW) (kW) (kW) (kW) Case 1 With- With 1.5 With 1.5 With 0.75 out Case 2 Case 3a 0.75 Without Case 3b
|Printer friendly Cite/link Email Feedback|
|Date:||Jan 1, 2019|
|Previous Article:||Study on the Prediction Models of Temperature and Energy by using DCIM and Machine Learning to Support Optimal Management of Data Center.|
|Next Article:||Novel Ventilation Concepts for Efficient Heating Dynamics of Vehicle Passenger Compartments.|