The rise of artificial intelligence (AI) and machine learning (ML), alongside high-performance computing, has created new challenges for thermal management in data centers. As chip, server, and rack densities surge, traditional air-cooling methods struggle to manage the resulting heat levels effectively on their own. This has led to an evolutionary shift in IT thermal management towards a hybrid model that integrates air and liquid cooling technologies. Today, as many as one in five data centers (17%) are utilizing liquid cooling options, with over 61% considering its adoption, signaling a clear trend towards this more efficient and effective cooling solution.
In this context, coolant distribution units (CDUs) have emerged as a crucial component in the deployment of liquid cooling solutions. CDUs regulate the distribution of cooling liquid to maintain the temperature and pressure of the liquid within the required parameters. Additionally, these units are responsible for separating primary chilled water loops from secondary fluid networks, whose quality is crucial to the function of GPUs and cold plates. Overall, the role of CDUs is pivotal for the performance and longevity of liquid cooling systems.
Whether in newly built facilities designed for AI workloads or when retrofitting existing data centers, the strategic placement and effective management of CDUs are paramount to achieving the efficiency and performance benefits of liquid cooling. The following sections will delve into the integral role of CDUs for liquid-cooled data centers, the ideal conditions for their implementations, and the best strategies for installing and managing them.
What are CDUs?
Coolant distribution units, or CDUs, are integral to modern liquid cooling systems in data centers. They function as the main control center, regulating coolant distribution to ensure that the temperature and pressure remain within the optimal range. This control is paramount as it directly affects server cold plates' performance, efficiency, longevity, and overall system stability. Enhancing the data center's system-wide efficiency helps lower the cumulative ownership costs associated with high-density computing environments.
How Do CDUs Work?
CDUs are designed to create an isolated secondary loop from the chilled water supply. This allows data center operators to hold and control fluids in liquid cooling systems and oversee their pressure, temperature, flow rate, and even filtration quality. By sustaining the secondary fluid network's supply temperature above the data center's dew point, the CDU effectively averts the risk of condensation and ensures optimal sensible cooling.
Irrespective of the configuration—in-row or in-rack—liquid-to-liquid CDUs require a facility water system (FWS) connection through a heat exchanger that removes heat from the IT equipment cooling loop. This process enables highly efficient heat extraction, ensuring the server's performance is not compromised due to overheating.
Role of CDUs
The role of CDUs extends beyond just cooling. These units are equipped with supply-side filters, usually 50 microns in size, that prevent system contamination, keeping the server cold plate robust and ensuring optimal performance. This solution enhances system reliability and reduces the need for maintenance.
The importance of CDUs in the context of liquid cooling systems is evident. They ensure the smooth operation of the cooling system, the efficient transfer of heat, and the preservation of equipment lifespan, making them invaluable when deploying liquid cooling solutions in a facility. Data center operators may use this equipment with rear-door heat exchangers (RDHx), cold plates, and immersion cooling.
Best Practices for CDU Deployment and Management
Application engineers consider several factors to determine the requirements of CDU deployment. These include the number of racks to maintain, the heat-to-liquid ratio for liquid-cooled IT equipment within each rack, the required flow rate, and the overall facility layout.
For instance, CDUs supporting systems with high flow rate demands must have sufficient pumping capacity to handle the heat generated. Once the requirements have been established, engineers will:
1. Connect CDUs to Key Equipment
Liquid-to-liquid CDUs require a connection to the facility's chilled water supply or another heat rejection source to provide the needed coolant for removing heat from liquids. Additionally, it must also be linked through inbound and outbound pipes for the liquid to be cooled. The CDU has a heat exchanger that transports heat into the chilled water supply, fulfilling the liquid-to-liquid cooling cycle.
Conversely, liquid-to-air CDUs offer efficient cooling even without a chilled water supply for rack liquid cooling. Instead, they provide a secondary independent fluid circuit to the rack, eliminating heat from IT components.
The heated fluid cycles back to the CDU and passes through heat exchanger (HX) coils, allowing fans to dissipate the absorbed heat into the data center environment. This integrated approach leverages the data center’s existing air-cooled systems, which expel the heat outdoors, maintaining an optimized thermal balance.
2. Determine CDU Placement
IT and facility teams may place CDUs in various locations. CDUs directly cool each rack, whether positioned within or along a row. Alternatively, they can be located at the end of a row, serving all the racks within that area. Deploying an in-rack CDU that serves only one rack is also an option.
These units can be consolidated and placed at the back of a data center, contributing to a shared fluid system that serves all racks in the facility. The decision to adopt this arrangement depends on factors such as the desired level of redundancy, the efficiency goals for liquid cooling, and the available capital expenditure (CapEx) budget for installing modern cooling facilities.
3. Determine the desired level of redundancy
Achieving N+1 redundancy at the row level necessitates using two CDUs: one to feed the racks and a second for backup purposes. However, this approach comes at the cost of valuable floor space and increased expenses. Nevertheless, multi-tenant data centers (MTDCs) may opt for this layout to ensure redundancy for individual customer technology deployments.
To ensure adequate fluid flow, achieve redundancy with fewer units, and scale more efficiently, enterprise IT and facility teams may want to centralize all CDUs. This not only optimizes operations but also enables faster scalability.
4. Determine Liquid Cooling Requirements
As established, the required liquid cooling to sustain a 1 MW IT load varies depending on workload temperatures and flow rates. The IT and facilities teams assess rack power needs and hydraulic prerequisites.
The table below shows an example of a direct-to-chip liquid cooling system cooling twenty 50 kW racks. The description of heat-to-liquid, or the measurement of heat transferred from a source to a liquid coolant, refers to the cold plate liquid cooling technology. Cold plates are strategically positioned on major heat-generating IT hardware parts in this setting, while air cooling systems support other components.
Source: Deploying Liquid Cooling in the Data Center: A Guide to High-Density Cooling
The liquid flow rate per kilowatt is estimated at 1.5 liters per minute/kW. This measurement varies on the heat transfer effectiveness of the cold plate and the heat produced by the liquid-cooled IT components. The data center's specifying engineer will consult the IT equipment supplier to identify the maximum and minimum limits for the flow rate. Facility teams are responsible for reviewing flow rates periodically and adjusting allowable limits when the IT equipment is refreshed.
The IT and facility teams and contractors must inspect the liquid loop to verify that the piping can meet the required flow rate. They are also responsible for evaluating the dimensions of the pipework, secondary inlet, and return header, ensuring they align with the CDU connections.
Any hydraulic loops connecting to the CDU will employ pipework reducers, which are specialized fittings designed to connect pipes of different diameters, ensuring a smooth transition within a hydraulic loop. These reducers are essential in systems connecting to CDUs, as they adapt the pipework to match its connection size, maintaining optimal flow rates and system integrity.
Conclusion
Accelerated AI and machine learning advancements have escalated chip, server, and rack densities in data centers, exceeding traditional air-cooling capabilities. As a result, many data centers are adopting air-assisted liquid cooling systems, integrating liquid cooling technologies with existing air-cooled systems.
CDUs are vital for managing heat in modern data centers. Effective deployment of this cooling solution requires understanding key parameters and careful planning, including equipment connections, optimal CDU placement, redundancy levels, and specific cooling requirements. This strategy enhances efficiency, scalability, and reliability, cementing CDUs' role in advancing high-performance computing data centers.
White Paper: Deploying Liquid Cooling in the Data Center
To learn more about how liquid cooling manages the thermal output of data centers, we have a white paper available for download. Titled "Deploying Liquid Cooling in the Data Center: A Guide to High-Density Cooling," this document thoroughly analyzes the technology and provides insights into its operations and benefits.
At Vertiv, we strive to innovate and inform our customers about technological advancements. With our expertise in modern digital infrastructure, we provide solutions that are designed to meet future needs. Sign up below to stay informed on industry trends and drive your business toward a resilient digital future.
