Water cooling system specification and requirements

Learn about the specific water treatment procedures and requirements that are needed for situations where the facilities water system (FWS) water or the technology cooling system (TCS) water can be used to directly cool the datacom equipment.

Overview

The datacom equipment cooling system (DECS) is a water loop in which the water comes in contact with the components to be cooled. There are cases where the DECS water is supplied by an in-rack coolant distribution unit (CDU) or can be supplied by an external CDU that service multiple racks. For details of potential liquid cooling systems and loops within a data center and the terminology that is used, see Figure 1.

The water quality standards that are specified pertain only to the DECS water loop that comes into contact with compute components. The ongoing monitoring and maintenance procedures are also described.

The cooling loop hardware consists mainly of corrosion resistant alloys, such as copper alloys and stainless steels. Ethylene propylene diene monomer (EPDM) rubber must form the inner lining of all the hoses in the system. The chemistry of the cooling water must be properly maintained to avoid system disruption or shutdown due to any of the four common water-related problems of corrosion, microbiological growth, scale formation, and fouling.

The details of the water treatment depend on whether the local municipality allows the disposal of water that contains some cleaning chemicals down a sanitary drain. If the local municipality does not allow the disposal of contaminated water down a sanitary drain, a deionizing bypass can be included in the water cooling loop to allow the cleaning of the water to purity levels corresponding to resistivity > 0.1 MΩ.cm (conductivity < 10 μS/cm) before pouring the water down the drain. You are responsible for verifying the local regulations before you dispose any water.

Figure 1. Example of liquid cooling systems and loops within a data center

Water-related problems

Proper treatment of the water is necessary to avoid the following common water-related problems: corrosion, microbiological growth, scale formation, and fouling. Any of these problems can significantly reduce the cooling efficiencies and increase the risk of system downtime.

Corrosion - corrosion can take on many forms. The common forms of corrosion that are relevant to the cooling loop include the following forms:
- Uniform corrosion, also referred to as general corrosion, is the spatially uniform removal of metal from the surface. It is the typical expected mode of corrosion.
- Pitting corrosion is a localized attack of a metal surface that in the case of copper tubes can lead to water leaks with a typical mean time to failure of around 2 years.
- Galvanic corrosion arises when two metals that are wide apart in the galvanic series are in electrical contact and immersed in the same water environment. The potential difference that arises between the two metals in contact, forces electrons to flow from the less noble to the more noble metal. On the less noble metal surface, corrosion occurs, giving off electrons that are consumed on the more noble metal surface by a reduction reaction that can take many chemical forms. Examples are the reduction of metal ions or the consumption of oxygen and water to form hydroxyl ions. Even when not in electrical contact, aluminum can be galvanically attacked by copper because of dissolved copper ions in low concentrations that deposit on the aluminum surface forming the galvanic corrosion couple.
Microbiological growth - microbiological growth in water cooling systems can lead to deposition, fouling, and corrosion within the cooling loop. Prevention of microbiological growth involves making sure that the cooling loop hardware is assembled from components that are free of biological organisms and treatment with biocides to control the bacteria population. To avoid biological growth, the water cooling loops must be shipped and stored dry. Every effort must be made to blow out the water and dry the water cooling loop as much as possible before shipping and storage.
Scale formation - scaling is the deposition of dense, adherent material on the cooling loop surfaces. Scaling occurs when the solubilities of salts in water are exceeded because of high concentrations or because of increased temperature.
Fouling - fouling of cooling loops is the deposition of non-scale-forming substances such as corrosion products and organics. Fungi, such as Fusarium sp, are known to grow, foul, and plug filters and fine finned heat sinks. They generally grow at the water line in cooling tower basins or sumps.

Avoiding water-related problems

The following best practices can be used to avoid water-related problems:

Design clean - Restrict the water-wetted metallurgies to copper alloys and stainless steels. Avoid the use of plain-carbon steel hardware that can rust and foul the water cooling loop.
Build clean - Ensure that the cooling loop components are clean and free of bacteria and fungi. The cooling loop assembly must be free of soldering, brazing fluxes, or both. Clean water must be used in the assembly operations. Any residual water must be blown out of the assembly. The finished assembly must be clean and dry.
Ship clean - Any residual water from the assembly, testing operations, or both must be blown out from the cooling loop before any shipping to avoid corrosion and microbiological growth. As a final step, use nitrogen gas to dry the system. Plug ends and ship the system with the cooling loop pressurized with nitrogen gas.
Install clean - The cooling loop must be kept clean during the installation step. Brazing is preferred over soldering. Problem with soldering is porous joints that keep leaching out flux residue. All flux residues must be cleaned off. Fill the system with clean water and, if possible, include a secondary step to deionize the water in the cooling loop before the addition of biocide and corrosion inhibitors.
Maintain clean - Monitor and maintain the pH, water conductivity, bacteria count, and the corrosion inhibitor concentration.

Water quality requirements

Use the following requirements to plan for the water quality in your system:

The water that is required to initially fill the system side cooling loop must be reasonably clean, bacteria-free water (less than 100 CFU/ml), such as demineralized water, reverse osmosis water, deionized water, or distilled water.
The water must be filtered with an in-line 50 μm filter.
If reasonably clean water is not available, the following guidelines are recommended. It is especially useful for large cooling loops: In this method, the water is deionized before any of the racks are connected to the water loop.
- It is important to ensure that the system water is cleaned before any chemicals are added to the water. This can be accomplished by deionizing the water by using deionizing cartridges that are installed in the cooling loop. An example is shown in Figure 2. Even if deionized water is used to fill the system, a deionizing step is prudent for two reasons: the first is to ensure that the starting water is deionized and the second is to remove any ions that might leach off the walls of the cooling loop.
- When the water needs to be deionized, the valves V2 and V3 can be opened and valve V1 partially closed to bypass some of the water through the deionizing canister.
- During this deionizing step, the cooling loop and the computers can keep operating normally.
- When the deionization is complete, the V2 and V3 valves must be closed and V1 fully opened.
- The deionization step raises the resistivity of the water greater than 1 MΩ.cm.
- Under normal operation, the V2 and V3 valves are closed and V1 valve is fully open.

Figure 2. Deionizing the water using deionizing cartridges installed in the cooling loop

Water quality chemical requirements

The following measurements must be performed before you add any chemicals to the water loop. It is the starting point that defines a clean water base.

All metals less than or equal to 0.10 ppm
Calcium less than or equal to 1.0 ppm
Magnesium less than or equal to 1.0 ppm
Manganese less than or equal to 0.10 ppm
Phosphorus less than or equal to 0.50 ppm
Silica less than or equal to 1.0 ppm
Sodium less than or equal to 0.10 ppm
Bromide less than or equal to 0.10 ppm
Nitrite less than or equal to 0.50 ppm
Chloride less than or equal to 0.50 ppm
Nitrate less than or equal to 0.50 ppm
Sulfate less than or equal to 0.50 ppm
Conductivity less than or equal to 10.0 μS/cm. Conductivity must be measured at 20°C - 25°C (68°F - 77°F). Conductivity increases approximately 5% for every degree Celsius rise in temperature
pH 6.5 – 8.0
Turbidity (NTU) less than or equal to 1

Plumbing materials requirements

All piping must be composed of specified materials to prevent scaling and to allow proper reactions with the chemistry of the water within the system. Threaded joints must not be sealed with polytetrafluoroethylene tape, as particles from the tape may enter the water stream and create clogs. Instead, a thread sealant, must be used to seal threaded fittings. Piping must be large enough, as dictated by industry best practices, to avoid excessive water velocity as well as undue pressure drop.

Material selection and installation is a complex issue that is governed by building codes and other local requirements. You are encouraged to consult with appropriate authorities that have jurisdiction (such as building inspectors, fire departments, insurance providers, and code compliance officers) before you plan and install cooling distribution systems. The following information is provided for chemical compatibility purposes.

The following alloys must be avoided in the plumbing system:

Aluminum and aluminum alloys.
Brass with greater than 15% zinc.
Free-cutting brasses, especially brasses that contain lead. One example of such brass is copper alloy C36000 called free-cutting yellow brass.
High lead brasses are especially a concern because of stress corrosion cracking when subjected to high tensile stress.
Steels that are not stainless steel.
Stainless steels that are not properly solution treated.

The following materials are preferred:

Copper alloys:
- Lead-free copper alloys with less than 15% zinc.
Stainless steels:
- Low carbon stainless steels are preferred.
- Must be solution that is treated. Stainless steel that is solution-treated undergoes a specific heat treatment to improve its resistance to corrosion.
- Passivation is desirable as long there is low possibility of acid entrapment in crevices.
- Avoid sensitization during welding.
- Avoid brazing; welding is preferred.
Polyvinyl chloride (PVC) (not allowed inside IBM® products due to flammability concerns, but can be used at a facility level. Appropriate authorities that have jurisdiction must be consulted).
EPDM rubber is the preferred material for hoses:
- Flammability rating must be CSA or UL VW-1 or better.
- Peroxide cured hoses are preferred because they do not absorb triazoles.

Metal joining operations:

Solder joints that come in contact with water must be avoided. Solder joints are porous and they leach flux residue into the cooling loop. Solder joints might pass inspection and pressure tests as manufactured, but still might be unreliable.
Brazed joints are preferred for joining copper plumbing pieces.
Braze joints must not be used to join stainless steels. Tungsten inert gas (TIG) and metal inert gas (MIG) welding are preferred for joining stainless steels. Sensitization must be avoided. Welded assembly must be cleaned and, if possible, passivated if there is a low possibility of acid entrapment in crevices.

Deionizing equipment

Deionizing equipment is optional. It is recommended for use in large cooling loops. When the water needs to be deionized, some of the water can be bypassed to flow through the deionizing cartridge.

Dosing Equipment

The following equipment is used to dose the cooling loop:

Recommend using a stainless steel or fiberglass chemical shot feeder.
System volumes less than 378.5 liters (100 gallons) use a 0.38 liter (0.1 gallon) size feeder
System volumes less than 3875 liters (1000 gallons) use a 3.8 liter (1 gallon) size feeder
System volumes greater than 3875 liters (1000 gallons) use a 9.5 liter (2.5 gallon) size feeder.
Chemical pump according to Nalco or another water treatment contractor specification.

Monitoring equipment

The following equipment is used to monitor the cooling loop:

3D TRASAR® Controller (#060-TR5500.88) for systems larger than 250 gallons to enable precise and continuous monitoring of system water chemistries: conductivity, pH, corrosion rate, and turbidity.
Azole test kit
- Nalco P/N 460-P3119.88 – Triazole Reagent Set, 25 mL
- Nalco P/N 500-P2553.88 – UV Lamp with power supply, 115 VAC
- Nalco P/N 400-P0890.88 – Nalco DR/890 Colorimeter
- Nalco P/N 500-P1204.88 – 25 mL graduated cylinder
Nalco bacteria test kit
- Nalco P/N 500-P3054.88 – Bacteria dip slides
Water resistivity monitor with 0-10 MΩ.cm range
- Nalco P/N 400-C006P.88

Required materials and equipment

The following items must be available to properly and safely complete the initial system start:

De-ionizing cartridges of appropriate capacity (optional).
Nalco treatment chemicals in appropriate quantities.
- System with 75.7 liters (20 gallons) or less coolant: Use a suggest prepackaged cleaner and inhibitor solution: Nalco 460-CCL2567 or Nalco CCL2567 and Nalco 460-CCL100 or Nalco CCL100. If exposure to bacterial is suspected or a concern, biocides such as Nalco H-550 or Nalco 73500 can be used. If fungi are suspected or a concern, Nalco 77352 can be used.
- System with greater than 75.7 liters (20 gallons) of coolant: Use a suggest using concentrated chemicals. The cleaner in concentrated form is Nalco 2567. The inhibitor in concentrated form is Nalco 3DT-199. If exposure to bacterial is suspected or a concern, biocides such as Nalco H-550 or Nalco 73500 can be used. If fungi are suspected or a concern, Nalco 77352 can be used.
A method to add chemicals: Use an installed system chemical shot feeder, an appropriately sized chemical feed pump, or both.
Source of demineralized water, reverse osmosis water, deionized water, or distilled water.
Proper personal protective equipment.
Approved drainage to drain pre-cleaning waters (for example, sanitary sewer). You are responsible for the drainage process according to local regulations.
Appropriate test kits to monitor Nalco 3DT-199 residual and bacteria count after Nalco H-550, Nalco 73500 or Nalco 77352 addition.
Water resistivity monitor with 0-10 MΩ.cm range.

Initial treatment for systems smaller than 75.7 liters (20 gallons)

Use the following procedure to clean your system:

Note: This procedure must be performed on the cooling loop before any computer racks are connected to the system.

System must be empty. If it is not empty, you must drain the system completely.
Remove all the filters from the filter housings.
Ensure that bypass hoses are connected between the supply and return portions of the cooling loop to ensure the cleaning of all sections of the system.
One of the following two cleansing procedures can be used:
1. Chemical cleaning - This method is the most effective way for cleaning the plumbing loop.
  1. Fill the system with cleaning solution. The suggested cleaning solutions are Nalco 460-CCL2567 or Nalco CCL2567.
  2. Circulate the cleaning solution for a minimum of 30 minutes (longer if time permits) to ensure that it reaches all sections of the system.
  3. Drain the system completely, disposing of the cleaning solution according to local regulations
  4. Refill with demineralized water, reverse osmosis water, deionized water, or distilled water.
  5. Circulate the water for 15 minutes.
  6. Drain the system completely, disposing of the cleaner according to local regulations.
  7. Immediately proceed to fill the system with water that contains premixed inhibitor and preservative.
2. Cleaning with deionized water. This procedure can be used if the cleaning chemical cannot be obtained or if local laws prevent the disposal of the chemicals.
  1. Completely fill the system with demineralized water, reverse osmosis water, deionized water, or distilled water.
  2. Deionize the water by bypassing some of the water flow through the deionizing cartridge or cartridges and circulate the water normally through the complete system until the resistivity of the water increases above 1 MΩ cm.
  3. Proceed to the chemical dosing procedure.

Chemical dosing procedure for systems smaller than 75.7 liters (20 gallons)

Use the following procedure for chemical dosing:

Install a new or cleaned 50 μm filter in the filter housings.
One of the following two dosing procedures can be used:
1. If the system was cleaned by using cleaning solution Nalco 460-CCL2567 or Nalco CCL2567 and if at the end of the cleaning step, the system was empty with no water in it, complete the following steps:
  1. Fill the coolant reservoir with Nalco 460PCCL100 / Nalco CCL100. Add 120 ppm of Nalco 3DT-199 to raise the azole concentration to 40 ppm.
  2. If bacteria or fungi is suspected or is a serious concern, add one of the following biocides:
    - 100 parts per million (ppm) of Nalco H-550 (glutaraldehyde)
    - 200 ppm of Nalco 73500 (glutaraldehyde)
    - 100 ppm of Nalco 77352 (isothiazolone)
    The choice of biocide depends on the expected microbiological material in the cooling loop. Glutaraldehyde biocide is more effective against anaerobic bacteria. Isothiazolone is more effective against aerobic bacteria, fungi, and algae. When in doubt, use the isothiazolone biocide.
  3. Confirm azole residual using Nalco azole test kit.
  If the system was cleaned by using deionized water only and the system is full of deionized water, complete the following steps:
  1. Add one of the following biocides:
    - 100 parts per million (ppm) of Nalco H-550 (glutaraldehyde)
    - 200 ppm of Nalco 73500 (glutaraldehyde)
    - 100 ppm of Nalco 77352 (isothiazolone)
    The choice of biocide depends on the expected microbiological material in the cooling loop. Glutaraldehyde biocide is more effective against anaerobic bacteria. Isothiazolone is more effective against aerobic bacteria, fungi, and algae. When in doubt, use the isothiazolone biocide.
  2. Add 120 ppm of Nalco 3DT-199 to achieve 40 ppm azole concentration.
  3. Confirm azole residual using Nalco azole test kit.

Initial treatment for systems larger than 75.7 liters (20 gallons)

Use the following procedure to clean your system:

Note: This procedure must be performed on the cooling loop before any computer racks are connected to the system.

System must be empty. If it is not empty, you must drain the system completely.
Remove all the filters from the filter housings.
Ensure that bypass hoses are connected between the supply and return manifolds of the cooling loop to ensure the cleaning of all surfaces of the cooling loop.
One of the following two cleansing procedures can be used:
1. Chemical cleaning - This method is the most effective way for cleaning the plumbing loop.
  1. Fill the system with demineralized water, reverse osmosis water, deionized water, or distilled water.
  2. Add the required volume of cleaning solution Nalco 2567 according to the manufacturer recommendation.
  3. Circulate the cleaning solution for a minimum of 4 hours.
  4. Drain the system completely utilizing all drain ports available, disposing of the cleaning solution according to local regulations
  5. Refill with demineralized water, reverse osmosis water, deionized water, or distilled water.
  6. Circulate the water for 1 hour.
  7. Drain the system completely utilizing all drain ports available, disposing of the cleaning solution according to local regulations
  8. Refill with demineralized water, reverse osmosis water, deionized water, or distilled water.
  9. Circulate for 15 minutes.
  10. Immediately proceed to the chemical dosing procedure.
2. Cleaning with deionized water. This procedure can be used if the cleaning chemical cannot be obtained or if local laws prevent the disposal of the chemicals.
  1. Completely fill the system with demineralized water, reverse osmosis water, deionized water, or distilled water.
  2. Deionize the water by bypassing some of the water flow through the deionizing cartridge or cartridges and circulate the water normally through the complete system until the resistivity of the water increases above 1MΩ cm.
  3. Proceed to the chemical dosing procedure.

Chemical dosing procedure for systems larger than 75.7 liters (20 gallons)

Use the following procedure for chemical dosing:

Note: The dosing procedure for systems larger than 75.7 liters (20 gallons) is the same regardless of the cleaning technique.

Install a new or cleaned 50 μm filter in the filter housings.
One of the following two dosing procedures can be used:
1. If the system was cleaned by using cleaning solution Nalco 460-CCL2567 or Nalco CCL2567 and if at the end of the cleaning step, the system was empty with no water in it, complete the following steps:
  1. Fill the coolant reservoir with Nalco 460PCCL100 / Nalco CCL100. Add 120 ppm of Nalco 3DT-199 to raise the azole concentration to 40 ppm.
  2. If bacteria or fungi is suspected or is a serious concern, add one of the following biocides:
    - 100 parts per million (ppm) of Nalco H-550 (glutaraldehyde)
    - 200 ppm of Nalco 73500 (glutaraldehyde)
    - 100 ppm of Nalco 77352 (isothiazolone)
    The choice of biocide depends on the expected microbiological material in the cooling loop. Glutaraldehyde biocide is more effective against anaerobic bacteria. Isothiazolone is more effective against aerobic bacteria, fungi, and algae. When in doubt, use the isothiazolone biocide.
  3. Confirm azole residual using Nalco azole test kit.
  If the system was cleaned by using deionized water only and the system is full of deionized water, complete the following steps:
  1. Add one of the following biocides:
    - 100 parts per million (ppm) of Nalco H-550 (glutaraldehyde)
    - 200 ppm of Nalco 73500 (glutaraldehyde)
    - 100 ppm of Nalco 77352 (isothiazolone)
    The choice of biocide depends on the expected microbiological material in the cooling loop. Glutaraldehyde biocide is more effective against anaerobic bacteria. Isothiazolone is more effective against aerobic bacteria, fungi, and algae. When in doubt, use the isothiazolone biocide.
  2. Add 120 ppm of Nalco 3DT-199 to achieve 40 ppm azole concentration.
  3. Confirm azole residual using Nalco azole test kit.

System monitoring and maintenance

Use the following guidelines for system monitoring and maintenance:

It is important to conduct a bacteria test on a quarterly basis and add 100 ppm of Nalco H-550 or 200 ppm of Nalco 73500 biocide if the bacteria count is greater than 1000 CFU/ml. Nalco 77352 fungicide can be added if fungi have been a concern in the past.
- Fungi might not be detected in the water, even though it can grow and cause blockage of cooling channels in cold plates that are used to cool the computer processors. Reduced coolant flow rate through the cold plates might be an indication of blocked channels due to fungi growth.
On large systems that have more than 250 gallons of water, Nalco 3D TRASAR® controller must be installed on the system cooling loop to enable precise and continuous monitoring of system water chemistries, conductivity, pH, corrosion rate, and turbidity.
It is important to conduct an azole test on an annual basis and add Nalco 3DT-199 to bring the azole concentration to the wanted 40 ppm level or any other desirable ppm level.

Multiple racks

Use the following guidelines when adding additional racks:

Racks arrive from IBM ready for installation.
Install rack or racks and open flow from existing system.
Make sure that the automated water make up on the chiller coolant reservoir is activated. If there is no automated water make up feature, top off the system side reservoir.
Within 2 hours of installing the new rack or racks, add one of the following biocides:
- 100 parts per million (ppm) of Nalco H-550 (glutaraldehyde)
- 200 ppm of Nalco 73500 (glutaraldehyde)
- 100 ppm of Nalco 77352 (isothiazolone)
The choice of biocide depends on the expected microbiological material in the cooling loop. Glutaraldehyde biocide is more effective against anaerobic bacteria. Isothiazolone is more effective against aerobic bacteria, fungi, and algae. When in doubt, use the isothiazolone biocide.
Add 120 ppm of Nalco 3DT-199 to achieve 40 ppm of azole concentration. The amount of inhibitor dosage is calculated based on the volume of the makeup water.
Confirm azole residual using Nalco azole test kit.

Refreshing the water

Situations might arise where the water must be refreshed (for example, the system must be cleaned and the biocide and corrosion inhibitor is added again). To refresh the water, use one of the two following procedures:

If you prefer that no water goes down the sanitary sewer, use the following procedure:

Remove the inline 50 μm filter from the filter housing.
Insert new deionization cartridges into the canisters and bypass some of the water through the deionizing cartridges until the resistivity of the water increases above 1 MΩ.cm. During this period, the systems and cooling system can be left on and be fully functional.
Stop the bypassing of the water flow through the deionizing filter and add a new or cleaned 50 μm filter to the inline filter housings.
Add one of the following biocides:
- 100 parts per million (ppm) of Nalco H-550 (glutaraldehyde)
- 200 ppm of Nalco 73500 (glutaraldehyde)
- 100 ppm of Nalco 77352 (isothiazolone)
The choice of biocide depends on the expected microbiological material in the cooling loop. Glutaraldehyde biocide is more effective against anaerobic bacteria. Isothiazolone is more effective against aerobic bacteria, fungi, and algae. When in doubt, use the isothiazolone biocide.
Circulate for 30 minutes.
Add 120 ppm of Nalco 3DT-199 to achieve 40 ppm of azole concentration.
Circulate for 30 minutes.
Confirm azole residual using Nalco azole test kit.

If the water can be poured down the sanitary sewer, use the following procedure:

Pour the water down the drain with the permission of the local municipality.
Fill the system by using one of the following procedures:
- Systems less than 75.7 liters (20 gallons): Initial treatment for systems smaller than 75.7 liters (20 gallons).
- Systems larger than 75.7 liters (20 gallons): Initial treatment for systems larger than 75.7 liters (20 gallons).

Moving or storing the system

If you need to move your system or put it into storage, the water in the system must be drained. You can drain the water in one of two ways:

The water can be deionized to a purity corresponding to resistivity greater than 0.1 MΩ.cm and then can be poured down any municipal drain.
The water can be poured down a sanitary drain with the permission of the local municipality.

Disposal of water and cartridges

The deionizing cartridges must be disposed off in accordance with local municipality ordinances.

IBM is not responsible for the disposal of water. You are responsible for determining the local regulations that govern the disposal of water.

Troubleshooting

If you encounter any problems with your water-cooled system, use the following table to troubleshoot your problem.

Table 1. Troubleshooting
Problem	Solution
Poor cooling performance	Contact IBM Service
Reduced water flow	Contact IBM Service.
Chemical pump issues (where installed and used)	Follow procedures that are provided by installer, contact your site water treatment contractor or local Nalco office, or both.
3D TRASAR® alarms or operational issues	Contact the local Nalco office.
Discolored water	Might be an indication of corrosion and or microbiological issues. Refresh the water supply.
Slime in flow meter areas	Might be an indication of corrosion and or microbiological issues. Refresh the water supply.
Elevated microbial counts	For systems of smaller than 75.7 liters (20 gallons), refresh the water supply. For systems of larger than 75.7 liters (20 gallons), add 100 parts per million (ppm) of Nalco H-550 or Nalco 73500 biocide. Retest the bacteria content between 24 – 48 hours after biocide dosing. If the bacteria level is not less than 100 CFU/ml contact Nalco or your water treatment company.
Fungi	Contact Nalco or your water treatment company.
Any other problems	Contact IBM Service.