Site preparation and environment

These operations are needed to prepare for the arrival of IBM Storage Ready Nodes.

Important: The scope of this documentation focuses on a single empty rack that is dedicated to an IBM Storage Ceph cluster on the IBM Storage Ready Nodes. However, you might want to install the cluster in an existing rack that is partially populated with other items, or across multiple racks. IBM can provide recommendations for cluster deployment in shared racks and across multiple racks.

Floor space sizing

Floor space sizing involves determining rack positioning relative to related equipment, providing sufficient front and rear service access aisle space, and orients the rack for proper component cooling and airflow. This space sizing determines whether the floor can support the total assembled rack weight as some floors are raised and have limited weight capacities.

Orientation

Orientation per related equipment is a function of your facility and how your network is physically configured. It might not always be possible to place the rack in the immediate vicinity of your client racks.

Service access

Service access is pre-determined by the existing positioning of rows and aisles in your facility with the rack that is placed similar to those racks already in place. Regardless, there must be sufficient front space to allow installation and service of Ready Nodes and data and management switches. Similarly, you must have sufficient rear space for service and network and power attach activities.

Cooling

Cooling ensures that appropriate temperature, humidity, altitude considerations, and volumetric airflow are available to meet the specifications of all rack components. Racks and front doors need to be selected such that component airflow is not impeded. Cabling and power cordage must be dressed to avoid airflow impedance and allow for access to rear-side Customer Replaceable Units (CRUs). Open rack positions should have filler panels to avoid airflow short circuits.

Note: Contemporary switches have the configuration option of front-to-rear or rear-to-front airflow. To simplify cable management, IBM recommends that the data and management switches be mounted, and airflow configured, to support the switch ports in the rear of the rack.

Total weight

The total weight is determined by summing the individual specified weights of each rack component along with the PDUs and the rack itself. The total rack weight, which is distributed across the area that is used by the rack, must be within the facility floor rating. The following table is a simple worksheet to calculate total rack weight using the component specifications.

In the following table, the vendor specification refers to the values from customer-supplied items.

Table 1. Calculate total rack weight
Component	Weight lbs (Kg)	Quantity
Empty rack	Vendor specification	1
PDU	Vendor specification	Number required
Data switch	Vendor specification	2
Management switch	Vendor specification	1
Ready Node	77.82 lbs (35.3 Kg)	Total number of Ready Nodes
		Total:

Rack height

Rack height requirement is determined by the total number of present and planned Ready Nodes, in addition to the data and management switch heights. The following table is a simple worksheet to calculate total rack height using the component specifications.

Table 2. Calculate total rack height
Component	Height (RU)	Quantity	Total height (RU)
Data switch	1	2	2
Management switch	1	1	1
Ready Node	2	Total Number of Ready Nodes
		Total:

Note: The rack itself must be able to support all mounted components. Stacking rules are used to prevent possible tipping issues. Heavier items are stacked first while switches are typically at the top of the rack. See Figure 2 for details.

Note: A single-rack IBM Storage Ceph on an IBM Storage Ready Nodes could involve as few as four Ready Nodes or up to 18 Ready Nodes, two data switches, and one management switch in a 42U rack.

Remember: Leave 1RU space between switches and adjacent Ready Nodes. Also, keep some space to allow access to cable ports.

Facility power distribution

Facility power distribution involves providing required system power while contributing to the overall system availability model. Individual Ready Nodes and switches support dual power supply units (PSUs) in a 1+1 sharing configuration that allows failure of any single PSU without loss of node or system functionality. To facilitate even higher availability, the facility power line feeds to these PSUs can also be configured in a manner to support varying degrees of line availability. The closest level involves dual PDU sets in the rack each feeding line power to one PSU of each system Ready Node and switch. For instance, one or more PDUs with sufficient outlets that are mounted on the right side for the rack can be connected to adjacent Ready Node and switch PSUs with jumper cords. Similarly, one or more PDUs with sufficient outlets that are mounted on the left side for the rack can be connected to adjacent Ready Node and switch PSUs with jumper cords.

However, it is important to understand the power line feed construction of your facility to fully leverage power line availability. There are several possibilities that are depicted as follows:

Dual-power installation - Redundant distribution panel and switch

This configuration requires that the system receives power from line feeds of two separate facility power distribution panels. Each distribution panel receives power from a separate piece of facility switch gear. This level of redundancy might not be available at most facilities.
Figure 1. Redundant distribution panel and switch
Dual-power installation - Redundant distribution panel

This configuration requires that the system receives power from line feeds of two separate facility power distribution panels. The two distribution panels receive power from the same piece of facility switch gear. Most facilities might be able to achieve this level of redundancy.

Figure 2. Redundant distribution panel
Single distribution panel - Dual circuit breakers

This configuration requires that the system receives power from line feeds of two separate circuit breakers in a single facility power panel. This configuration provides less power line redundancy. However, it is acceptable if a second power distribution panel is not available.
Figure 3. Dual circuit breakers

Facility power line quality

Contemporary server and switch PSUs are designed to operate over a wide input voltage range, international AC line frequency values, and tolerate some line disturbances. Ready Nodes and contemporary switches have the following AC line operational specifications:

Table 3. Operational specifications
Parameter	Specification	Tolerance
Line voltage	100-200Vac	+/- 10%
Line frequency	50 or 60Hz	+/- 0.5Hz
Minimum line dropout	10 ms

While Ready Nodes can tolerate certain power disturbances or transients, large disturbances can cause equipment power failures or errors. Transients can come into the site on the power utility company lines, but are often caused by other electrical equipment that is installed in the building. For example, transients can be produced by welders, cranes, motors, induction heaters, elevators, and other equipment. To prevent problems caused by power disturbances, have a transient-producing equipment on a separate power service than the one that supplies power to your information technology equipment.

Note: The selection of supplied AC power depends on facility AC line voltages, different geographies, available line current, and number of phases (single or three-phase). Because of the unit power requirements, and the possibility of expansion, IBM recommends that the AC line voltage be in the range of 200-240Vac. Smaller configurations might be supported, for example, by a single phase 208Vac 30A source. Larger configurations may require three-phase power.

Power calculation

Power calculation involves the determination of the total required power for the system. The examples that are provided in this document are for a rack that is dedicated to a Ready Node system, that is, a rack containing Ready Nodes, two data switches, and a management switch. If the rack also supports other, non-Ready Node components, their power requirements must be considered.

Note: A Ready Node system might involve as few as four Ready Nodes or up to 18 Ready Nodes, two Data switches, and one Management switch in a 42U rack.

Note: Each of the two facility power feeds to the rack must have the capacity to power the entire rack so that the loss of one facility power feed does not impact system operation.

The following table is a simple worksheet to calculate total rack power and number of PDU outlets utilizing the component specifications. In the following table, the vendor specification refers to the values from customer-supplied items.

Table 4. Power specifications
Component	Line power (VA)	Unit quantity	Number of PDU outlets required per side
Data switch	Vendor specification	2	2
Management switch	Vendor specification	1	1
Ready Node	Vendor specification	Total number of Ready Nodes	Total number of Ready Nodes
		System Totals:

For example,

250 VA Data switch (vendor specification)
200 VA Management switch (vendor specification)
12 Ready Nodes
Total line power - 9340 VA

PDU examples based on geographical electrical codes:

North America (NA) 3-phase 208V 30A supplies up to 8650VA (208V delta - 20% current derate)
European Union (EU) 3-phase 230V 32A supplies up to 22080VA (400V wye – No current derate)

In the first case, there is insufficient PDU capacity to power the entire rack. Hence, four (two per side for HA) PDUs would be required along with four total facility line drops. While this configuration would leave the PDUs underutilized, it provides headroom for future added Ready Nodes up to a total of 18 in a 42U rack.

In the second case, only two PDUs (one per side for HA) would be required along with two facility line drops. This also provides headroom for future added Ready Nodes up to a total of 18 in a 42U rack.

The selected PDUs must also have enough outlets to support the total number of rack components. In the example of 12 Ready Nodes, 2 Data switches, and 1 Management switch, a total of 15 PDU outlets per side are required. However, note that with a range of 200-240VA, the required individual device inlet current would allow for the use of Y-cords whereby a single PDU outlet might support 2 rack devices.

Acoustics

Acoustics involves any data center working personnel safety and working condition considerations for high ambient noise levels. This is because the cumulative noise from Ready Node cooling fans, plus other rack and facility cooling systems, can be excessive.

Shock and vibration

Shock and Vibration considerations involve making sure that any external source levels, as measured at the rack, are within specifications of the rack components including switches and Ceph Ready Nodes. Devices such as HDDs can be especially susceptible to excessive shock and vibration.

Networking

Networking includes separate data and Out of Band (OOB) management networks and hence separate switches and related considerations.

The data switches support the chosen number of Ready Nodes. There are 2 such switches configured as 1+1 HA and must support inter-switch port link aggregation and jumbo frames (9K). These data are considered leaf switches on your data network and should be connected to a corresponding 1+1 HA spine with appropriately sized uplinks.
The management switch also requires uplinks as necessary for access by a management console (laptop or workstation).
Network configuration details are described in the Installation and configuration section.

Note: You might have a reason for mounting the switches in a different spot in the rack, such as, the middle. Such positioning improves cable routing and management.

Note: Your chosen cable lengths depend on the distance from Ready Node to switch port as per your Ready Node stacking. Cable lengths must consider the need to be dressed to the sides of the rack to avoid impeding airflow and service access. Multi-mode fiber optical cable bend radii should be no less than 1.2” (30.5 mm).

Note: First install the rack, necessary PDUs and facility power drop connections, plus switches and cables so as to be prepared for installation of Ready Nodes once they arrive.