SSLL2F_1.2.5 - Documentation Index
Table of Contents
Welcome
IBM Cloud Infrastructure Center
Overview
Terminology
What's new in IBM Cloud Infrastructure Center 1.2.5
Features and support matrix
IBM Cloud Infrastructure Center high availability architecture
Roles and tasks
Getting started tutorials
Getting started with the new standalone installation of the IBM Cloud Infrastructure Center 1.2.5
Getting started with KVM
Getting started with z/VM®
Getting started with the new installation of the IBM Cloud Infrastructure Center 1.2.5 multi-node cluster
Getting started with the upgrade to release 1.2.5
Planning
Hardware and software requirements
Planning
Hardware and software requirements for KVM
Hardware and software requirements for z/VM
Hardware and software requirements for multi-node cluster
Dependent RPM list for RHEL 8.x Management Node (z/VM / KVM)
Dependent RPM list for the RHEL 8.x and RHEL 9.x (z/VM and KVM) compute nodes
Planning for compute nodes and hypervisors
Planning for local storage
Planning for Local Storage on z/VM®
Planning for local storage on KVM
Planning shared local storage
Planning for shared local storage on z/VM®
Planning for shared storage on KVM
Planning for live migration
Planning for live migration for KVM
Planning for live guest relocation for z/VM
Planning for Host groups
Planning for availability zone
Planning for secure execution
Planning to get console output for z/VM
Planning for TCPIP networks
Planning for z/VM® TCPIP networks
Planning for KVM TCPIP networks
Planning for persistent storage and fabric zoning
Planning for persistent storage
Planning for persistent storage through Fibre Channel Protocol
Planning for FCP PCHID capacity
Planning for FCP multipath template
Planning for IBM Storage® DS8000®
Planning for IBM Storage FlashSystem
Fibre Channel switch zoning
Planning for Fibre Channel and Network Connections
Boot from volume
Planning for persistent storage with software defined storage
Planning for IBM Storage Scale System
Storage scheduler
Storage filters
Storage weighers
Planning for advanced storage scheduler
Planning for allowed or blocked storage providers per compute host
Planning for fabric zoning
Planning for license
Planning for security
Service users and permissions
Ports used by IBM Cloud Infrastructure Center
Security header
Package signing
Required packages
Notes for KVM console access
Security Recommendations
Planning for multi-node cluster
Swift object storage planning
Setting up the environment
Setting up z/VM®
Special settings for the compute user IDs
Configuring z/VM® SMAPI
Setup z/VM® VM directory management tool
Setting up z/VM® VSWITCH
Profile management on z/VM
Setting up external security manager (ESM)
Setup N_Port Identifier Virtualization (NPIV) enabled FCP channels
Setup environment for z/VM Live Guest Relocation(LGR)
Setting up KVM
Management node and compute node setup and verification
Set up the management and compute node
Verify management node and compute nodes network connectivity
Security configuration
Enhancing TCP SYN flood attack
Fixing possible portmapper UDP-Based amplification attacks
Fixing unrestricted administrative ports
Fixing weak SSH ciphers in use
Set up storage controller and SAN switch
Installing, uninstalling and upgrading
z/VM® Standalone
Installing IBM Cloud Infrastructure Center
Prerequisites
Installation
Post-installation validation
Install the IBM Cloud Infrastructure Center feature of Infrastructure Suite for z/VM and Linux
Upgrading IBM Cloud Infrastructure Center from 1.2.3 or 1.2.4 to 1.2.5
Uninstalling IBM Cloud Infrastructure Center
RHEL Linux in LPAR mode/ RHEL KVM standalone management node
Installing IBM Cloud Infrastructure Center
Prerequisites
installation
Post-installation Validation
Upgrading IBM Cloud Infrastructure Center from 1.2.3 or 1.2.4 to 1.2.5
Uninstalling IBM Cloud Infrastructure Center
Multi-node cluster
Initiate installation through operation manager
Create multi-node cluster defintion
Deploy multi-node cluster
Post multi-node cluster deployment tasks
Convert IBM Cloud Infrastructure Center standalone deployment to multi-node cluster deployment
Upgrade from previous versions of the IBM® Cloud Infrastructure Center
Upgrade from previous versions of IBM Cloud Infrastructure Center
Applying an ifix to the multinode cluster
Uninstallation of multi-node cluster
Uninstall operation manager
IBM Cloud Infrastructure Center upgrade validator overview
PR/SM Standalone
Add host
Locate your IODF file
Deleting stale jobs in HCD
Build production IODF from the work IODF
Deleting a WORK IODF via HCD
Define resource list configuration
Platform management through Terraform
Limitations
Administrator tasks
Getting started as an administrator
Verifying your environment
Monitoring resource health
Split IBM Cloud Infrastructure Center Cluster
Working with hosts
Adding hosts
Note for z/VM host management
Note for KVM host management
Add host with SSH key
Editing hosts
Removing hosts
Understanding schedulers and placement policies
Note for host management of multi-node cluster
Working with networks
Adding a network
Editing a network
Removing a network
Network topology rendering
Working with a security group(RHEL KVM Only)
Additional setup for RHCOS virtual machine on KVM hypervisor
Working with layer 3 networks
Working with Geneve networks
Working with routers
Create, Edit, and Remove routers
Set and Clear Gateway
Add and Remove interfaces
Add and remove static routes
Working with floating IPs
Allocate and release a floating IP
Associate and disassociate a floating IP
Multiple flats configuration
KVM OSAs or HiperSockets network cards Multiple Flat networks(the first time configuration)
KVM OSAs or HiperSockets network cards Multiple VLAN networks that pass through different Network Cards(the first time configuration)
KVM RoCE express multiple flat networks
KVM RoCE Express Multiple VLAN networks that pass through different network cards
z/VM multiple Flat networks(the first time configuration)
z/VM multiple VLAN networks pass through different z/VM vSwitches(the first time configuration)
KVM multiple Flat and VLAN networks pass through different network cards(configured already)
RoCE Express configuration
Setup MacVTap guide
Remove the KVM MacVTap configuration guide
Setup OVN guide
Remove the KVM OVN configuration guide
Setup RoCE Express multiple adapters
Working with images
Creating images
Creating images from ISO for z/VM
Requirements for creating images used to deploy virtual machines from volume
Requirements for creating images used to deploy virtual machines from ECKD or FBA disk
Installation and configuration of IUCV service in the Linux server
Installation and configuration of zvmguestconfigure on the Linux server
Installation and configuration of cloud-init on the Linux server
Configuration of LVM partition on the Linux server
Creating images from ISO for KVM
Linux requirements
Installation and configuration of cloud-init on the Linux server
Generate an Secure Execution image
RHCOS images creation(download)
Uploading Images
Uploading Image to Deploy Virtual Machines from Local Storage
Uploading Image for Booting Virtual Machines from Volume
Export and Import Images
Export an image to a backup service
Import an image
Creating deploy templates
Downloading Images
Working with virtual machines
Create virtual machines
Cross projects vm management
Create virtual machines with indicated userid for z/VM
Create virtual machines with a specified maximum memory size for z/VM
Delete a virtual machine
Export virtual machine info into csv file
Live migrate virtual machines in KVM
Live guest relocation in the z/VM
Cold migrate virtual machines
Resizing virtual machines
Suspend, pause and resume virtual machines
Lock and unlock virtual machines
Attach and detach the NIC (Network Interface Card) to/from the virtual machines
Capture virtual machines
Backup and restore virtual machines
Console access
Edit security groups of a virtual machine(RHEL KVM only)
Add floating IP address to a network interface of a virtual machine with Geneve networks(RHEL KVM only)
Managing requests of virtual machine deployment
Optimize KVM virtual machines
Create virtual machines with a custom user direct for z/VM
Manage I/O thread for KVM virtual machines
Rebuild Volume-Backed z/VM Virtual Machine
Command to add multiple disk pools
Deploy image with multiple disk pool
Managing existing virtual machines (onboarding) in z/VM®
Managing existing virtual machines (onboarding)
Live Resize and Live Migrate virtual machines (onboarding)
Limitations for managing existing virtual machines
Managing existing virtual machine (Onboarding) blocklist
Onboarding externally created virtual machines
Working with virtual machines that are booted from IBM Storage Scale volumes
Create KVM virtual machine that is booted from IBM Storage Scale volume
Virtual machines backup
Virtual machines restore
Virtual machines backup delete
Adding a second disk to a virtual machine
Working with Storage
Add Storage Provider into IBM Cloud Infrastructure Center
Edit Storage Provider in IBM Cloud Infrastructure Center
Working with IBM Storage Scale Block Storage
Setup FCP device(s) for agent node of storage provider
Note for storage management of multi-node cluster
Work with volumes in Storage Provider
Create volumes
Attach volumes to virtual machines
Detach volumes from virtual machines
Delete volumes
Edit volumes
Manage volumes
Ansible helper script to modify bootloader Configuration
Working with Consistency Groups and consistency group snapshots
Implementation mechanism of consistency groups and snapshots
Considerations of working with consistency groups and snapshots
Working with consistency groups
Working with consistency group snapshots
The naming conventions in the IBM Cloud Infrastructure Center
Remove Storage Provider from IBM Cloud Infrastructure Center
Storage migration utility for z/VM
Working with fabric zoning
Configure and customize
Configuring multi-tenancy and credential management
Managing projects
Creating a project and assign user roles to projects
Setting project quotas
Editing project quotas
Setting project policies
Non-root user authorization on IBM Cloud Infrastructure Center
Configuring operating system users and groups
Managing roles
Configuring LDAP
Configure Multi-Factor authentication for a user
Customizing IBM Cloud Infrastructure Center
Collocation rules
Compute templates
Storage templates
Customizing SSH Session Pool for Connecting to IBM Storage FlashSystem
FCP multipath templates
Replacing Messaging Certificates
Replacing multi-node Cluster Certificate
Consume Messaging Notification
Resource overcommit ratio and maximum allocation unit
Configuring email
Changing the IP address or hostname of the management node
Monitoring
Monitoring first steps
Enabling and disabling monitoring services
Enabling and disabling Monitoring services in multi-node cluster
Enabling and disabling event service
Enabling and disabling event services in multi-node cluster
Planning for monitoring data storage
Running SMAPI health check
Working with multi-node cluster
Management view
Configure users for multi-node cluster deployment
Swift Object Storage for multi-node cluster deployment
Add node to single node cluster
Replace failed nodes of the multi-node deployment
Changing Virtual IP address
Synchronize configuration of multi-node cluster deployment
Auto synchronize policy files
Enable storage provider with high availability
Fencing agent configuration
Fencing agent configuration for z/VM management nodes
Fencing agent configuration for KVM management nodes
Backing up IBM Cloud Infrastructure Center data
Backup and restore standalone environment
Backing up
Disaster recovery for stand-alone environment
Recovering the IBM Cloud Infrastructure Center data
Backup and restore multi-node cluster environment
Backing up multi-node cluster
Recover data for multi-node cluster deployment
License tracking
License tracking metrics
Configuring your license tracking details
Running the IBM License Metric Tool(ILMT) disconnected scans
User tasks
Working with virtual machines
Creating virtual machines
Creating z/VM® virtual machines without operating system installed
Creating RHCOS virtual machines
Tips for managing virtual machines status
Virtual machine Unknown status
Managing requests
Viewing usage information
IBM® Cloud Infrastructure Center commands and descriptions
Setting environment variables
icic-config image file-store-datadir
icic-config image swift-store
icic-config storage backup-service
cross_az_attach
icic-config storage fc-zone zone-masking
icic-config network manage-service
icic-migrate command
icic-opsmgr
icic-config storage ds8000 --cg-lss
icic-fcp-pchid-max-capacity
icic-sync-fcp
icic-config compute unmanage
icic-config compute volume-migration-state
icic-detect-fcp
icic-config storage migration
icic-config compute storage-provider
icic-config storage volume-creation
icic-utility inspect icic_files
icic-utility inspect icic_rpms
icic-config storage pool_rename
icic-config compute validate_onboard
icic-config onboard-volumes
icic-config network physical_vswitch_mappings
icic-services
Troubleshooting
Techniques for troubleshooting problems
Collecting diagnostic data
Getting fixes from Fix Central
Steps to trouble shooting for typical problems on z/VM®
Trouble shooting and solutions
Troubleshooting of general questions
Add a volume to a consistency group in IBM Storage® DS8000®, the volume WWN changes, or an error is reported
Overcommit ratio for virtual cpu, virtual memory and disk
Solution to backup failure due to fernet keys rotation
Fail to create a consistency group or a consistency group snapshot due to reaching the limit
DBDeadlock during neutron operations
Health status changes to 'Attention' when password of host is changed
Health status is not 'OK'
HTTPd service fail to start
Add host failed for connection timeout to the KVM(or z/VM®)
Services are down after management or compute node is rebooted
Logs receives warning of yacc table
Image is stuck at "queued" state
Create multiple partitions when creating a new virtual machine
Reset virtual machine state in order to delete it
Nova compute log can get the password info from the user_data
Neutron policy restrictions don't work for admin users
The Keystone service is temporarily unavailable 503 Service Unavailable
Failed to ssh to a deployed VM with correct username/password
Failed to start rabbitmq-server.service
No allocation candidates can be found
Switch setting for enable/disable public image deploy template different behavior.
Switch the setting for enable/disable shutoff of virtural machines which are expired.
Extend volume stats timeout to resolve the storage provider health unstable issue
Environment Checker OPTION LNKNOPAS Failure
Use application credential in IBM Cloud Infrastracture Center
IBM Cloud Infrastructure Center GUI login failed with 504 error code after management node is restarted
Change a virtual machine's IP after it is deployed
Timeout for Swift object query with high workloads after configured Gnocchi with Swift storage backend
ZooKeeper can not be connected error
Login failures with Error 500 when LDAP password expiration or changed.
Host is down due to no local node identity found
Troubleshooting of install and upgrade
Failed to add two hosts with the same hostname
Failed to remove the host after editing it without display name
Failed to yum update
Failed to add host
Install process is stuck with MSGXXX
Resolution to recover failed fixpack update installation
Resolution to recover failed standalone upgrade
Resolution how to recover failed multi-node upgrade
Resolution to recover Environment Checker after upgrade
Failure Configuring or Installing MariaDB
Troubleshooting of KVM
KVM hosts move to attention state when the KVM standalone management node is upgraded from 1.2.1 to 1.2.3 build
Add host failed for ovsdb-server prevented by tmpfs
Binding port fails during the VM creation
IBM Storage Scale (GPFS) storage commands need input login password
No network setup after RHCOS created on KVM
RHCOS OS root filesystem is not resized
Virtual machine's host is not changed after live migration and no live migration completion messages pops up on the UI
Fail to add host due to potential network loop
Disk name of attached volume changed after reboot
UI warning message "Existing access key files" for add one KVM host
Add host nova uid/gid Warning information and fix
Virtual machine deployment with a DHCP network succeeds, but no IP address is configured in the VM
Virtual machine deployment fails for Permission denied of deleting the file console.log
VM status become Unknown after compute node reboot
Not able to reach the KVM deployed virtual machine
Not able to restore a KVM host using the backup archive with different host network capabilities
Failed to deploy a KVM virtual machine if an OSA or HiperSocket network card is reused by OVN and MacVTap
Replace RoCE Express interface or bonded interface for a virtual machine on a KVM host
Recover broken MacVTap VM interfaces on the KVM host
Not able to ssh login the deployed KVM virtual machine within an environment where IBM Storage Scale is setup
Virtual machine migration/resize/deploy fails with No space left on device
Compute node host name changed lead to deployment failure
Neutron ovn metadata agent failed to start
OpenVSwitch bridge flow tables are missing
Set VIP(Virtual IP address) for some KVM virtual machines
Troubleshooting of z/VM®
SMAPI get -109 return error
Compute logs contain 8/3016 error or 8/3017 error
SMAPI get -100 return error
compute node memory percent 0.0 is returned by gnocchi
Conflict address of FCP device and NIC address
Onboarding virtual machines - cannot onboard network information successfully
Failed to get console logs of virtual machines
A virtual machine with more vCPUs does not have more computing power issue
Failed to create Virtual Interface
Failed to deploy VM when available disk is low
Deploy virtual machines fails due to "Invalid input for field/attribute userid"
Frequent DIRMAINT call duiring IBM® Cloud Infrastructure Center idle status
Size of root file system of virtual machine less than the specified Disk size in Compute Template
Deploy or delete of virtual machines fails due to "Failed to synchronize the placement service"
Deploy of VM failed with error "Failed to add mdisk to userid XXXXXX"
Virtual machine deployed from captured image enters emergency mode
deployment task stuck in 70% when deploying virtual machine from a DASD image
Deploy virtual machine failed with compute node is not able to connect to the virtual machine error
Manage Virtual Machine wrongly list the existing virtual machines
Virtual machines still exist after deleting from UI
Virtual machine's directory entry contents fetching from SMAPI are out of date
VM can't be created due to request denied
Failed to live resize memory due to memory size to be increased greater than maximum reserved memory size
Ports locked after VM deployment failure
Skip format step of Ephemeral disk
Update LOGONBY statement in USER DIRECT of deployed VM
Network connection of a RHEL 8.2 virtual machine is down after live guest relocation(LGR)
Compute node has additional disk linked after abnormal terminate
Manually extend the z/VM LVM partition.
Storage in a z/VM guest does not show correctly
Live guest relocation fails in the z/VM 7.3.0.
VM status becomes Unknown after it is manually relocated
Renew the using ssh key pairs for z/VM hypervisor
IUCV crash triggered by Linux Kernel Module installation
Troubleshooting of z/VM® volume related questions
Volume related operations failed with error, wwpn already mapped to an existing host in the backend
Failed to setup FCP device(s) on the agent node of storage provider
IBM Storage DS8000 user keeps being locked even after password reset
Leftover inside the virtual machine when a in-use volume is detached
Not enough available FCP devices found
No space left in image_conversion_dir path (/var/lib/cinder/conversion) while fetching image
Booting virtual machines from volumes or detaching volumes failed because of volume protection
BFV failed because devNode doesn't exist
booting from volume failed because failed to copy image to volume and traceback errors returned
booting from volume failed because no spaces left on device
booting from volume or creating bootable volume failed because no FCP online on storage provider agent node
Deploying virtual machines from volume failed with error: timeout receiving packet
Clean up leftovers of multipath
Failed to attach multiple volumes simultaneously
Create host failed when creating bootable volumes or booting virtual machine from volume
Booting virtual machine from volume failed with error: Refresh bootmap fails
Virtual machine deployed from captured image failed with invalid snapshot
Deploy VM from SCSI image failed with error: qemu-img: Unable to initialize gcrypt
Volume related operation failed because the volume is fast formatting
Problematic FC connection may fail detaching volumes
FCP devices are being used in z/VM host but unallocated in the IBM Cloud Infrastructure Center
FCP devices are not found in z/VM
Unable to find a Fibre Channel volume device
How to add volumes from a new storage template to an existing consistency group
Create consistency groups against IBM Storage® DS8000® failed with no LSS is configured
Failed to delete an in-use volume that is attached to a non-exist VM
Failed to add a second IBM Storage Scale(GPFS) storage provider
Fail to show an FCP multipath template with 'PCHID for the CHPID xx not found'
Fail to add volumes to a consistency group against IBM Storage® DS8000® if CKD type LSS is selected
A compute node frequently initiates SSH sessions to an IBM Storage FlashSystem, causing it to block SSH connections.
Instance deploy failed with Ubuntu snapshot or Ubuntu image where the kernel version is previously Ubuntu 20.04.
Troubleshooting of multi-node cluster
Failed to connect to the host via ssh: mm_send_fd
Galera fails to start after rebooting one of the management node due to 'Unable to detect last known write sequence number'
Failed to save data to Swift Object Storage
Failed to deploy a KVM virtual machine after the primary node is down or after replacing a node.
Operation that failed because UID or GID of glance user already exists
Pacemaker fails to start galera
Converting to multi-node fails at configure db root password on all nodes
Gnocchi component does not work properly after add node operation
Troubleshooting of PR/SM
Onboarding of multiple or all partitions into IBM Cloud Infrastructure Center fails
Known issues or limitations
Ephemeral disks are not resized
Swap and ephemeral disk disappeared after virtual machine resize
Live resize for z/VM guests fails at memory resize, but CPU count is still increased
East-west network flow does not work for KVM guests whose network interfaces are sharing the same OSA adapter
OpenSSH 8.0 – configuration notes for port 22
Weak KEX, MAC and Hostkey Algorithms
Updating base OS from RHEL9.4 to RHEL9.6
Break and resume does not work for first level hypervisor
Unmanage volume results in error
FAQ of IBM Cloud Infrastructure Center
FAQ of IBM Cloud Infrastructure Center on z/VM®
z/VM® related settings reference
FAQ of IBM Cloud Infrastructure Center on KVM
IBM Cloud Infrastructure Center APIs
Getting started with APIs
Using self_service user to deploy virtual machines through API
Authenticating with IBM Cloud® Infrastructure Center
Create network and subnet
Deploy servers
Register a compute host
Register a storage host
Register images
Adding a second disk to a virtual machine
Northbound REST APIs
IBM Cloud Infrastructure Center services
Identity (Keystone) APIs
Supported OpenStack Identity (Keystone) APIs
Compute (Nova) APIs
Compute host registration
FCP multipath templates APIs
Capture a VM
Cold migrate a VM
Live migrate a VM
Resize a VM
Supported OpenStack Compute (Nova) APIs
Nova placement policy
Automatic flavors
Changes for the OpenStack API of booting from volume
Attach multiple volumes to a virtual machine
Changes for the OpenStack volume attachment API
Get allocation details of all PCHIDs
Block storage (Cinder) APIs
Supported OpenStack block storage (Cinder) APIs
Retrieve additional metrics of storage providers
Manage Existing Volumes
Storage controller registration
SAN fabric registration
Supported volume type extra-specs
Images Used to Boot From Volume
Create bulk volumes
Clone consistency groups or create consistency group from group snapshot
Manage Virtual Machine Backup/Restore Operation
Image (Glance) APIs
Supported OpenStack image APIs
Create image API
Network (Neutron) APIs
Monitoring (Telemetry) APIs
Supported Gnocchi APIs
Supported OpenStack Event APIs
Create a Deploy Template
Validate environment APIs
Integration with upper layers via OpenStack API
Chargeback integration
Terraform resources
Instana Integration
vProtect Integration
Sample configuration
z/VM - Installation Sample
Sample config for availability zone and host group
Sample config for collocation rules
Reference architecture for z/VM
Reference architecture for KVM