Customizing the cluster with the config.yaml file

The config.yaml file contains all the configuration settings that are needed to deploy your cluster.

From the config.yaml file, you can customize your installation by using various parameters.

The config.yaml also contains a list of the Docker images that are pulled from Docker Hub by the installer during the IBM® Cloud Private-CE (Community Edition) installation process. For an IBM Cloud Private-CE (Community Edition) installation, you can also store these installation images in a private image registry in place of pulling directly from Docker Hub. If images are stored in a private image registry, update the config.yaml file to point to the installation images in your private image registry. For an IBM Cloud Private installation, these installer Docker images are commented out since the images are available in the downloaded installer package.

Note: Before you update any section of the config.yaml file, review the inline comments in that section.

You can set or update the following parameters by modifying the config.yaml file.

  1. Open the /<installation_directory>/cluster/config.yaml file.

  2. Add or modify the parameters and values. The format for defining a parameter and values is <parameter_name>:<value>.

General settings

Table 1. General settings
Parameter Description Default value
cluster_name The name of your cluster.

In a multiple cluster environment, specify a distinct name for each cluster.

cluster_name must consist of lowercase alphanumeric characters only.

 mycluster
cluster_CA_domain Specify the certificate authority (CA) domain to use in your cluster. {{ cluster_name }}.icp
cluster_domain A Kubernetes internal DNS domain name. cluster.local
offline_pkg_copy_path The directory to hold the temporary installation files during offline installation. This location must have at least 50 GB of available disk space.

If your /tmp directory has less than 50 GB of space, you must set this parameter to a location that has the available disk space requirement.

 /tmp
firewall_enabled If set to true, the installer keeps the local operating system firewall running and opens the ports that IBM Cloud Private needs on the firewall. false
wait_for_timeout This parameter specifies the default timeout value for operations. A setting of 3600 is ideal in most environments. 600
management_services Use this parameter to disable the management services. You can disable the following services: custom-metrics-adapter, image-security-enforcement, istio, metering, monitoring, service-catalog, storage-minio, storage-glusterfs, and vulnerability-advisor. For more information about management services, see Enabling and disabling IBM Cloud Private management services.

istio: disabled

vulnerability-advisor: disabled

storage-glusterfs: disabled

storage-minio: disabled
router_http_port The http port that is used by the IBM Cloud Private management ingress, which acts as a proxy for all IBM Cloud Private management services. 8080
router_https_port The https port that is used by the IBM Cloud Private management ingress, which acts as a proxy for all IBM Cloud Private management services. 8443
ingress_http_port The http port that is used by the ngnix ingress controller, which acts as a proxy for all user services. 80
ingress_https_port The https port that is used by the ngnix ingress controller, which acts as a proxy for all user services. 443
loopback_dns Set this parameter to true if the boot node uses a loopback IP, for example an IP address that starts with 127, as the DNS server. false

Kubernetes settings

Table 2. Kubernetes settings
Parameter Description Default value
kube_apiserver_extra_args Sets extra apiserver configurations for Kubernetes. Accepts a list of apiserver arguments that are provided in --key=value format. None
kube_apiserver_secure_port Sets the Kubernetes apiserver secure port. 8001
kube_controller_manager_extra_args Sets extra controller configurations for Kubernetes. Accepts a list of controller arguments that are provided in --key=value format. None
kube_proxy_extra_args Sets extra proxy configurations for Kubernetes. Accepts a list of proxy arguments that are provided in --key=value format. None
kube_scheduler_extra_args Sets extra scheduler configurations for Kubernetes. Accepts a list of scheduler arguments that are provided in --key=value format. None
kubelet_extra_args Sets extra configurations for kubelet. Accepts a list of kubelet arguments that are provided in --key=value format. For example, to set the maximum number of pods that can run on a kubelet, set the following configuration:kubelet_extra_args: ["--max-pods=110"] None
auditlog_enabled Enables the Kubernetes audit log, which records the chronological sequence of activities by individual users, administrators, or other components of the system that modified the system. Set this parameter to true to enable the audit log. false

Log settings

Table 3. Log settings
Parameter Description Values Default value
metrics_max_age Sets the maximum number of days to store system and application metrics. Metrics older than this specified number of days are removed. Metrics are removed at 23:59 on the specified day. Number of days 1
kibana_install IBM Cloud Private Version 3.1.0 deploys Kibana by default and integrates it with the IBM Cloud Private management console. If you do not want to install Kibana in your cluster, add kibana_install: false in the config.yaml file before you install your cluster. true or false true
logs_maxage Sets the maximum number of days to store logs in Elasticsearch. Number of days 1

Network settings

Table 4. Network settings
Parameter Description Values Default value
network_type Type of network management in your cluster. calico or nsx-t calico
network_helm_chart_path Absolute path to the network helm chart. None None
calico_ipip_enabled Allows Calico to be run on IP over IP mode. This setting is needed when worker nodes are in different subnetwork and BGP is not enabled in routers between the worker nodes. This setting is also needed in some cloud environment such as OpenStack, where virtual machines aren’t allowed to work as routers. true or false true
calico_tunnel_mtu The IPIP for Calico has a default MTU of 1430. If the main interface of your host has an MTU that is less than 1450, Calico IPIP has poor performance. Set the MTU such that the MTU of the host main interface minus the default MTU of the Calico IPIP tunnel is greater than or equal to 20. Positive integers 1430
network_cidr The IPv4 network to use for the entire network. This value must be in CIDR format. When you create a network_cidr, ensure that you select an IP range that does not conflict with the existing host network or with the service_cluster_ip_range. In most environments, you can use the default value.
Important: You cannot modify this parameter after installation since all the platform management services will be using these IP ranges and the Kube-proxy is also aware of this network cidr.		 IP address in CIDR format 10.1.0.0/16
cluster_lb_address In an environment that has multiple network interfaces (NICs), use cluster_lb_address to set a public or external IP address for the management services in your cluster. You can specify a fully-qualified domain name instead of the IP address.
This public address is assigned to the master node, used to access the console, and also used to configure kubectl.
In an HA environment, cluster_lb_address masks the cluster_vip as the leading master IP.
  • IP address
  • Fully-qualified domain name
  • OpenStack floating IP address
 None
proxy_lb_address In an environment that has multiple network interfaces (NICs), use proxy_lb_address to set a public or external IP address that is to be used by the NodePort resource to allow external access to services. You can specify a fully-qualified domain name instead of the IP address.
  • IP address
  • Fully-qualified domain name
  • OpenStack floating IP address
 None
calico_ip_autodetection_method You can configure the Calico node to auto-detect the IP address that is used to route between nodes. You can use one of the following methods:
  • calico_ip_autodetection_method: first-found: This method uses the first valid IP address on a valid interface that is found first.
  • calico_ip_autodetection_method: interface: This parameter accepts a comma-separated list of regular expression names as value. It uses the first IP address that is found on the specified interface.
Examples:
  • calico_ip_autodetection_method: interface=eth0
  • calico_ip_autodetection_method: interface=eth.*
  • calico_ip_autodetection_method: interface=eth.*,ens.*
  • calico_ip_autodetection_method: can-reach=<remote IP address or host name>: The can-reach method uses your local routing to determine the IP address that is used to reach the specified destination. This parameter accepts a remote IP address or domain name as value.
    • Note:
  • In an environment that has multiple network interfaces (NICs), use the can-reach method to specify the network to be used for your workloads. In IBM Cloud Private, you can set calico_ip_autodetection_method: can-reach=<Master node IP address>.
  • The network interface names cannot contain the following strings: "docker.*", "cbr.*", "dummy.*", "virbr.*", "lxcbr.*", "veth.*", "lo", "cali.*", "tunl.*", or "flannel.*".
  • Some IPs are not recognized by Calico. Ensure that your interfaces do not have IPs in the following ranges:
    • 10.0.2.15/24 - this IP range is the default vagrant/virtualbox NAT interface address range.
    • 92.168.122.* - this IP range is the default libvirt VM interface address range.
    		•
    • first-found

    • interface=INTERFACE-REGEX

    • can-reach=<remote IP address or domain name>

    		 can-reach={{ groups['master'][0] }}
    service_cluster_ip_range The Kubernetes service cluster IP range. This configuration allocates a block of IPs for services. When you create a service_cluster_ip_range, ensure that you select an IP range that does not conflict with the existing host network or with the network_cidr. service_cluster_ip_range is a virtual network. In most environments, you can keep the default value.
    You can also extend the service cluster IP range post installation of your cluster, see Extending the services IP range.    		 IP address in CIDR format 10.0.0.1/24

    Docker settings

    Note: These configurations can be set for the supplied IBM Cloud Private Docker packages only. See IBM Cloud Private Docker packages.

    Table 5. Docker settings
    Parameter Description Format Default Value
    docker_version Specify the version of Docker that you want to install.

    If you want to install Docker by using the IBM Cloud Private installer, a package for the required version must be available in the /<installation_directory>/cluster/runtime-engine directory.

    		 string 18.03.1
    install_docker Allows the installer to automatically install Docker on your cluster nodes. true or false true
    docker_env Sets the environment for Docker. For example, you might configure an https_proxy location if Docker runs behind a firewall. The environment location is stored in the /etc/systemd/system/docker.service.d/docker-env-icp.conf file. "HTTP_PROXY=http://httphost:port/", "NO_PROXY=localhost,127.0.0.1" None
    docker_config Sets Docker configuration parameters. For more information about Docker configuration parameters, see dockerd Opens in a new tab docker_config:
         log-opts:
            max-size: "100m
            max-file: "10"
         storage-driver: devicemapper

    Proxy HA settings

    Table 6. Proxy HA settings
    Parameter Description Values Default
    vip_manager The service that manages the virtual IP (VIP) on the master, proxy, or master and proxy nodes. If you configure HA for master or proxy nodes, you can set this value. You set this value only once.
    • etcd
    • keepalived
    		 etcd
    proxy_vip_iface Sets the virtual IP interface for a proxy node HA environment. eth0 n/a
    proxy_vip Sets the virtual IP address for a proxy node HA environment. 127.0.1.1
    Do not specify the subnet in the IP address.    		 n/a

    Master HA settings

    Table 7. Master HA settings
    Parameter Description Values Default
    vip_manager The service that manages the virtual IP (VIP) on the master, proxy, or master and proxy nodes. If you configure HA for master or proxy nodes, you can set this value. You set this value only once.
    • etcd
    • keepalived
    		 etcd
    cluster_vip Sets the virtual IP address for IBM Cloud Private HA environment. 127.0.1.1
    Do not specify the subnet in the IP address.    		 n/a
    vip_iface Sets the virtual IP interface for IBM Cloud Private HA environment. eth0 n/a

    User settings

    Table 8. User settings
    Parameter Description
    ansible_user
    ansible_ssh_pass
    ansible_become
    ansible_port
    ansible_become_pass    		 IBM Cloud Private uses the ansible_user and ansible_ssh_pass parameter values to access your cluster nodes during installation.
    If you use a non-administrator account that has sudo privileges to connect to a master or worker node, set the ansible_user to the user name and ansible_become to true.
    If you run sudo with a password, you must set the ansible_become_password parameter to the value of your non-root (sudo user) password. This variable is optional if you set NOPASSWD in the /etc/sudoers file.
    If you have a customized ssh port, you can also set the ansible_port parameter in the <installation_directory/cluster/hosts file. The default port is 22.
    To configure these parameter values, see Configuring password authentication for cluster nodes.
    default_admin_user Sets a customized cluster administrator user name. If LDAP is enabled and you have an LDAP user with the user name admin, change the default_admin_user parameter value to something else to avoid a conflict of cluster administrator name and LDAP user name.
    default_admin_password Sets a customized cluster administrator password. If LDAP is enabled, this parameter sets the LDAP administrator password.

    GlusterFS settings

    Table 9. GlusterFS settings
    Parameter Description
    storage-glusterfs Provision storage on dedicated nodes.

    To set up GlusterFS, you must set several parameters. See, Configuring GlusterFS during IBM Cloud Private installation.

    Cloud Provider settings

    Table 10. vSphere Cloud Provider settings
    Parameter Values Description
    cloud_provider vsphere

    vSphere Cloud Provider settings

    Sets up vSphere Cloud Provider.

    To set up vSphere Cloud Provider, you must set several parameters. See Configuring a vSphere Cloud Provider.

    Note: If you want to configure a vSphere Cloud Provider after you install your IBM® Cloud Private cluster, ensure that you set kubelet_nodename: hostname in the <installation_directory>/cluster/config.yaml file during IBM® Cloud Private installation.
    aws

    AWS Cloud Provide settings

    Sets up AWS Cloud Provider.

    To set up AWS Cloud Provider, you need to set two parameters: cloud_provider to aws and kubelet_nodename to nodename.

    For more details about setting up IBM Cloud Private-CE (Community Edition) on an Amazon Web Services (AWS) cloud platform, see Run IBM Cloud Private on Amazon Web Services (AWS) cloud platform Opens in a new tab.

    Encrypting cluster data network traffic with IPsec

    Table 12. Encrypting cluster data network traffic with IPsec
    Parameter Description
    ipsec_mesh
    enable: true Enables IPsec.

    To set up IPsec, you must set several parameters. See Encrypting cluster data network traffic with IPsec.

    Vulnerability Advisor settings

    Table 13. Vulnerability advisor settings
    Parameter Description Default value
    management_services By default, the vulnerability advisor is disabled.

    To enable the vulnerability advisor, set vulnerability-advisor: enabled in the management_services section.

    Integrating VMware NSX-T 2.3 with IBM Cloud Private

    Table 14. Integrating VMware NSX-T 2.3 with IBM Cloud Private
    Parameter Description
    network_type NSX-T
    nsx_t You must set several parameters to integrate VMware NSX-T 2.3 with IBM Cloud Private. See Integrating VMware NSX-T 2.3 with IBM Cloud Private.

    Environment isolation

    Table 15. Environment isolation
    Parameter Description
    isolated_namespaces: [] Use this parameter if you want to isolate a namespace. Add a namespace and allot a host group with dedicated worker nodes to the namespace. The syntax is {namespace: hostgroup}. For example, [{production: prod}, {devops: dev}]. The namespaces are created during IBM Cloud Private installation.
    isolated_proxies: [] Use this parameter if you want to allot proxy node host groups to a namespace. Add a namespace and allot a host group with dedicated proxy nodes to the namespace. The syntax is {namespace: hostgroup}. For example, [{production: prod}, {proxydevops: dev}]. If you are specifying a namespace that is not defined in the isolated_namespaces parameter, you must manually create the namespace after you install IBM Cloud Private.

    etcd settings

    Table 16. etcd settings
    Parameter Description Default value
    etcd_extra_args Sets extra etcd configurations. Accepts a list of arguments that are provided in --key=value format. ["--grpc-keepalive-timeout=0", "--grpc-keepalive-interval=0", "--snapshot-count=10000"]
    etcd_data_dir etcd data directory. For example, /var/lib/etcd. /var/lib/etcd
    etcd_wal_dir etcd write-ahead (wal) log directory. For example, /var/lib/etcd-wal. You can set the directory to a centralized remote log directory for persistent logging. The default value that etcd uses is --max-wal=0, which means that the maximum number of wal files that are retained is unlimited. To manually purge down to five, edit the /etc/cfc/pods/etcd.json file to set --max-wals=5 on each etcd node. /var/lib/etcd-wal

    Istio addons security settings

    Table 17. Istio addons security settings
    Parameter Description Default value
    istio_addon Configure Istio addons security settings.
    grafana Configure Grafana Istio addon to view service mesh traffic data. Configure the user name and password to access the interface. None
    kiali Configure Kiali Istio addon to view microservices data in the Istio service mesh. Configure the user name and password to access the interface. None

    NGINX ingress controller

    Table 18. NGINX ingress controller
    Parameter Description
    nginx-ingress:ingress Customize the NGINX ingress controller configuration. The ingress controller is enabled by default. For a list of configurable parameters, see https://github.com/kubernetes/ingress-nginx/blob/nginx-0.16.2/docs/user-guide/nginx-configuration/configmap.md#keep-alive-requests Opens in a new tab.
    config:
    disable-access-log Disables access log. Set the value to true or false.
    keep-alive-requests Sets the maximum number of requests that can be served through one keep-alive connection. For example, 10000.
    upstream-keepalive-connections Activates the cache for connections to upstream servers. The connections parameter sets the maximum number of idle keepalive connections to upstream servers that are preserved in the cache of each worker process. When this number is exceeded, the least recently used connections are closed. For example, 64.
    worker-processes Sets the number of worker processes. For example, 2.
    extraArgs:
    publish-status-address Specify the address that the ingress controller must use. For example, "{{ proxy_external_address }}".
    enable-ssl-passthrough Send TLS connections directly to the pod instead of allowing NGINX to decrypt the communication. Set the value to true or false.

    Minio settings

    Table 19. Minio settings
    Parameter Description
    storage-minio To configure Minio, you must set several parameters. See, Configuring Minio during IBM Cloud Private installation.