Customizing the cluster with the config.yaml file

The config.yaml file contains all the configuration settings that are needed to deploy your cluster.

From the config.yaml file, you can customize your installation by using various parameters.

The config.yaml also contains a list of the Docker images that are pulled from Docker Hub by the installer during the IBM® Cloud Private-CE (Community Edition) installation process. For an IBM Cloud Private-CE (Community Edition) installation, you can also store these installation images in a private image registry in place of pulling directly from Docker Hub. If images are stored in a private image registry, update the config.yaml file to point to the installation images in your private image registry. For an IBM Cloud Private installation, these installer Docker images are commented out since the images are available in the downloaded installer package.

Note: Before you update any section of the config.yaml file, review the inline comments in that section.

You can set or update the following parameters by modifying the config.yaml file.

  1. Open the /<installation_directory>/config.yaml file.
  2. Add or modify the parameters and values. The format for defining a parameter and values is <parameter_name>:<value>.

General settings

Table 1. General settings
Parameter Description Default value
cluster_name The name of your cluster.

In a multiple cluster environment, specify a distinct name for each cluster.

cluster_name must consist of lowercase alphanumeric characters only.

mycluster
cluster_CA_domain Specify the certificate authority (CA) domain to use in your cluster. {{ cluster_name }}.icp
cluster_domain A Kubernetes internal DNS domain name. cluster.local
offline_pkg_copy_path The directory to hold the temporary installation files during offline installation. This location must have at least 30 GB of available disk space.

If your /tmp directory has less than 30 GB of space, you must set this parameter to a location that has the available disk space requirement.

/tmp
tiller_http_proxy This parameter allows access to the IBM Cloud Private catalog in environments that run behind a firewall or environments that do not have access to the internet.
To enable access to the IBM public Helm repository, set this parameter before installation of your cluster.
Accepted values: Standard HTTP proxy URLs, for example http://123.4.5.6.3128
None
firewall_enabled Set this parameter to true to allow for installation of an IBM Cloud Private cluster in an environment that has firewall enabled. false
docker_api_timeout Sets the maximum wait time for docker-py to start a container. 100 s
wait_for_timeout This parameter specifies the default timeout value for operations. A setting of 3600 is ideal in most environments. None
disabled_management_services Use this parameter to disable the default catalog, monitoring, or metering service.

For example, to disable the catalog, metering, and monitoring services, set this parameter to ["service-catalog", "metering", "monitoring"].

The Vulnerability Advisor, which is available as a technology preview, is disabled by default. For more information about vulnerability advisor settings, see Table 13. Vulnerability advisor settings

["va"]

Kubernetes settings

Table 2. Kubernetes settings
Parameter Description Default value
kube_apiserver_extra_args Sets extra apiserver configurations for Kubernetes. Accepts a list of apiserver arguments that are provided in --key=value format. None
kube_apiserver_insecure_port Sets the Kubernetes apiserver insecure port. 8888
kube_apiserver_secure_port Sets the Kubernetes apiserver secure port. 8001
kube_controller_manager_extra_args Sets extra controller configurations for Kubernetes. Accepts a list of controller arguments that are provided in --key=value format. None
kube_proxy_extra_args Sets extra proxy configurations for Kubernetes. Accepts a list of proxy arguments that are provided in --key=value format. None
kube_scheduler_extra_args Sets extra scheduler configurations for Kubernetes. Accepts a list of scheduler arguments that are provided in --key=value format. None
kubelet_extra_args Sets extra configurations for kubelet. Accepts a list of kubelet arguments that are provided in --key=value format. For example, to set the maximum number of pods that can run on a kubelet, set the following configuration:kubelet_extra_args: ["--max-pods=110"] None
auditlog_enabled Enables the Kubernetes audit log, which records the chronological sequence of activities by individual users, administrators, or other components of the system that modified the system. Set this parameter to true to enable the audit log. false

Log settings

Table 3. Log settings
Parameter Description Values Default value
metrics_max_age Sets the maximum number of days to store system and application metrics. Metrics older than this specified number of days are removed. Metrics are removed at 23:59 on the specified day. Number of days 1
kibana_install When true, the installer deploys Kibana and integrates it with the IBM Cloud Private management console. true or false false

Network settings

Table 4. Network settings
Parameter Description Values Default value
calico_ipip_enabled Allows Calico to be run on IP over IP mode. This setting is needed when worker nodes are in different subnetwork and BGP is not enabled in routers between the worker nodes. This setting is also needed in some cloud environment such as OpenStack, where virtual machines are not allowed to work as routers. true or false true
calico_tunnel_mtu The IPIP for Calico has a default MTU of 1430. If the main interface of your host has an MTU that is less than 1450, Calico IPIP has poor performance. Set the MTU such that the MTU of the host main interface minus the default MTU of the Calico IPIP tunnel is greater than or equal to 20. Positive integers 1430
network_cidr The IPv4 network to use for the entire network. This value must be in CIDR format. When you create a network_cidr, ensure that you select an IP range that does not conflict with the existing host network or with the service_cluster_ip_range. In most environments, you can use the default value. IP address in CIDR format 10.1.0.0/16
cluster_lb_address In an environment that has multiple network interfaces (NICs), use cluster_lb_address to set a public or external IP address for the management services in your cluster. You can specify a fully-qualified domain name instead of the IP address.
This public address is assigned to the master node, used to access the console, and also used to configure kubectl.
In an HA environment, cluster_lb_address masks the cluster_vip as the leading master IP.
  • IP address
  • Fully-qualified domain name
  • OpenStack floating IP address
None
proxy_lb_address In an environment that has multiple network interfaces (NICs), use proxy_lb_address to set a public or external IP address that is to be used by the NodePort resource to allow external access to services. You can specify a fully-qualified domain name instead of the IP address.
  • IP address
  • Fully-qualified domain name
  • OpenStack floating IP address
None
calico_ip_autodetection_method You can configure the Calico node to auto-detect the IP address that is used to route between nodes. You can use one of the following methods:
  • calico_ip_autodetection_method: first-found: This method uses the first valid IP address on a valid interface that is found first.
  • calico_ip_autodetection_method: interface: This parameter accepts a comma-separated list of regular expression names as value. It uses the first IP address that is found on the specified interface.
Examples:
  • calico_ip_autodetection_method: interface=eth0
  • calico_ip_autodetection_method: interface=eth.*
  • calico_ip_autodetection_method: interface=eth.*,ens.*
  • calico_ip_autodetection_method: can-reach=<remote IP address or host name>: The can-reach method uses your local routing to determine the IP address that is used to reach the specified destination. This parameter accepts a remote IP address or domain name as value.
  • Note:
    • In an environment that has multiple network interfaces (NICs), use the can-reach method to specify the network to be used for your workloads. In IBM Cloud Private, you can set calico_ip_autodetection_method: can-reach=<Master node IP address>.
    • The network interface names cannot contain the following strings: "docker.*", "cbr.*", "dummy.*", "virbr.*", "lxcbr.*", "veth.*", "lo", "cali.*", "tunl.*", or "flannel.*".
    • Some IPs are not recognized by Calico. Ensure that your interfaces do not have IPs in the following ranges:
      • 10.0.2.15/24 - this IP range is the default vagrant/virtualbox NAT interface address range.
      • 92.168.122.* - this IP range is the default libvirt VM interface address range.
    • first-found

    • interface=INTERFACE-REGEX

    • can-reach=<remote IP address or domain name>

    can-reach={{ groups['master'][0] }}
    service_cluster_ip_range The Kubernetes service cluster IP range. This configuration allocates a block of IPs for services. These service IPs do not need to be routable, since kube-proxy converts service IPs to Pod IPs before traffic leaves the node. When you create a service_cluster_ip_range, ensure that you select an IP range that does not conflict with the existing host network or with the network_cidr. service_cluster_ip_range is a virtual network. In most environments, you can keep the default value. IP address in CIDR format 10.0.0.1/24

    Docker settings

    Note: These configurations can be set for the supplied IBM Cloud Private Docker packages only. See IBM Cloud Private Docker packages.

    Table 5. Docker settings
    Parameter Description Format Default Value
    docker_version Specify the version of Docker that you want to install.

    A package for the required version must be available in the /<installation_directory>/cluster/docker-engine directory.

    string 17.09
    install_docker Allows the installer to automatically install Docker on your cluster nodes. true or false true
    docker_env Sets the environment for Docker. For example configure, you might configure an https_proxy location if Docker runs behind a firewall. The environment location is stored in the /etc/systemd/system/docker.service.d/docker.conf file. ["HTTP_PROXY=http://httphost:port/", "NO_PROXY=localhost,127.0.0.1"] None
    docker_extra_args Sets Docker configuration parameters. For more information about Docker configuration parameters, see dockerd Opens in a new tab. ["--storage-driver=devicemapper"] None
    docker_log_max_size The maximum size of the log file before old entries are removed. Use this parameter only if you allow IBM Cloud Private to install Docker on your non-boot nodes. A positive integer and a modifier that represents the unit of measure for file size. You can use the following modifiers: "k" for kB, "m" for mB, or "g" for gB. 50m
    docker_log_max_file The maximum number of log files. Use this parameter only if you allow IBM Cloud Private to install Docker on your non-boot nodes. A positive integer. 10

    Proxy HA settings

    Table 6. Proxy HA settings
    Parameter Description Values Default
    vip_manager The service that manages the virtual IP (VIP) on the master, proxy, or master and proxy nodes. If you configure HA for master or proxy nodes, you can set this value. You set this value only once.
    Note. If your proxy node is a Linux® on Power® 64-bit LE system, you cannot set this parameter to etcd.
    • ucarp
    • etcd
    ucarp
    proxy_vip_iface Sets the virtual IP interface for a proxy node HA environment. eth0 n/a
    proxy_vip Sets the virtual IP address for a proxy node HA environment. 172.16.12.123
    Do not specify the subnet in the IP address.
    n/a

    Master HA settings

    Table 7. HA settings
    Parameter Description Values Default
    vip_manager The service that manages the virtual IP (VIP) on the master, proxy, or master and proxy nodes. If you configure HA for master or proxy nodes, you can set this value. You set this value only once.
    Note. If your master node is a Linux® on Power® 64-bit LE system, you cannot set this parameter to etcd.
    • ucarp
    • etcd
    ucarp
    cluster_vip Sets the virtual IP address for IBM Cloud Private HA environment. 172.16.12.123
    Do not specify the subnet in the IP address.
    n/a
    vip_iface Sets the virtual IP interface for IBM Cloud Private HA environment. eth0 n/a

    Federation settings

    Table 8. Federation settings
    Parameter Description Default value
    federation_enable Deploy a federation control plane in this cluster. This cluster becomes the host federation cluster. A single federation host cluster can manage multiple clusters.

    For more information about federation clusters, see https://v1-9.docs.kubernetes.io/docs/tasks/federation/set-up-cluster-federation-kubefed/ External link icon.

    To deploy a federation plan in your cluster, during installation, set set this parameter value to true.

    false
    federation_cluster Name for the federation cluster federation-cluster
    federation_domain DNS suffix for the federation. Federated service DNS names are published with this suffix. cluster.federation
    federation_external_policy_engine_enabled Enable or disable an external policy engine in federation. false
    federation_apiserver_extra_args Sets extra federation apiserver arguments.

    Accepts a list of arguments that are provided in –key=value format.

    For example [“--arg1-value1”,”--arg2=value2”]

    federation_controllermanager_extra_args Sets extra federation controller manager arguments.

    Accepts a list of arguments that are provided in –key=value format.

    For example [“--arg1-value1”,”--arg2=value2”]

    cluster_zone Zone where the cluster is located. For example, if the cluster is in US South, set the zone to south. myzone
    cluster_region Region where the cluster is located. Multiple zones come together to make a region. For example, if a cluster is in the US South, set the region to us. myregion

    User settings

    Table 9. User settings
    Parameter Description
    ansible_user
    ansible_ssh_pass
    ansible_become
    ansible_port
    ansible_become_pass
    {{site.keyword.data.icp_notm}} uses the ansible_user and ansible_ssh_pass parameter values to access your cluster nodes during installation.
    If you use a non-administrator account that has sudo privileges to connect to a master or worker node, set the ansible_user to the user name and ansible_become to true.
    If you run sudo with a password, you must set the ansible_become_pass parameter to the value of your non-root (sudo user) password. This variable is optional if you set NOPASSWD in the /etc/sudoers file.
    If you have a customized ssh port, you can also set the ansible_port parameter in the <installation_directory/cluster/hosts file.
    To configure these parameter values, see Configuring password authentication for cluster nodes.
    default_admin_user Sets a customized cluster administrator user name. If LDAP is enabled and you have an LDAP user with the user name admin, change the default_admin_user parameter value to something else to avoid a conflict of cluster administrator name and LDAP user name.
    default_admin_password Sets a customized cluster administrator password. If LDAP is enabled, this parameter sets the LDAP administrator password.

    GlusterFS settings

    Table 10. GlusterFS settings
    Parameter Description
    glusterfs Provision storage on worker nodes.

    To set up GlusterFS, you must set several parameters. See, Adding GlusterFS storage.

    vSphere Cloud Provider settings

    Table 11. vSphere Cloud Provider settings
    Parameter Description
    cloud_provider Sets up vSphere Cloud Provider.

    To set up vSphere Cloud Provider, you must set several parameters. See Configuring a vSphere Cloud Provider.

    AWS Cloud Provider settings

    Table 12. AWS Cloud Provider settings
    Parameter Description
    cloud_provider Sets up AWS Cloud Provider.

    To set up AWS Cloud Provider, you need to set two parameters: cloud_provider to aws and kubelet_nodename to nodename.

    For more details about setting up IBM Cloud Private-CE (Community Edition) on an Amazon Web Services (AWS) cloud platform, see Run IBM Cloud Private on Amazon Web Services (AWS) cloud platform External link icon.

    Encrypting cluster data network traffic with IPsec

    Table 13. Encrypting cluster data network traffic with IPsec
    Parameter Description
    ipsec_mesh
    enable: true Enables IPsec.

    To set up IPsec, you must set several parameters. See Encrypting cluster data network traffic with IPsec.

    Vulnerability Advisor settings

    Table 14. Vulnerability advisor settings
    Parameter Description Default value
    disabled_management_services By default, the vulnerability advisor is disabled.

    To enable the vulnerability advisor, set the parameter as follows: disabled_management_services: [""]

    ["va"]
    va_api_server_nodePort The vulnerability Advisor API service NodePort. Change the default value if it conflicts with an existing service. 30610
    va_crawler_enabled Specify whether to enable crawler for the vulnerability advisor in your cluster.

    If you do not want vulnerability reports, you can disable this parameter at runtime.

    True

    Integrating VMware NSX-T 2.0 with IBM Cloud Private

    Table 15. Integrating VMware NSX-T 2.0 with IBM Cloud Private
    Parameter Description
    network_type nsx-t
    ingress_controller_in_hostnetwork Sets the behavior of the default ingress controller. Values: true (default) or false. The default value of true sets the default ingress controller to use the node IP address. If you need the default ingress controller to use NAT (network address translation) pool, you must set this value to false.
    nsx_t You must set several parameters to integrate VMware NSX-T 2.0 with IBM Cloud Private. See Integrating VMware NSX-T 2.0 with IBM Cloud Private.