Deploying an IBM Spectrum Symphony cluster with Kubernetes and Docker containers

Kubernetes is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications. Kubernetes is a good platform to deploy primary-secondary mode applications. You can deploy IBM® Spectrum Symphony with Kubernetes and run its services in Docker containers. This configuration is supported on Red Hat Enterprise Linux (RHEL) 64-bit primary and compute hosts.

1. Install Kubernetes, Docker, and etcd on all hosts:

   yum install -y kubernetes etcd

   Note: Docker is installed automatically after Kubernetes is installed.
2. Build an IBM Spectrum Symphony Docker image.
   1. Start the Docker daemon:

      systemctl start docker

   2. Use the docker build command to create an IBM Spectrum Symphony Docker image from the definition contained in the Dockerfile. For example, run:

      docker build -t sym720:V1 /root/73.1/

      where 73.1 is the current working directory that contains the Dockerfile.

      Note: The sample Dockerfile uses a Red Hat 7 base Docker image. If you want to use another base Docker image, ensure that it can be installed and run the IBM Spectrum Symphony package successfully.

      The following is an example of the Dockerfile:

      FROM rhel
      
      # Add user 'egoadmin' & install ssh-server and other basic tools
      RUN useradd -m egoadmin \
         && echo "egoadmin:egoadmin" | chpasswd \
         && echo "egoadmin   ALL=(ALL)       NOPASSWD: ALL" >> /etc/sudoers \
         && useradd -m hadoop \
         && echo "hadoop:hadoop" | chpasswd \
         && echo "hadoop   ALL=(ALL)       NOPASSWD: ALL" >> /etc/sudoers \
         && echo -e "[base] \nname=CentOS-7 - Base - centos.com\nbaseurl=http://mirror.centos.org/centos/7/os/$basearch/\ngpgcheck=1\ngpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-7">/etc/yum.repos.d/CentOS7-Base.repo \
          && yum clean all \
         && yum makecache fast \
         && yum install -y openssh-server which net-tools sudo wget hostname tar openssh-clients \
         && yum install -y gettext rsh iputils vim telnet gdb \
         && sed -i 's/UsePAM yes/UsePAM no/g' /etc/ssh/sshd_config \
         && ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key \
         && ssh-keygen -t dsa -f /etc/ssh/ssh_host_ed25519_key \
         && ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key \
         && mkdir /var/run/sshd
      
      #Copy entitlement file to container
      ADD sym_adv_entitlement.dat /opt/sym_adv_entitlement.dat
      
      # Download & Install Symphony package
      RUN wget --no-check-certificate -O /opt/sym-7.2.0.0_x86_64.bin https://example.ibm.com/engr/pcc/release_eng/work/sym/sym_mainline/last/sym-7.2.0.0_x86_64.bin \
        && export CLUSTERADMIN=root \
        && export CLUSTERNAME=mycluster \
        && export SIMPLIFIEDWEM=N \
        && export BASEPORT=10086 \
        && chmod +x /opt/* \
        && /opt/sym-7.2.0.0_x86_64.bin --prefix /opt/ibm/spectrumcomputing --dbpath /opt/ibm/spectrumcomputing/db --quiet \
        && rm -rf /opt/*.bin
      
      USER root
      CMD ["/usr/sbin/sshd", "-D"]
      

      Note: The IBM Spectrum Symphony entitlement file will be called while building the Docker image. This means the entitlement file must be in the same directory as the Docker image. The entitlement file from the local host will be added into the Docker image under the /opt/ directory (the operating system directory in the Docker container).
   3. Optional: Push the Docker image to a private Docker registry so that all Kubernetes minion nodes can pull the Docker image from the Docker registry automatically:

      docker tag sym712:v1 192.168.1.100:5000/sym73.1:v1
      docker push 192.168.1.100:5000/sym73.1:v1
      

      where 192.168.1.100:5000/sym73.1:v1 is the image in the Kubernetes YAML file.

      Note: 192.168.1.100:5000 is the private Docker registry.
3. Install the virtual network Weave on all hosts:

   sudo curl -L git.io/weave -o /usr/local/bin/weave
   
   sudo chmod +x /usr/local/bin/weave
   
   systemctl start docker
   
   weave setup

   Note: Weave will include the Docker image from the public Docker hub, so ensure all Docker hosts have Internet access. Weave will start the service in the Docker container, so do not try to stop those containers already started by Weave.
4. Start Weave on all hosts. Choose one host as the Weave primary node and the others as the Weave secondary nodes to join those hosts to one network.

   Primary node

   Run:

   weave launch-proxy --rewrite-inspect --without-dns
   
   weave launch-router
   
   weave expose

   Secondary node

   Run:

   weave launch-proxy --rewrite-inspect --without-dns
   
   weave launch-router weave_primary_host_name
   
   weave expose

5. Configure and start the Kubernetes primary server:
   1. Configure the etcd server. Edit the /etc/etcd/etcd.conf file, as follows:

      ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS="http://kube-primary:2379"

   2. Configure the Kubernetes server. Edit the /etc/kubernetes/config file, as follows:

      KUBE_MASTER="--master=http://kube-primary:8080"

   3. Configure the Kubernetes API server. Edit the /etc/kubernetes/apiserver file, as follows:

      KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
      KUBE_ETCD_SERVERS="--etcd-servers=http://kube-primary:2379"
      KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
      KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"

   4. Start the Kubernetes server:

      systemctl restart etcd kube-apiserver kube-controller-manager kube-scheduler

6. Configure and start the Kubernetes minion server:
   1. Edit the /etc/kubernetes/config file, as follows:

      KUBE_MASTER="--master=http://kube-primary:8080"

   2. Edit the /etc/kubernetes/kubelet file:

      KUBELET_ADDRESS="--address=0.0.0.0"
      KUBELET_HOSTNAME="--hostname-override=127.0.0.1"
      KUBELET_API_SERVER="--api-servers=http://kube-primary:8080"
      KUBELET_ARGS="--docker-endpoint=unix:/var/run/weave/weave.sock"

   3. Start the Kubernetes minion:

      systemctl restart kube-proxy kubelet

7. Create the IBM Spectrum Symphony primary host with the Kubernetes YAML file. Kubernetes will create a Docker container from the YAML file and run the container start command defined within the YAML file:

   kubectl create -f sym-primary-rc.yaml

   The following is an example of the sym-primary-rc.yaml file:

   kind: ReplicationController
   apiVersion: v1
   metadata:
    name: sym-primary-rc
   spec:
    replicas: 1
    selector:
      component: sym-primary
    template:
      metadata:
        labels:
          component: sym-primary
      spec:
        containers:
          - name: sym-primary
            image: sym720:V1
            command: ["/bin/sh", "-c", "source /opt/ibm/spectrumcomputing/profile.platform; egoconfig join 'hostname' -f; egoconfig setentitlement /opt/sym_adv_entitlement.dat; egosh ego start; /usr/sbin/sshd -D"]
            resources:
              requests:
                memory: 4096Mi
                cpu: 4.0
              limits:
                memory: 8192Mi
                cpu: 8.0
            securityContext:
                privileged: true

   1. Check all the minion nodes are started on the primary host.

      NAME                             LABELS                                                 STATUS     AGE
      wavm1.example.ibm.com    kubernetes.io/hostname=wavm1.example.ibm.com   Ready      2m
      wavm2.example.ibm.com    kubernetes.io/hostname=wavm2.example.ibm.com   Ready      4s

8. Create the IBM Spectrum Symphony compute nodes with the YAML file:
   1. After the primary host is created and running, obtain the primary host name and IP address.

      Here are examples, with output:

      kubectl get pods
      
      NAME                   READY     STATUS   RESTARTS    AGE
      sym-primary-rc-04yws    1/1       Running  0           46m

      kubectl describe pod sym-primary-rc-04yws|grep IP
      
      IP:                10.32.0.1

      The following is an example of the sym-compute-rc.yaml file:

      kind: ReplicationController
      apiVersion: v1
      metadata:  
        name: sym-compute-rc
      spec:  
        replicas: 3
        selector:
          component: sym-compute
        template:
          metadata:
            labels:
              component: sym-compute
          spec:
            containers:
              - name: sym-compute
              image: sym712:v1
              commmand: ["/bin/sh", "-c", "source /opt/ibm/spectrumcomputing/profile.platform; sudo chmod 777 /etc/hosts; 
                        echo '10.32.0.1 sym-primary-04yws' >> /etc/hosts; egoconfig join sym-primary-04yws -f; 
                        egosh ego start; sudo /usr/sbin/sshd -D"]
              resources:           
                requests:
                  memory: 2048M

   2. Manually add the IBM Spectrum Symphony primary host name and IP address to the command in the compute node YAML file. For example:

      "command: ["/bin/sh", "-c", "source /opt/ibm/spectrumcomputing/profile.platform; 
                sudo chmod 777 /etc/hosts; echo '10.32.0.1 sym-primary-rc-04yws' >> 
                /etc/hosts; egoconfig join sym-primary-rc-04yws -f; 
                egosh ego start; sudo /usr/sbin/sshd -D"]"

   3. Create the IBM Spectrum Symphony compute nodes with the YAML file:
      Here is an example, with output:

      kubectl create -f sym-compute-rc.yaml
      
      replicationcontroller "sym-compute-rc" created

9. Scale the compute node size after you have created the compute replication controller if necessary:
   Here is an example, with output:

   kubectl scale --replicas=10 replicationcontrollers sym-compute-rc
   
   replicationcontroller "sym-compute-rc" scaled

10. Create the Kubernetes service to expose the IBM Spectrum Symphonycluster management console:
    The following is an example of the Kubernetes service definition file:

    kind: Service
    apiVersion: v1
    metadata:
      name: sym-webgui
    spec:
      ports:
        - port: 8443
          targetPort: 8443
          protocol: TCP
          name: https
      selector:
        component: sym-primary
      type: NodePort

    To create this file, run the kubectl create command.

    Here is an example, with output:

    kubectl create -f sym-webgui-svc.yaml
    
    service "sym-webgui" created