Contents


Mesos and Kubernetes on a hybrid (IBM Power and x86) architecture scenario

Comments

About Mesos and Kubernetes

Currently, there are several popular containers clustering system for Linux®, such as Kubernetes, Mesos and so on. Google Kubernetes and Apache Mesos are probably two commonly used tools to deploy containers inside a cluster. Both are created as helper tools that can be used to manage a cluster of containers. However, they differ greatly in their approaches.

Always, our customs have a hybrid architecture that includes IBM® Power® and x86. So, this article provides the reference solutions about how to apply Mesos and Kubernetes into Linux on a hybrid architecture (including Power and x86) environment.

See Figure 1 for the reference architecture.

Figure 1. The architecture diagram of Kubernetes and Mesos on the hybrid (Power and x86) architecture

Deployment

Now let's set up and deploy the Docker cluster of Mesos and Kubernetes with hybrid architecture. Here, we will use the Docker images for each module in the master node.

Prerequisites

  • Understanding of Mesos, Kubernetes, and Docker
  • A cluster of mixed architecture with:
    • Linux terminal, Docker, and git
    • At least four x86 nodes, and one IBM Power node
  • Disable firewall in each node

Environment

We need at least three nodes with CentOS 7.2 to install the ZooKeeper cluster, Mesos master cluster, etcd cluster, and Kubernetes-Mesos cluster.

  • 192.168.1.2 (x86)
  • 192.168.1.3 (x86)
  • 192.168.1.4 (x86)

We need at least two nodes with CentOS 7.2 to install the Mesos slave cluster.

  • 192.168.1.5 (x86)
  • 192.168.1.6 (IBM Power)

Deploy the ZooKeeper cluster

Perform the following steps with root privileges to deploy the ZooKeeper cluster:

In 192.168.1.2:

				docker run -d \ 
				-e MYID=1 \ 
				-e SERVERS=192.168.1.2,192.168.1.3,192.168.1.4 \ 
				--name=zookeeper --net=host --restart always powerkvm/zookeeper-x86

In 192.168.1.3:

					docker run -d \
					-e MYID=2 \
					-e SERVERS=192.168.1.2,192.168.1.3,192.168.1.4 \ 
					--name=zookeeper --net=host --restart always powerkvm/zookeeper-x86

In 192.168.1.4:

					docker run -d \
					-e MYID=3 \ 
					-e SERVERS=192.168.1.2,192.168.1.3,192.168.1.4 \
					--name=zookeeper --net=host --restart always powerkvm/zookeeper-x86

Deploy the Mesos master cluster

Perform the following steps with root privileges to deploy the Mesos master cluster.

In 192.168.1.2:

					docker run -d -e MESOS_HOSTNAME=192.168.1.2 -e MESOS_IP=192.168.1.2 -e MESOS_QUORUM=2 \ 
					-e MESOS_ZK=zk://192.168.1.2:2181,192.168.1.3:2181,192.168.1.4:2181/mesos \ 
					--name mesos-master --net host --restart always powerkvm/mesos-master-x86

In 192.168.1.3:

					docker run -d -e MESOS_HOSTNAME=192.168.1.3 -e MESOS_IP=192.168.1.3 -e MESOS_QUORUM=2 \
					-e MESOS_ZK=zk://192.168.1.2:2181,192.168.1.3:2181,192.168.1.4:2181/mesos \ 
					--name mesos-master --net host --restart always powerkvm/mesos-master-x86

In 192.168.1.4:

					docker run -d -e MESOS_HOSTNAME=192.168.1.4 -e MESOS_IP=192.168.1.4 -e MESOS_QUORUM=2 \
					-e MESOS_ZK=zk://192.168.1.2:2181,192.168.1.3:2181,192.168.1.4:2181/mesos \
					--name mesos-master --net host --restart always powerkvm/mesos-master-x86

Deploy the etcd cluster

Perform the following steps with root privileges to deploy the etcd cluster:

In 192.168.1.2:

					docker run -d -v /usr/share/ca-certificates/:/etc/ssl/certs -p 4001:4001 -p 2380:2380 -p 2379:2379 \ 
				    --name etcd --restart always quay.io/coreos/etcd:v2.2.1 \ 
				    -name etcd0 \ 
				    -advertise-client-urls http://192.168.1.2:2379,http://192.168.1.2:4001 \ 
				    -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \ 
				    -initial-advertise-peer-urls http://192.168.1.2:2380 \ 
				    -listen-peer-urls http://0.0.0.0:2380 \
				    -initial-cluster-token etcd-cluster-1 \ 
				    -initial-cluster etcd0=http://192.168.1.2:2380,etcd1=http://192.168.1.3:2380,etcd2=http://192.168.1.4:2380\
				    -initial-cluster-state new

In 192.168.1.3:

					docker run -d -v /usr/share/ca-certificates/:/etc/ssl/certs -p 4001:4001 -p 2380:2380 -p 2379:2379 \ 
					--name etcd --restart always quay.io/coreos/etcd:v2.2.1 \ 
					-name etcd0 \
					-advertise-client-urls http://192.168.1.3:2379,http://192.168.1.3:4001 \ 
					-listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \ 
					-initial-advertise-peer-urls http://192.168.1.3:2380 \ 
					-listen-peer-urls http://0.0.0.0:2380 \
					-initial-cluster-token etcd-cluster-1 \ 
					-initial-cluster etcd0=http://192.168.1.2:2380,etcd1=http://192.168.1.3:2380,etcd2=http://192.168.1.4:2380\
					-initial-cluster-state new

In 192.168.1.4:

					docker run -d -v /usr/share/ca-certificates/:/etc/ssl/certs -p 4001:4001 -p 2380:2380 -p 2379:2379 \
					--name etcd --restart always quay.io/coreos/etcd:v2.2.1 \
					-name etcd0 \ 
					-advertise-client-urls http://192.168.1.4:2379,http://192.168.1.4:4001 \ 
					-listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \ 
					-initial-advertise-peer-urls http://192.168.1.4:2380 \ 
					-listen-peer-urls http://0.0.0.0:2380 \
					-initial-cluster-token etcd-cluster-1 \ 
					-initial-cluster etcd0=http://192.168.1.2:2380,etcd1=http://192.168.1.3:2380,etcd2=http://192.168.1.4:2380\
					-initial-cluster-state new

Deploy the Kubernetes-Mesos cluster

Perform the following steps with root privileges to deploy the Kubernetes-Mesos cluster:

Deploy apiserver, controller manager,and scheduler in 192.168.1.2:

  1. Add mesos-cloud.conf in /mnt/k8s:
    					cat /mnt/k8s/mesos-cloud.conf
    					[mesos-cloud]
    					mesos-master = zk://192.168.1.2:2181,192.168.1.3:2181,192.168.1.4:2181/mesos
  2. Run each Docker image.
    					docker run -d --privileged -v /mnt/k8s:/mnt/k8s --name apiserver --net=host --restart always powerkvm/k8s-mesos-x86:v1.4.0-alpha km apiserver \
    					--address=192.168.1.2 \
    					--etcd-servers=http://192.168.1.2:4001,http://192.168.1.3:4001,http://192.168.1.4:4001\
    					--service-cluster-ip-range=10.10.10.0/24 \ 
    					--port=8888 \ 
    					--cloud-provider=mesos \
    					--cloud-config=/mnt/k8s/mesos-cloud.conf \ 
    					--secure-port=0 \ 
    					--v=1 
    					
    					docker run -d --privileged -v /mnt/k8s:/mnt/k8s --name controller-manager --net=host --restart always powerkvm/k8s-mesos-x86:v1.4.0-alpha km controller-manager \
    					--master=192.168.1.2:8888 \ 
    					--cloud-provider=mesos \
    					--cloud-config=/mnt/k8s/mesos-cloud.conf \
    					--v=1
    					
    					docker run -d --privileged --name scheduler --net=host --restart always powerkvm/k8s-mesos-x86:v1.4.0-alpha kmscheduler \
    					--address=192.168.1.2 \
    					--mesos-master=zk://192.168.1.2:2181,192.168.1.3:2181,192.168.1.4:2181/mesos \
    					--etcd-servers=http://192.168.1.2:4001,http://192.168.1.3:4001,http://192.168.1.4:4001\
    					--mesos-user=root \
    					--api-servers=192.168.1.2:8888 \
    					--cluster-dns=10.10.10.10 \
    					--cluster-domain=cluster.local \
    					--v=2

Deploy apiserver, controller manager,and scheduler in 192.168.1.3:

  1. Add mesos-cloud.conf in /mnt/k8s:
    					cat /mnt/k8s/mesos-cloud.conf
    					[mesos-cloud]
    					mesos-master = zk://192.168.1.2:2181,192.168.1.3:2181,192.168.1.4:2181/mesos
  2. Run each Docker image:
    					docker run -d --privileged -v /mnt/k8s:/mnt/k8s --name apiserver --net=host --restart always powerkvm/k8s-mesos-x86:v1.4.0-alpha km apiserver \
    					--address=192.168.1.3 \
    					--etcd-servers=http://192.168.1.2:4001,http://192.168.1.3:4001,http://192.168.1.4:4001\
    					--service-cluster-ip-range=10.10.10.0/24 \
    					--port=8888 \
    					--cloud-provider=mesos \
    					--cloud-config=/mnt/k8s/mesos-cloud.conf \
    					--secure-port=0 \
    					--v=1
    					
    					docker run -d --privileged -v /mnt/k8s:/mnt/k8s --name controller-manager --net=host --restart always powerkvm/k8s-mesos-x86:v1.4.0-alpha km controller-manager \
    					--master=192.168.1.3:8888 \
    					--cloud-provider=mesos \
    					--cloud-config=/mnt/k8s/mesos-cloud.conf \
    					--v=1
    					
    					docker run -d --privileged --name scheduler --net=host --restart always powerkvm/k8s-mesos-x86:v1.4.0-alpha km scheduler \
    					--address=192.168.1.3 \
    					--mesos-master=zk://192.168.1.2:2181,192.168.1.3:2181,192.168.1.4:2181/mesos \
    					--etcd-servers=http://192.168.1.2:4001,http://192.168.1.3:4001,http://192.168.1.4:4001\
    					--mesos-user=root \
    					--api-servers=192.168.1.3:8888 \
    					--cluster-dns=10.10.10.10 \
    					--cluster-domain=cluster.local \ 
    					--v=2

Deploy apiserver, controller manager,and scheduler in 192.168.1.4:

  1. Add mesos-cloud.conf in /mnt/k8s:
    					cat /mnt/k8s/mesos-cloud.conf
    					[mesos-cloud]
    					mesos-master = zk://192.168.1.2:2181,192.168.1.3:2181,192.168.1.4:2181/mesos
  2. Run each Docker image:
    					docker run -d --privileged -v /mnt/k8s:/mnt/k8s --name apiserver --net=host --restart always powerkvm/k8s-mesos-x86:v1.4.0-alpha km apiserver \
    					--address=192.168.1.4 \
    					--etcd-servers=http://192.168.1.2:4001,http://192.168.1.3:4001,http://192.168.1.4:4001\
    					--service-cluster-ip-range=10.10.10.0/24 \
    					--port=8888 \
    					--cloud-provider=mesos \
    					--cloud-config=/mnt/k8s/mesos-cloud.conf \
    					--secure-port=0 \ 
    					--v=1
    					
    					docker run -d --privileged -v /mnt/k8s:/mnt/k8s --name controller-manager --net=host --restart always powerkvm/k8s-mesos-x86:v1.4.0-alpha km controller-manager \
    					--master=192.168.1.4:8888 \
    					--cloud-provider=mesos \
    					--cloud-config=/mnt/k8s/mesos-cloud.conf \
    					--v=1 
    					
    					docker run -d --privileged --name scheduler --net=host --restart always powerkvm/k8s-mesos-x86:v1.4.0-alpha km scheduler \
    					--address=192.168.1.4 \
    					--mesos-master=zk://192.168.1.2:2181,192.168.1.3:2181,192.168.1.4:2181/mesos \
    					--etcd-servers=http://192.168.1.2:4001,http://192.168.1.3:4001,http://192.168.1.4:4001\
    					--mesos-user=root \
    					--api-servers=192.168.1.4:8888 \
    					--cluster-dns=10.10.10.10 \
    					--cluster-domain=cluster.local \
    					--v=2

Deploy the Mesos slave cluster

The Docker image for Mesos slave is still under testing. So, we can use the packages to install it.

Deploy the Mesos slave in 192.168.1.5 (x86):

1. Install the Mesos slave.

					rpm -Uvh http://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-2.noarch.rpm && \
					yum -y install mesos

2. Configure the Mesos slave.

					#cat > /etc/mesos/zk <<EOF
					     zk://192.168.1.2:2181,192.168.1.3:2181,192.168.1.4:2181/mesos 
					EOF 
					
					echo 192.168.1.5 | sudo tee /etc/mesos-slave/ip
					cp /etc/mesos-slave/ip /etc/mesos-slave/hostname

3. Start the mesos-slave service.

 systemctl restart mesos-slave

Depoloy the Mesos slave in 192.168.1.6(Power):

  1. Add the repository.
    					#cat > /etc/yum.repos.d/cluster.repo <<EOF 
    					[cluster]
    					name=cluster baseurl=http://ftp.unicamp.br/pub/ppc64el/rhel/7_1/misc_ppc64el/
    					enabled=1
    					gpgcheck=0
    					priority=10
    					sslverify=1
    					EOF
  2. Install the Mesos slave.
     yum -y install mesos-0.22.1 python-mesos
  3. Configure the Mesos slave.

    Edit the variables (highlighted below) in /etc/sysconfig/mesos-slave:

    					# This file contains environment variables that are passed to mesos-slave. 
    					# To get a description of all options run mesos-slave --help; any option 
    					# supported as a command-line option is also supported as an environment 
    					    # variable.
    					    
    					# The mesos master URL to contact. Should be host:port for
    					# non-ZooKeeper based masters, otherwise a zk:// or file:// URL.
    					MESOS_master=zk://192.168.1.2:2181,192.168.1.3:2181,192.168.1.4:2181/mesos
    					MESOS_EXECUTOR_REGISTRATION_TIMEOUT=10mins 
    					MESOS_IP=192.168.1.6 
    					
    					# For isolated sandbox testing 
    					# MESOS_master=127.0.0.1:5050
    					
    					# For a complete listing of options execute 'mesos-slave --help' 
    					MESOS_log_dir=/var/log/mesos
    					MESOS_work_dir=/var/run/mesos
    					MESOS_containerizers=docker,mesos 
    					
    					# systemd cgroup integration 
    					MESOS_isolation='cgroups/cpu,cgroups/mem'
    					MESOS_cgroups_root='system.slice/mesos-slave.service'
    					MESOS_cgroups_hierarchy=/sys/fs/cgroup
  4. Start the mesos-slave service.
     service emsos-slave start

Deploy the Kubernetes dashboard

The framework of Kubernetes and Mesos for the hybrid architecture is complete. Now, we should deploy the dashboard for it. For this case, run the dashboard in the slaves of x86 architecture.

First, we should add the labels for each slave.

  1. Add the label arch=x86 for each x86 architecture-based slave.
    					docker run -e KUBERNETES_MASTER_IP=192.168.1.2 -e KUBERNETES_MASTER=192.168.1.2:8888 powerkvm/k8s-mesos-x86:v1.4.0-alpha kubectl label nodes 192.168.1.5 arch=x86
  2. Add the label arch=ppc64le for each Power architecture-based slaves.
    					docker run -e KUBERNETES_MASTER_IP=192.168.1.2 -e KUBERNETES_MASTER=192.168.1.2:8888 powerkvm/k8s-mesos-x86:v1.4.0-alpha kubectl label nodes 192.168.1.6 arch=ppc64le
  3. Check the result.
    					docker run -e KUBERNETES_MASTER_IP=192.168.1.2 -e KUBERNETES_MASTER=192.168.1.2:8888 powerkvm/k8s-mesos-x86:v1.4.0-alpha kubectl get nodes --show-labels
    					NAME                       STATUS         AGE           LABELS 
    					192.168.1.5                Ready          2d            arch=x86,kubernetes.io/hostname=192.168.1.5 
    					192.168.1.6                Ready          2d            arch=ppc64le,kubernetes.io/hostname=192.168.1.6
  4. Download the Kubernetes dashboard.
    					git clone https://github.com/kubernetes/dashboard.git /root/dashboard
  5. Add nodeSelector and apiserver in kubernetes-dashboard-canary.yaml
    					cat /root/dashboard/src/deploy/kubernetes-dashboard-canary.yaml 
    					…
    					  spec: 
    					     containers: 
    					     - name: kubernetes-dashboard-canary 
    					       image:gcr.io/google_containers/kubernetes-dashboard-amd64:canary 
    					     … 
    					       args: 
    					       - --apiserver-host=http://192.168.1.2:8888 
    					     … 
    					     nodeSelector: 
    					       arch: x86 …
  6. Run the Kubernetes service for the dashboard.
    					docker run --privileged -v /root/dashboard:/root/dashboard -e KUBERNETES_MASTER_IP=192.168.1.2 -e KUBERNETES_MASTER=192.168.1.2:8888 powerkvm/k8s-mesos-x86:v1.4.0-alpha kubectl create -f /root/dashboard/src/deploy/kubernetes-
    					dashboard-canary.yaml --namespace="kube-system"
  7. Check the service and pods.

    Check service:

    					docker run -e KUBERNETES_MASTER_IP=192.168.1.2 -e KUBERNETES_MASTER=192.168.1.2:8888 powerkvm/k8s-mesos-x86:v1.4.0-alpha kubectl get services --namespace="kube-system" 
    				    NAME                  CLUSTER-IP       EXTERNAL-IP     PORT(S)     AGE
    				    dashboard-canary      10.10.10.14      <nodes>         80/TCP      21m

    Get nodePort for dashboard service:

    					docker run -e KUBERNETES_MASTER_IP=192.168.1.2 -e KUBERNETES_MASTER=192.168.1.2:8888 powerkvm/k8s-mesos-x86:v1.4.0-alpha kubectl describe services dashboard-canary --namespace="kube-system"
    					Name: dashboard-canary 
    					Namespace: kube-system 
    					Labels: app=kubernetes-dashboard-canary
    					Selector: app=kubernetes-dashboard-canary 
    					Type: NodePort 
    					IP: 10.10.10.14 
    					Port: <unset> 80/TCP 
    					NodePort: <unset> 31810/TCP 
    					Endpoints: <none>
    					Session Affinity: None

    Check pods:

    					docker run --privileged -v /root/dashboard:/root/dashboard -e KUBERNETES_MASTER_IP=192.168.1.2 -e KUBERNETES_MASTER=192.168.1.2:8888 powerkvm/k8s-mesos-x86:v1.4.0-alpha kubectl get pods --namespace="kube-system" 
    					NAME                                 READY    STATUS    RESTARTS    AGE
    				    kubernetes-dashboard-canary-x5wqg    1/1      Running   0           1h

Web UI

Now, we have deployed Kubernetes and Mesos successfully. Let's check the webpage for Mesos and the dashboard.

For Mesos UI, we can get the home page by accessing port 5050 with the IP of any Mesos master.

For example, [any Mesos master's IP] : 5050

Figure 2. Mesos home page

For dashboard, we can get the home page by accessing the nodePort with the IP of any Mesos slave.

For example: [any Mesos slave's IP] : nodePort.
You can get the nodePort from the description of the dashboard service.

You can get the nodePort by using the following command:

					docker run -e KUBERNETES_MASTER_IP=192.168.1.2 -e KUBERNETES_MASTER=192.168.1.2:8888 powerkvm/k8s-mesos-x86:v1.4.0-alpha kubectl describe services dashboard-canary --namespace="kube-system"
Figure 3. Kubernetes dashboard

Conclusion

This article explained how to deploy kubernetes and Mesos on a hybrid (Power and x86) architecture scenario. It also described how to deploy the service on x86 by using label and nodeSelector.

Resource

Refer to the help guide for Kubernete on Mesos for whole x86 architecture.


Downloadable resources


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Linux
ArticleID=1037868
ArticleTitle=Mesos and Kubernetes on a hybrid (IBM Power and x86) architecture scenario
publish-date=10032016