Before you can get started, make sure you have the latest CLI’s for Code Engine, Kubernetes and Ray installed.
Install the pre-requisites for Code Engine and Ray
- Get ready with IBM Code Engine
- Set up the Kubernetes CLI
- Install Python3 libraries for Ray and Kubernetes:
Step 1: Define your Ray cluster
In order to start a Ray cluster, you need to define the cluster specification and where to run the Ray cluster. Since Code Engine exposes the Kubernetes API, we can use the Kubernetes provider from Ray.
To tell Ray which Kubernetes environment to use, you would need to run the following commands:
- If not already done, create a project in Code Engine:
ibmcloud ce project create -n <your project name>
- Select the Code Engine project and switch the `kubectl` context to the project:
ibmcloud ce project select -n <your project name> -k
- Extract the Kubernetes namespace assigned to your project. The namespace can be found in the NAME column in the output of the command:
- Export the namespace:
export NAMESPACE=<namespace from above (3.)>
With the following command you can download a basic Ray cluster definition and customize it for your namespace:
$ curl -L https://gist.githubusercontent.com/mhenke1/5674e3caefbd19b94274765d400f1462/raw/6d00b3abd35cdf50a87ada8a9a52428c3a54baa2/example-cluster.yaml | sed "s/NAMESPACE/$NAMESPACE/" > example-cluster.yaml
The downloaded example describes a Ray cluster with the following characteristics:
- A cluster named
example-cluster with up to 10 workers using the Kubernetes provider pointing to your Code Engine project - A head node type with 1 vCPU and 2 GB memory using the ray image rayproject/ray:1.5.1
- A worker node type with 0.5 vCPU and 1 GB memory using the ray image rayproject/ray:1.5.1
- The default startup commands to start Ray within the container and listen to the proper ports
- The autoscaler upscale speed is set to 10 for quick upscaling in this short and simple demo
Please see the section “One final note for advanced Kubernetes users” below for more information.
Step 2: “Ray up” your cluster
Now you can start the Ray cluster by running ray up example-cluster.yaml . This command will create the Ray head node as Kubernetes Pod in your Code Engine project. When you create the Ray cluster for the first time, it can take up to three minutes until the Ray image is downloaded from the Ray repository. Ray will emit “SSH still not available” error messages until the image has been downloaded and the pod has started.
Step 3: Submit a very simple but distributed Ray program
The following very simple Ray program example illustrates how tasks can be executed in the remote Ray cluster. Function f simulates a workload that is called for 200 times. The 200 workloads will be distributed among the Ray cluster nodes:
# import all helper libraries
from collections import Counter
import socket
import time
import ray
import math
# connect to ray
ray.init(address='auto', _redis_password='5241590000000000')
# f: returns the hostname/ip of the current container executing the function. f is annotated
# with @ray.remote to indicate that it should be distributed among the nodes in the Ray cluster.
@ray.remote(num_cpus=1)
def f(counter, total):
host = socket.gethostbyname(socket.gethostname())
print("inside f({}/{}), host={}".format(counter + 1, total, host))
# simulate work for 5 seconds
time.sleep(5)
# return IP address
return host
# execute function f for 200 times and fetch the results
n = 200
object_ids = [f.remote(i, n) for i in range(n)]
ip_addresses = ray.get(object_ids)
# print a histogram of the number of tasks executed per hostname/ip
print('Tasks executed')
for ip_address, num_tasks in Counter(ip_addresses).items():
print(' {} tasks on {}'.format(num_tasks, ip_address))
After having copied the above program to a file example.py, you can run the program by submitting it to the Ray cluster:
# Check that the head node pod status is Running.
# If the status is ContainerCreating, the Ray image download is still running.
kubectl get pods
# Submit the program to the Ray cluster
ray submit example-cluster.yaml example.py
This will copy the Python program to the Ray head node and start it there. In parallel, the autoscaler will begin to scale up the Ray cluster with additional worker nodes. The Python program will be copied to the newly created worker nodes and run there in addition to the head node. The autoscaler scales up the Ray cluster in chunks until the maximum number of workers (10) is reached as defined in example-cluster.yaml with available_node_types.worker_node.max_workers. Finally, one minute (see idle_timeout_minutes in example-cluster.yaml) after the program has terminated, the autoscaler starts to downscale the Ray cluster until all worker nodes are deleted and only the head node is left.
Example output: Program output and autoscaler event messages
- 172.30.192.43 is the head node
inside f(<index>/200), host=<ip> reflects the n-th invocation of function f, index=1,2,…200- The output ends with a summary that shows the distribution of the 200 function calls among the 11 nodes of the Ray cluster (1 head node and 10 worker nodes). Since we have started in a Ray cluster with no initial worker nodes, the distribution is not equal among the nodes. It took some time until all workers were created by the autoscaler. When you re-submit the example within less than a minute where all worker nodes still run, the distribution would be more like an ideal, equal distribution, until all 200 tasks are terminated:
