Structuring your deployment

This article provides guidance and best practices for planning your installation and usage of IBM Cloud Pak for Integration.

Some questions you may need to answer are:

  • How many IBM Red Hat OpenShift clusters should I deploy?
  • How should I deploy the IBM Cloud Pak® for Integration operators and operands within those clusters?
  • Can I share clusters between different user namespaces, environments, or lifecycle phases?
  • What are the implications of sharing a cluster between two or more logically separate activities?
  • How can I best exploit the functionality of namespaces (OpenShift projects)?

There is no “one size fits all” approach to deployment. The requirements for different scenarios may vary significantly in terms of size, scope, and environment. Deployment approaches may also vary depending on your specific usage.

For information on installing Red

Note: Throughout this article we use namespace (a Kubernetes term) to refer to the concept of project that is used in OpenShift. We also use "in a namespace" and "in all namespaces" to refer to A specific namespace on the cluster and All namespaces on the cluster operator installation modes.

Contents:

Terminology

namespace
A Kubernetes object which, together with other namespaces, divides a single Kubernetes cluster into multiple virtual clusters. For additional detail see namespaces in the Kubernetes documentation.
operator
An operator provides custom resource types in a Kubernetes cluster. Operators react to custom resources (CRs) to create managed resources. See Operators on Red Hat OpenShift for more information.
pod

Pods are the smallest deployable units of computing that you can create and manage in Kubernetes. See Pods in the Kubernetes documentation for more information.

Sharing infrastructure across workloads in Red Hat OpenShift

While many organizations successfully use namespaces to provide a degree of separation between teams, workloads, and namespaces, OpenShift itself does not natively support strong isolation of different workloads running in a single cluster. This challenge applies equally for workloads with any of the following:

  • Different environments in the development lifecycle for an application (such as development, test, and production environments for the same namespace).
  • Different user applications at the same environment (for example, development for Application A vs. Application B vs. Application C).
  • Workloads for different customer organizations in the traditional sense of SaaS or hosted service multi-tenancy.

Comparing OpenShift with a traditional deployment (on virtual machines) helps clarify the challenge of providing workload isolation:

  • An OpenShift cluster does not offer the same isolation as a configuration where different VMs are deployed for each workload, even if those VMs are running on the same hypervisor.
  • Because Kubernetes worker nodes are often VMs themselves, the equivalent scenario is running multiple workloads inside the same VM.

Considerations before deployment

The techniques within OpenShift for sharing a cluster across workloads have a range of benefits and drawbacks that you should take into account when planning a deployment architecture. This sections describes some potential benefits and trade-offs for the various layers of a deployment.

User roles and organizational structure

To ensure your deployment approach addresses the needs of various user roles, you might consider the organizational structure of your company:

  • Does a single administration team manage both the OpenShift cluster and the product-level administration of capabilities and runtimes (such as App Connect or MQ), or are these responsibilities handled by different teams?

    The equivalent scenario in a virtual-machine deployment would be if there is a different team that manages the product deployment (MQ, ACE) compared to the virtual machine—or possibly hypervisor—in which the product runs.

  • Is there a single team for all capability and runtime components, or does each type have its own specialized admin team?
  • How much access is required to the OpenShift layer by the application developers that are the ultimate consumers of the capabilities and runtimes that are deployed?
    • Some capabilities—such as App Connect—may require frequent OpenShift access by developers in order to deploy updated integration flows for testing.
    • For others, like MQ, API Connect, and DataPower, the server is more commonly configured up-front. In this case, developers may be granted product-level access inside that deployment rather than at the OpenShift level in order to perform self-service tasks.

OpenShift Container Platform (Kubernetes-based)

There are several technical characteristics of OpenShift Container Platform that are inherited from the upstream Kubernetes project which affect the ability to provide full isolation between tenants.

You should consider these constraints:
  • Network bandwidth cannot be easily isolated to a specific workload.
  • Custom resources (CRs) deployed in a namespace may over-consume CPU and memory, which impacts pods in other namespaces running on the same node. This is difficult to prevent. See Node placement for guidance on placing Cloud Pak capabilities and runtimes on specific nodes.
  • Ephemeral storage on a worker node is not managed well by current versions of Kubernetes, at least up to Red Hat OpenShift Container Platform 4.6. Running out of ephemeral storage causes nodes to be tainted, resulting in the mass eviction of pods, and impacting end-users.
  • A cluster runs a single version of Red Hat OpenShift Container Platform (OCP) that applies to all namespaces. For this reason, all workloads must be ready to support new versions of OCP at the same time, and any issues introduced by upgrading to a new version of OCP will apply immediately to everything that is deployed there.

Operators for OpenShift and Cloud Pak for Integration

An operator consists of:
  • Custom resource definitions (CRDs) that define OpenShift objects. CRDs are cluster-scope objects and can not be constrained to individual namespaces.
  • Controllers that manage custom resources (CRs, also known as operands), which are instances of objects defined by the CRD.

    Because in OpenShift, CRDs are objects that are shared across all namespaces, if you use two controllers that manage the same kind of object in different namespaces, the CRD must be compatible with both controllers. In general, the operator’s conversion webhook converts between the versions of the API provided by the CRD. However, having two controllers does increase the complexity of the system and with that, the chance of encountering unexpected errors.

Within OpenShift, both operators and dependencies are installed and managed by Operator Lifecycle Manager (OLM). OLM enforces the following rules for installation:
  • Because each individual CR can be managed by only one controller, if an operator is installed in a specific namespace, it cannot also be installed in all namespaces.
  • OLM installs an operator only if its CRs and its dependent CRs are each managed only by one controller. For this reason, Cloud Pak for Integration cannot be installed in all namespaces if any of its dependencies, for example, Couch DB, is installed in a specific namespace.
The implications of installing an operator in all namespaces vs. in a specific namespace are:
  • If an operator is installed in all namespaces, its controller manages all of its owned CRs within all namespaces, and there can only be one instance of that operator.
  • There is not a one-to-one correspondence between an operator and its CR, and Cloud Pak for Integration operators are tested to be backwards compatible with previous CR versions. This limits the risk in using operators that are available to all namespaces in a cluster to manage CRs in different namespaces.
  • If an operator is installed in a specific namespace, its controller manages its owned CRs only within the selected namespace. For this reason, it is possible to install copies of the operator in more than one namespace.

Platform Navigator

Once installed, the Platform Navigator allows you to easily deploy all available capabilities, such as App Connect and MQ.

The Platform Navigator operator can be installed either in all namespaces or a specific namespace on the cluster. For each Platform Navigator (PN) operator installed, only one Platform Navigator CR can be created. If the PN operator is installed:
  • in all namespaces, only one Platform Navigator per cluster can be installed and it displays instances of capabilities from the whole cluster.
  • in a specific namespace on the cluster, a separate instance must be created in each namespace where it is required, and it displays only instances of capabilities from that namespace.

Platform Navigator respects the access rights granted to each authenticated user of the cluster. This means it shows a user only the instances of capabilities and runtimes to which they have been granted access.

IBM Cloud Pak foundational services

The IBM Cloud Pak foundational services in IBM Cloud Pak for Integration enable functionality such as authentication, authorization, and licensing.

By default, only one instance of foundational services is deployed per cluster, installed in a namespace called ibm-common-services, and shared between installations. You can configure multiple instances of foundational services using custom namespaces. See Installing in a custom namespace.

The foundational services instance is typically configured to integrate with the corporate LDAP server. This allows the instance to adopt the same user registry and group structure as used elsewhere in the installation environment.

Recommended deployment approach

IBM recommends you use the approach described here for structuring your IBM Cloud Pak for Integration installation, which takes into account the considerations discussed in the previous section. These are guidelines, not requirements. You may need to follow an alternative installation approach to satisfy organizational goals and constraints.

Deploy a separate cluster for the environment in each stage of the development life cycle (development, test, production, and so on).

To restrict access to customer data and live systems, and to prevent any non-production activities (development or test, for example) from affecting customer-facing endpoints, industry best practice is to deploy a production workload on different infrastructure than non-production deployments. Even within non-production environments, the safest option is to use a separate cluster for each environment.

This practice offers the best separation and flexibility for implementing new functionality without the risk of affecting established environments. For example, to upgrade to a new version of OCP—without the risk of breaking your test environment—you should use a separate cluster for development.

Note: Using separate clusters for each environment requires deploying and managing additional resources, so users commonly share a cluster for certain environments. This practice trades isolation (improved security) for reduced infrastructure costs. You should have a clear understanding of the risks involved before embarking on this approach.

When deploying a separate cluster in each environment, install operators across all namespaces to ensure a simpler, more manageable deployment.

IBM makes this recommendation because when there is a single purpose for a cluster, there is little benefit to deploying operators in a specific namespace on the cluster.

If the IBM Cloud Pak for Integration operator is installed in all namespaces:
  • There can only be one instance of the Platform Navigator for that cluster.
  • That instance will provide access-controlled management to CR instances deployed across all namespaces.
Note: If the operator for any capability or dependency of IBM Cloud Pak for Integration is installed in all namespaces with OLM, then all capability and runtime dependencies must be installed in all namespaces with OLM.

Use a namespace for each logical grouping of users that manage deployments in the cluster.

If a single team is managing all deployed capability and runtime instances, this means using a single namespace as a starting point, subject to a need to subdivide into more than one namespace. If there are multiple independent teams, different namespaces can be used to hold each team's deployments. This arrangement benefits from the role-based access control (RBAC) provided by OpenShift for each namespace.

Depending on your organizational structure (whether there a single team for all product capabilities, or each has its own specialized administration team), you might need a specific namespace:
  • For a cross-functional team that manages instances of a variety of different IBM Cloud Pak for Integration capabilities or runtimes for a given application.
  • For a domain-specific team that manages a single type of capability (for example, MQ) on behalf of multiple applications.

Grouping a set of instances into a namespace allows the administrator to efficiently apply controls for things like access permissions, resource quotas (CPU, memory) and network policies, and to more easily filter log output.

Supporting configuration items such as Secrets and Config Maps should be created only in the namespace in which they are required by the CR instances, in order to restrict access to the minimum necessary set of users and workloads.

It can be convenient to subdivide your deployment into more than one namespace, even if the same team of people is managing instances in all those namespaces.

Using a specific namespace for each application domain or business domain can make it easier to group smaller numbers of instances together or to help match the scope of other consumers (such as application developers).

Using multiple namespaces may be particularly beneficial if there are large numbers (for example, hundreds) of instances.

If you are installing software other than IBM Cloud Pak for Integration in the same cluster, remember that IBM Cloud Pak for Integration must be installed in a specific namespace on the cluster if any of its dependencies are installed in a specific namespace). This limitation on installing operators in all namespaces may impact any additional (separate) use you may make of the same dependencies.

Alternative installation approach (common for non-production)

This section describes the characteristics of an alternative approach, in which a cluster is shared across different non-production environments.

A cluster can be shared between different environments in the development life cycle, and between different teams.

Typically, sharing a cluster between different workload is achieved by using a different namespace for each environment to maintain logical separation between the different deployments.

Because network bandwidth cannot be easily isolated to a specific workload, OpenShift does not provide perfect multi-tenancy within a cluster. This increases the risk that activity—such as performance testing or testing of an upgrade process in one namespace—will have an impact on workloads running in other namespaces. One potential drawback of sharing a cluster is that when you need to upgrade to a new version of OCP, any issues or failures that are introduced immediately affect all the environments (such as Development and Test) that are running in that cluster.

An inefficient use of resources may occur:
  • while using separate clusters in smaller organizations, where there are a limited number of instances per environment. This can be an issue even when using small worker nodes.
  • in non-production environments, where the impact of problems caused by interference between the environments seem easier to handle, for example, where a performance run in a test environment is run outside normal working hours for the developers.

In a shared cluster approach, there is some benefit to installing Operators in a specific namespace instead of in all namespaces.

The versions of the IBM Cloud Pak for Integration operators in each namespace can be independently controlled.

For this reason, there may be some additional benefit to maintaining separate upgrade cycles between different namespaces. For example, you may be able to try out a new version of Cloud Pak for Integration alongside an existing deployment.
Note: Since the Operator CRDs are objects that are in all namespaces on the cluster, upgrade behavior is not completely isolated between namespaces. A change in the CRD will apply to operators in all namespaces. This allows independent upgrade of the Platform Navigator instance in different namespaces, and the Navigator will only display CR instances running in the same namespace.
There are some exceptions:
  • In this setup, you need to create one Platform Navigator instance in each namespace. The Platform Navigator is a stateless component, so changes in the Navigator version are unlikely to affect running workloads. For this reason, there is a low risk involved in running Platform Navigator in all namespaces.
  • IBM Cloud Pak foundational services can only be installed in one namespace per cluster, so any changes in the foundational services version apply to deployments in all namespaces.
  • If an operator is installed in a specific namespace, it cannot also be installed in all namespaces.

A single instance of a resource-intensive capability or runtime, such as API Connect, can be shared across multiple user domains to optimize resource requirements.

In high-availability configurations, or any situation where multiple user namespaces need to utilize API Connect, consider deploying only one instance of each capability or runtime and allowing shared usage by the essential user workloads.

You can also exploit the multi-tenancy-style features of OpenShift that allow separation by using a different provider organization per namespace in API Connect. However, there are limits to the tenant isolation provided in this scenario.

Note: Sharing an instance may add additional complexity to implementation of the applications that use that instance. For example, it may be possible to dynamically configure the API endpoint that is used, rather than depending on a single, fixed name throughout the environments.