Azure VM Deployments
Applies to: IBM StreamSets as a Service
You can create an Azure Virtual Machine (Azure VM) deployment for an active Azure environment.
When you create an Azure VM deployment, you define the engine type, version, and configuration to deploy to the Azure virtual network (VNet) specified in the environment. You also specify the number of engine instances to deploy. Each engine instance runs on a dedicated VM instance.
When you start an Azure VM deployment, Control Hub connects to the Azure VNet specified in the environment and then uses an Azure Resource Manager template to create an Azure Deployment. Azure Resource Manager provisions the group of VM instances in the VNet and then deploys and launches one IBM StreamSets engine instance on each VM instance.
Azure Resource Manager manages the provisioning and monitoring of the VM instances. Control Hub simply receives the status of the deployed engine instances and sends any updates to Resource Manager.
When you stop an Azure VM deployment, Resource Manager deletes the existing VM instances.
For more information about Azure Resource Manager, see the Azure Resource Manager documentation.
Before you create an Azure VM deployment, you must complete several prerequisites.
VM Instance Details
Engine Type and Version | Software |
---|---|
Data Collector 5.11.x and later |
|
Data Collector 5.8.x - 5.10.x |
|
Data Collector 5.7.x and earlier |
|
Transformer 6.0.x and later |
|
Transformer 5.8.x - 5.9.x |
|
Transformer 5.7.x and earlier |
|
Secrets
- Authentication token that the deployment uses to communicate with IBM StreamSets.
- Proxy credentials, including the HTTP and HTTPS proxy user and password, when you configure engines to use a proxy server.
Control Hub creates a unique key vault for each Azure VM deployment.
Prerequisites
- Create a Microsoft Azure (Azure) environment
- Create and activate an Azure environment in Control Hub, as described in Azure Environments.
- Configure a managed identity
- Ask your Azure administrator to configure a managed identity in Azure to associate with the provisioned VM instances. If a default managed identity is defined for the parent Azure environment, you can skip this prerequisite and simply use the default. If a default is not set or if you'd like to override the default for the deployment, see Configure Managed Identities for VM Instances.
- Create a resource group
- Ask your Azure administrator to create a resource group in Azure that the provisioned VM instances are assigned to. If a default resource group is defined for the parent Azure environment, you can skip this prerequisite and simply use the default. If a default is not set or if you'd like to override the default for the deployment, see Configure Resource Groups for VM Instances.
- Create an SSH key pair
- Control Hub does not use a secure shell (SSH) key pair to access the VM instances.
However, Azure requires that an SSH key be assigned to all VM instances.
Designate the SSH key to use in one of the following ways:
- Create a local key pair. When you create the deployment, enter the full contents of the public key to assign to the provisioned VM instances.
- Ask your Azure administrator to create a new key pair or to designate an existing key pair in Azure. When you create the deployment, select the key pair name to assign to the provisioned VM instances. For more information on using the Azure portal to create SSH keys to access Linux VM instances, see the Azure Virtual Machines documentation.
- Optionally, set up an external resource archive
- When your pipelines require external resources and when you plan to deploy multiple engine instances, you must set up an external resource archive that all engine instances can access. When your pipelines do not require external resources or when using a single engine instance to get started with IBM StreamSets, you do not need to complete this prerequisite.
Init Script for Custom DNS Servers
When the Azure VNet uses custom DNS servers and you deploy Data Collector 5.7.x and earlier or any version of Transformer, the Azure deployment requires an initialization script so that the engine can detect the hostname of the provisioned VM instance.
#!/bin/sh
set -eux
sed -i '/^ExecStart=/a\
--hostname %H --add-host %H:127.0.0.1 \\
' /etc/systemd/system/sdc.docker.service
systemctl daemon-reload
If needed, you can add additional commands to the end of this init script.
For more details about defining an init script, see Init Script.
Configuring an Azure VM Deployment
To create a new deployment, click Create
Deployment icon: .
To edit an existing deployment, click Edit.
in the Navigation panel, click the deployment name, and then clickDefine the Deployment
Define the deployment essentials, including the deployment name and type, the environment that the deployment belongs to, and the engine type and version to deploy.
Once saved, you cannot change the deployment type, the engine version, or the environment.
Configure the Engine
Define the configuration of the engine to deploy. You can use the defaults to get started.
Configure Azure VM Zones
Select the zones to provision the VM instances in. If the Azure region selected for the parent environment does not support zones, the deployment wizard skips this step.
Configure the Azure VM Autoscaling Group
Configure details about the Azure VM instances that will be provisioned.
Configure Azure VM SSH Access
Configure the SSH key to assigned to all provisioned VM instances.
Share the Deployment
By default, the deployment can only be seen by you. Share the deployment with other users and groups to grant them access to it.
Review and Launch the Deployment
You've successfully finished creating the deployment.
Editing an Azure VM Deployment
You can edit an Azure VM deployment while it is deactivated or active.
When you stop a deployment, all existing VM instances are deleted. After you edit properties and then restart the deployment, Control Hub uses Azure Resource Manager to provision a new group of VM instances and launch a new IBM StreamSets engine instance on each VM instance.
When you edit a deployment while it is active, existing VM instances might be deleted, depending on the following types of edited properties:
- General deployment or engine properties
- When you edit general deployment or engine properties while the deployment is active, Azure Resource Manager continues running the existing VM instances. Changes are replicated to all engine instances on the next restart of the engines.
- Azure VM properties
- When you edit Azure VM properties while the deployment is active, Azure Resource Manager might replace all of the existing VM instances, depending on the change. If a replacement is needed, Resource Manager replaces all of the existing VM instances. This results in engine downtime while the new instances are being provisioned.
To edit a deployment, locate the deployment in the Deployments
view. In the Actions column, click the
More icon () and then click Edit.
Tracking URL
When you view the details of an active Azure VM deployment, you can access a tracking URL to the Azure portal. Use the URL to view additional information about the Azure resources automatically provisioned for the deployment.
To access the tracking URL, click an Azure VM deployment name in the Deployments view and then locate the Tracking URL property in the deployment details.
Click the URL to open the Azure portal. The portal displays the overview page of the Azure deployment created for your IBM StreamSets deployment, listing the resources provisioned for the deployment.
Use the Azure portal to explore details about each resource and locate errors that might have occurred.
For example, if you configured the Azure VM deployment to attach a public IP address to the provisioned VM instances, you can expand the deployment details to access the public IP addresses, as follows:
The following topic provides brief tips on finding the most useful information about the provisioned resources. For more details about monitoring an Azure deployment, see the Microsoft Azure documentation.
Instances
In the overview page of the Azure deployment, click the virtual machine scale set resource, and then click
. The Azure portal lists all VM instances provisioned for the deployment.Click an instance name to view specific details about the VM instance, including the public and private IP addresses. For example, the following image displays a sample VM instance details page: