Azure Environments
Applies to: IBM StreamSets as a Service
A Microsoft Azure (Azure) environment represents the Azure virtual network (VNet) in your Azure account where engines are deployed.
Your Azure administrator must create a VNet in your Azure account and configure Azure credentials for Control Hub to use. You then create an Azure environment in Control Hub that represents the VNet. When you activate the environment, Control Hub connects to the VNet using the configured credentials, provisions the Azure resources needed to run engines, and deploys engine instances to those resources.
While the environment is in an active state, Control Hub periodically verifies that the Azure VNet exists and that the credentials are valid. Control Hub does not provision resources in the VNet until you create and start a deployment for this environment.
Before you create an Azure environment, your Azure administrator must complete several prerequisites.
Feature Versions
At this time, Azure environments include the initial AZURE_2022_10_22 feature version that includes all available features for Azure environments and deployments.
Prerequisites
- Register the Azure resource providers for your subscription.
- Create an Azure VNet for the IBM StreamSets Azure environment to use.
- Configure the Azure credentials that Control Hub uses to access and provision resources in your Azure VNet.
- Configure managed identities to associate with the provisioned VM instances.
- Configure resource groups that the provisioned VM instances are assigned to.
Register Azure Resource Providers
Verify that your Azure subscription is registered for the Microsoft.KeyVault resource provider.
For instructions on registering Azure resource providers, see the Azure Resource Manager documentation.
Create an Azure VNet
Create an Azure virtual network (VNet) in your Azure account.
You can use an existing VNet. However, as a best practice, create a new VNet for the exclusive use of each IBM StreamSets Azure environment.
You can use private or public subnets within the VNet, as long as the subnets can send outbound traffic to the internet.
You can configure the VNet to use the default Azure-provided DNS servers or to use custom DNS servers. If using custom DNS servers, then you must include a specific init script for all Azure VM deployments created for this environment.
For instructions on creating a VNet and on allowing subnets internet access, see the Azure Virtual Network documentation.
Network Security Group
Create a network security group that defines the required inbound and outbound rules to the VNet. You can use an existing network security group or create a new group.
- Inbound and outbound connections required by IBM StreamSets engines, as described in Firewall Configuration.
- Outbound connections to Azure Key Vault. Add the IP address of the
https://vault.azure.net
host as an allowed destination.For the list of Azure IP addresses, download the list of Azure IP ranges and service tags from this Microsoft Azure page.
Configure Azure Credentials
You grant Control Hub access to your Azure account using a service principal with a client secret. Control Hub uses the credentials to access and provision resources in your Azure VNet.
Complete the following steps to configure Azure credentials for Control Hub:
- Application (client) ID
- Client secret
- Directory (tenant) ID
- Subscription ID
Register an Application and Create a Service Principal
Use the Azure portal to register a Microsoft Entra ID (previously known as Azure AD) application. When you register an application using the Azure portal, a service principal is created automatically. The service principal functions as the identity of the application instance.
After registering the application, create a client secret for the service principal to use for authentication with Entra ID. Control Hub uses this service principal to perform tasks in your Azure account.
Create and Assign a Custom Role to the Application
Create a custom role in Azure that delegates limited access to Control Hub. Grant the role the permissions that IBM StreamSets requires. You can make compatible changes to the permissions as needed.
Configure Managed Identities for VM Instances
Configure user-assigned managed identities for VM instances in your Azure account. When Control Hub provisions VM instances for an Azure VM deployment belonging to this environment, it associates these managed identities with the VM instances.
You can configure the user-assigned managed identity used by a deployment in the following ways:
- Configure a default managed identity for the environment
- Create a default managed identity for the parent Azure environment. When you create an Azure VM deployment for this environment, you can simply use the default managed identity configured for the environment.
- Configure a unique managed identity for each deployment
- Do not configure a default managed identity for the parent Azure environment. When you create an Azure VM deployment for this environment, you must configure the managed identity to use for the deployment.
- Configure a default managed identity and override as needed
- Configure a default managed identity for the parent Azure environment. When you create an Azure VM deployment for this environment, you can use the default managed identity configured for the environment, or you can override the default and configure a different managed identity for the deployment to use.
For more information on Azure managed identities, see the Azure Active Directory documentation.
Create a Managed Identity
In Azure, create a user-assigned managed identity to associate with the provisioned VM instances. You can create a single default managed identity to use for all deployments belonging to the parent environment, or you can create a unique managed identity for each deployment.
Control Hub grants the managed identity the required permissions when it associates the managed identity with the provisioned VM instances.
Additionally, when Azure VM deployments managed by this environment are configured to use an external resource archive file stored in a private Azure Blob Storage or Azure Data Lake Storage Gen2 container, the managed identity requires read access to the container.
Configure Resource Groups for VM Instances
Configure resource groups for VM instances in your Azure account. When Control Hub provisions VM instances for an Azure VM deployment belonging to this environment, it assigns the VM instances to these resource groups.
You can configure the resource group used by a deployment in the following ways:
- Configure a default resource group for the environment
- Create a default resource group for the parent Azure environment. When you create an Azure VM deployment for this environment, you can simply use the default resource group configured for the environment.
- Configure a unique resource group for each deployment
- Do not configure a default resource group for the parent Azure environment. When you create an Azure VM deployment for this environment, you must configure the resource group to use for the deployment.
- Configure a default resource group and override as needed
- Configure a default resource group for the parent Azure environment. When you create an Azure VM deployment for this environment, you can use the default resource group configured for the environment, or you can override the default and configure a different resource group for the deployment to use.
For instructions on managing Azure resource groups, see the Azure Resource Manager documentation.
Note the names of the resource groups that you configure. You will enter the resource group names when you create Azure environments or deployments in Control Hub.
Configuring an Azure Environment
To create a new environment, click Create
Environment icon: . in the Navigation panel, and then click the
To edit an existing environment, click Edit.
in the Navigation panel, click the environment name, and then clickDefine the Environment
Define the environment essentials, including environment name and type, and optional tags to identify similar environments.
Configure Azure Credentials
Select the Azure Region
Select the region for the Azure VNet created as a prerequisite by your Azure administrator.
Configure Defaults for Azure VM Instances
Optionally, configure the default managed identity and resource group to associate with the VM instances provisioned for all deployments belonging to this environment.
Configure the Azure VNet
Select the Azure VNet created as a prerequisite by your Azure administrator, and optionally define Azure tags to apply to provisioned Azure resources.
Configure the Azure Subnet
Select the subnet and security group to use within the Azure VNet created as a prerequisite by your Azure administrator.
Share the Environment
By default, the environment can only be seen by you. Share the environment with other users and groups to grant them access to it.
Review and Activate the Environment
You've successfully finished creating the environment. Activate the environment so that you can create deployments for the environment.
- Exit - Saves the environment and exits the wizard, displaying the Deactivated environment in the Environments view.
- Activate & Add Deployment - Activates the environment and opens the deployment wizard so that you can create a deployment for the environment.
- Activate & Exit - Activates the environment and exits the wizard, displaying the Active environment in the Environments view.