Managing Engines
- Assign labels to engine instances.
- Override resource thresholds for engine instances.
- Restart engine instances.
- Balance jobs on Data Collector engine instances when the jobs are enabled for pipeline failover.
- Shut down engine instances. If you shut down an engine instance, you can later use the command line to run a command to start the instance.
- Delete engine instances.
You cannot shut down or delete engine instances belonging to Control Hub-managed deployments, such as GCE and Kubernetes deployments. Instead, engine instances are automatically shut down or deleted as needed when you edit the deployment. For example, if you edit a GCE deployment to decrease the number of engine instances from 3 to 2, Google Deployment Manager shuts down one of the engine instances and then deletes the engine and the VM instance where the engine runs.
Restarting Engines
Restart an engine instance to apply updates made to the engine configuration in the deployment.
For example, if you edit a deployment to add additional stage libraries, you must restart all engine instances managed by the deployment. During the restart, the stage libraries are installed on the engine instances.
During the restart process, the engine instances shut down and then automatically restart.
Balancing Jobs on Engines
From the Engines view, you can balance all jobs enabled for failover and running on specific Data Collector engine instances.
When balancing the jobs, Control Hub redistributes the pipeline load evenly across all available Data Collectors that have the necessary labels and that have not exceeded any resource thresholds.
For more information about balancing jobs on Data Collector engines, see Balancing Jobs Enabled for Failover.
Shutting Down Engines for Self-Managed Deployments
You can shut down an engine instance belonging to a self-managed deployment, and then manually start the engine instance at a later time. Alternatively, you can stop a self-managed deployment to shut down all engine instances belonging to the deployment.
After an engine instance shuts down, you cannot use the Control Hub UI to manage the engine instance. You must manually start the engine instance from the command line.
- In the Control Hub Navigation panel, click .
- Click an engine type tab.
- Select one or more engine instances that you want to shut down.
-
In the toolbar above the engines list, click the More
icon (
) and then click Shut Down Engines.
- Click OK to confirm, and then click Close.
Starting Engines for Self-Managed Deployments
If an engine instance belonging to a self-managed deployment shuts down, you cannot use the Control Hub UI to manage the instance. You first must manually start the engine instance from the command line.
Starting from a Tarball Installation
You can manually start an engine instance that has been installed from a tarball.
Starting When Using a Proxy Server
To manually start an engine instance version 5.0.0 and later that has been installed from a tarball and has been configured to use a proxy server, you must define the proxy properties as environment variables when you start the engine. You can copy the required environment variables from the installation script generated for the self-managed deployment.
- Data Collector documentation
- Transformer documentation
- Transformer for Snowflake documentation - Applicable when your organization uses a deployed Transformer for Snowflake engine.
Starting from Docker
You can manually start an engine instance that has been installed from a Docker image.
Deleting Engines from a Self-Managed Deployment
You can delete an engine instance from a self-managed deployment.
You cannot delete engine instances from Control Hub-managed deployments. Instead, engine instances are automatically deleted as needed when you edit the deployment.