Managing deployment jobs

A job is a way of running a batch deployment, script, or notebook in Watson Machine Learning. You can choose to run a job manually or on a schedule that you specify. After you create one or more jobs, you can view and manage them from the Jobs tab of your deployment space.

Types of jobs

You can create jobs for:

  • Models
  • Python functions
  • Scripts (Python or R)
  • Notebooks
  • Flows (SPSS, Data Refinery)

From the Jobs tab of your space, you can:

  • See the list of the jobs in your space
  • View the details of each job. You can change the schedule settings of a job and pick a different environment template.
  • Monitor job runs
  • Delete jobs

Refer to these sections for various aspects of job management:

Creating a job for a batch deployment

When you create a batch deployment, you then create a job for the deployment. To create a job:

  1. From the Deployments tab, select your deployment.
  2. Click New job
  3. Follow instructions in Creating a job

Creating jobs from the Assets list

From the Assets list you can create a job for a notebook, script, or flow. These assets use notebook environments and do not require a batch deployment for a run.

To create a job from the Assets page, from the asset action menu, click New job and then follow instructions in Creating a job.

Creating jobs

  1. Enter job name and description.
  2. Select the environment runtime for your job.
  3. Optional: Use environment variables to customize your job at runtime.
  4. Optional: Configure the retention options for the job. To avoid consuming resources by retaining all job metadata, you can set thresholds for saving a set number of job runs and associated logs, or set a time threshold for saving artifacts for a specified number of days.
  5. Optional: Select to schedule the job. Select the start date for this schedule. Click the calendar to select a date. Specify the start time, the time zone, and the repeat settings that depend on the frequency that you selected. Note that if you exclude certain weekdays, the job might not run as you would expect. The reason is due to a discrepancy between the time zone of the user who creates the schedule, and the time zone of the main node where the job runs.
  6. Create the job. You can create and run the job immediately, for example if you didn't specify a schedule, or you can create the job and run it later, either manually or as specified in the schedule.

Viewing jobs in a space

You can view all of the jobs that exist for your deployment space from the Jobs page. You can also delete a job from this page.

To view the details of a specific job, click the job. From the job's details page, you can:

  • View the runs for that job and the status of each run. If a run failed, you can select the run and view the log tail or download the entire log file to help you troubleshoot the run. A failed run might be related to a temporary connection or environment problem. Try running the job again. If the job still fails, you can send the log to Customer Support.
  • When a job is running, a progress indicator on the information panel displays information about relative progress of the run. You can use the progress indicator to monitor a long run.
  • For a notebook job, you can view the output for a completed notebook run. Use this feature to review visualizations or to easily compare output from multiple runs.
  • Edit schedule settings or pick another environment template.
  • Run the job manually by clicking the run icon from the job action bar. You must deselect the schedule to run the job manually.

Managing job metadata retention

When a job completes, either by executing successfully or failing, the metadata for jobs you created through the UI is stored by default for 30 days. You can override this setting when you create a deployment job. You can also manage metadata retention for deployment jobs programmatically.

Managing metadata retention and deletion programmatically

If you are managing a job programmatically by using the Python client or REST API, you can retrieve metadata from the deployment endpoint by using the GET method during the 30 days.

To keep the metadata for more or less than 30 days, change the query parameter from the default of retention=30 for the POST method to override the default and preserve the metadata. Note that changing the value to retention=-1 will cancel the auto-delete and preserve the metadata.

To delete a job programmatically, specify the query parameter hard_delete=true for the Watson Machine Learning DELETE method to completely remove the job metadata. For example:

DELETE /ml/v4/deployment_jobs/{JobsID}

Learn from samples

Refer to Machine learning samples and examples for links to sample notebooks that demonstrate creating batch deployments and jobs by using the Watson Machine Learning REST API and Watson Machine Learning Python client library.

Parent topic: Managing predictive deployments