Running tests in an Azure DevOps pipeline

You must create tests in IBM® DevOps Test Integrations and APIs (Test Integrations and APIs) for the application that you are testing. You must install the IBM DevOps Test Workbench extension in your organization and configure the IBM DevOps Test Integrations and APIs Task that you want to run as a job in the Azure DevOps pipelines.

You must have completed the following tasks:

Completed the prerequisites tasks. See Prerequisites for Integration with Azure DevOps.
Installed the IBM DevOps Test Workbench extension in your organization. See Installing the IBM DevOps Test Workbench extension.
Installed an agent in the Azure DevOps pipeline. For more information, refer to Azure Pipeline agents.
Created test cases under test plans if you want to view results of the test runs on the Test Plans dashboard.
Created encrypted secrets or environment variables for your personal access token (PAT), if you want to use an encrypted PAT.

After you add the IBM DevOps Test Workbench extension in your Azure DevOps organization, you can use an existing pipeline or create a new pipeline to add the IBM DevOps Test Integrations and APIs Task. You can install an agent or use the agent that you installed in your default agent pool. You can add the Test Integrations and APIs tests as a task to your agent job, configure the task, and then run the task in the Azure DevOps pipeline.

If you have created test cases under test plans in a project on Azure DevOps, you must provide the details of the Azure DevOps URL, test plan, test case, and your PAT while you configure the test job in a pipeline so that you can view the results of the test run on your Test Plan dashboard.

Open your Organization page in Azure DevOps and perform the following steps:
1. Click the project that you want to use.
2. Initialize the repository by performing the following steps:
  1. Click Repos from the left pane.
  2. Click Initialize from the Initialize with a README or gitignore section.
    Note:
    Select the Add a README check box if it is not selected.
3. Click Pipelines from the left pane.
4. Click New pipeline.
5. Click Use the classic editor to create a pipeline without YAML.
6. Verify the project, repository, and branch for manual and scheduled builds, and then click Continue.
7. Click Empty job to open a template.
Select the pipeline and complete the following steps:
1. Change the name for the build pipeline, if required.
2. Change the name of the agent job, if required.
3. Select the Agent pool for your build pipeline.
  
  You can use the agent from the default agent pool or use the agent that you installed.
4. Select the Agent Specification for the agent.
Add a task to the agent job by completing the following steps:
1. Click the Add Task icon for the agent job.
  
  The Add tasks pane is displayed.
2. Search for the IBM DevOps Test Integrations and APIs Task that is supported by the IBM DevOps Test Workbench extension.
  
  The task that you can select is displayed.
3. Click the IBM DevOps Test Integrations and APIs Task option, and then click Add to add the task to the agent job.
  
  The selected task is added to the agent job. The task is displayed with a warning that some settings require your attention. You must configure the settings mentioned in Step 4.
Note:
You can remove the tasks that are not required in your job. Right-click the tasks in the list and click Remove selected task(s) to remove them.

Select the IBM DevOps Test Integrations and APIs Task option and configure the settings by performing the following steps:

Select the latest version of the task.

Perform the action for the task by referring to the following table:

Note:

All mandatory fields are marked with an asterisk (*) in the UI.

The following table provides the options for the task that you must configure.
Field	Description	Action
Display name	The default name of the task. The default name that is displayed is IBM DevOps Test Integrations and APIs Task.	Enter a name for the task that helps you to identify the task by changing the default name.
Configuration Type	The type of test to run.	Select from the list. You can select from the following options: Simple Parameter file Note: If you select the Parameter file option, you must provide the full path to the parameter file that you created.
Product Path	The path where Test Integrations and APIs is installed on your computer.	Enter the full path to the location of Test Integrations and APIs. For example, the path can be: C:\Program Files\IBM\DevOpsTestAPI
Parameter File	The full path to a parameter file that contains run arguments for one or more tests.	Enter the full path to the location of the parameter file that you created. For example, the path can be: D:\Projects\`my_test_parameter_file`.file
Project Directory	The path where the project containing the test is located on your computer.	Enter the full path to the location of the project containing the test. For example, D:\Projects\`my_project`.
Project Name	The name of the project containing the test.	Enter the name of the project containing the test. For example, `my_project`.
Environment	The environment defined in the test to use in the test run.	Enter the name of the environment defined in the test asset.
Test(s) to run	The name of the tests or Tests Suites to run.	Enter the name of the tests or Test Suites that you want to run by using a semicolon (;) to separate the tests or Test Suites.
JunitDir	The path where the JUnit report for the test must be saved on your computer.	Enter the full path to the location on your computer where you want the JUnit report for the test to be saved. For example, D:`\Projects\my_project\my_test_results\`.
Azure DevOps Project URL	The URL of the test project in the organization on the Azure DevOps server. Note: You must enter the details for this option if you have created test cases under Test plans in your Azure project and want to view the test results on the Test Plans Dashboard.	Enter the URL of the test project in the organization on the Azure DevOps server in the following format: `https://host/orgname/projectName` You must use this option with the following options: Azure DevOps PAT Test Plan Name Test Case Name
Azure DevOps PAT	The personal access token (PAT) of the Azure server where the test project that contains the test plan is hosted. You can also enter the variable for the encrypted PAT that you create in Azure DevOps. Note: The token must have read and write access.	Enter your personal access token. You must use this option with the following options: Azure DevOps Project URL Test Plan Name Test Case Name
Test Plan Name	The name of the test plan in your Azure project.	Enter the name of the test plan. You must use this option with the following options: Azure DevOps Project URL Azure DevOps PAT Test Case Name
Test Case Name	The name of the test case under the test plan in your Azure project.	Enter the name of the test case. You must use this option with the following options: Azure DevOps Project URL Azure DevOps PAT Test Plan Name

Expand the Control Options section and configure the settings that you want for your task.
Expand the Output Variables section and configure the settings that you want for your task.

Perform any of the following actions:
- Click Save to save the configured settings for the task.
  Note:
  The task is not queued for a run.
  You can save the task to a build pipeline and then decide to run the build at a later time.
- Click Save & queue to save the configurations and queue the run in the pipeline.
  The Run pipeline dialog is displayed.
Complete the following steps:
1. Enter a comment for the test in the Save comment field.
2. Select the agent that you configured for the test from the Agent pool list.
3. Select the branch from the Branch/tag list.
4. Add the variables and demands for the task run from the Advanced Options pane, if required.
5. Select the Enable system diagnostics check box so that the test log is generated for the test run.
6. Click Save and run.
  
  The Azure DevOps dashboard displays the progress of the job run.

You have run the tests for the application that you are testing as a job in the Azure DevOps pipeline.

You can perform any of the following actions in Azure DevOps:

View the progress of the job run on the Summary page of the pipeline.
View the test results on the Task page by clicking the task.

Important:
Although the status of the job result is displayed as a success, you must see the log file of the job run to check for any errors reported. When tests run successfully as a job, no errors are reported in the log. If errors are reported in the log, you must fix those errors and then run the job again.
View the task logs from the pipeline summary page for the job you ran.
View the test results on the Test Plans Dashboard if you created test cases under test plans in your Azure project.