Test and simulation customization details

You run the sample in the Decision Center Business console.

Running the sample

In this sample, you do the following tasks:

  • Define and run a test suite that uses an Excel scenario data file.
  • Run a predefined simulation.
  • Define and run a test suite that uses a custom data provider (see Setting up custom data providers).
  • Define and run a simulation that uses different data providers and custom KPIs (see Creating custom KPIs).

Running a test and a simulation normally

You go through the usual activities for running a test and a simulation.

Defining and running a test suite that uses an Excel scenario data file
You create a test suite that includes data with expected results for tests. Then, you run the suite and see the actual results in a report.
  1. Click Tests to open the tab, and then Test Suites to open the subtab.
  2. Click the Add button "" to create a new test suite.
  3. Select loan validation with score and grade for the operation, and click OK.
  4. Change the name of the test suite to ExcelTest.
  5. For the file, click Choose and browse to <InstallDir>/teamserver/tutorials/assessmentFailure.xlsx.
  6. Click Save and Run. In the windows that opens, click Create New Version , and then OK to run the test suite.
  7. On the Reports subtab, open the test report when its status shows a yellow mark. It shows that one test ran successfully and two tests failed. If you expand the failed tests and open the Details window, you can see the actual results and the expected results.
  8. Close the report.
Running a predefined simulation
You use a predefined simulation that includes representative data to see the output of the sample decision service.
  1. Click Simulations to open the tab, and then open the Simulations subtab by clicking its name.
  2. Hover over File Simulation, and click the Run button "".
  3. Click OK in the Starting simulations window.

    The Reports tab opens and shows the progress of the simulation. When the simulation finishes, its status changes to green.

  4. Click File Simulation Report to open the report.
  5. Close the report.

Running custom tests and simulations

You go through the activities for customizing tests and simulations.

Creating a custom test provider
You create a custom test provider for the decision service.
  1. Open the Tests tab, and then open the Test Suites subtab.
  2. Click the Generate Scenario File button.
  3. Change the scenario file format to Test Data Provider. A section opens to define the tests.
  4. Expand the loan report in the Tests section, and select approved. Keep the operator as equals.
  5. Enter Approved as the identifier, and Approved is as the name.
    Note: When you define a test, you provide a name that is used in the test report to display the results, and an ID that is used in the provider code to identify the test.
  6. Select messages, and select contains as the operator. Note that the operator varies according to the type of the selected value.
  7. Enter MessagesContain as the identifier, and Messages list contains as the name.
  8. In Expected execution details, select The number of rules fired and equals as the operator. Note that you can select numeric comparison in the list of operators.
  9. Enter RulesFiredCount as the identifier, and Count of fired rules as the name.
  10. Click main in the breadcrumbs to return to the Test Suites tab. You do not have to save this description because the sample uses predefined descriptions.
Defining and running a test suite that uses a custom data provider
You set up and run a test by defining a test suite that uses a custom data provider. After the run, you can check the test results in a report.
  1. On the Test Suites tab, and click the Add button "" to create a new test suite.
  2. Change the name of the test suite to CsvTest.
  3. For the file to use, click Choose and browse to <InstallDir>/teamserver/samples/simulationcustomdatasource/simulationdatasource/data/Loan Csv Test Data Provider.tstx.

    The file format changes to Test Data Provider, which is the custom provider that is defined by the selected file. Keep testdataprovider as the name of the dataset.

  4. Click Save and Run. In the windows that opens, click Create New Version , and then OK to run the test suite.
  5. On the Reports subtab, open the test report when its status shows a green check mark. It shows that eight tests ran successfully and provides further information on the run.
  6. Close the report.
You can create another test by using the following data file, which randomly generates data and values: <InstallDir>/teamserver/samples/simulationcustomdatasource/simulationdatasource/data/Loan Random Test Provider.tstx
Checking deployed resources in the Rule Execution Server console
You look for the resources that were deployed during the tests.
  1. Sign out of the Business console.
  2. Use the following URL to open the Rule Execution Server console:

    http://localhost:<PORT>/res

    The port number can be 9090. It might differ on your computer.

  3. Sign in by using resAdmin as the username and the password.
  4. In the Explorer tab, expand Resources in the navigator tree. It shows the resources that were deployed during the tests:
    • loan-validation-xom.zip: Contains the XOM classes of the Loan Validation Service decision service.
    • simulationSample.zip: Contains the Java™ code of the custom providers.
  5. Sign out of the Rule Execution Server console.
Defining custom KPIs
You create custom KPIs to use in simulations.
  1. Sign back in to the Business console as Paul, and return to the main branch of the Loan Validation Service decision service.
  2. Click Simulations to open the tab, and then open the KPIs subtab.
  3. Click the Add button, and choose the loan validation with score and grade operation.
  4. Enter Messages for Rejected Loans for the name.
  5. Select When selected, this KPI is computed with a custom implementation.
  6. Enter MessagesForRejectedLoans for the KPI ID.
  7. Save your work.
  8. Repeat steps 3-7 to create another custom KPI:
    • Name: Message Occurrences for Rejected Loans
    • ID: MessageOccurrencesForRejectedLoans
Defining a report template that uses the custom KPIs
You create a template that uses the custom KPIs to generate reports for simulations.
  1. Open the Report Formats subtab.
  2. Hover over the existing report Loan Report Template and click the Duplicate this report format button.
  3. Edit the copy:
    1. Change the name to Template with custom KPIs.
    2. Add a new section by clicking on the Add section button. Scroll to the bottom of the report to find the new section. Move it to the top by using Move up in the Actions menu.
    3. Name it Messages for rejected loans.
    4. Drag the Messages for Rejected Loans KPI to this section.
    5. Drag the Message Occurrences for Rejected Loans KPI to this section, and edit its configuration:
      1. Hover over the KPI, and click the KPI Display Configuration button "".
      2. Choose Bar as the display type.
      3. Enter Messages as the X-Axis title.
      4. Enter Occurrences count as the Y-Axis title.
  4. Save your work.
Defining a simulation that uses a custom data provider and custom KPIs
You set up a simulation that uses a customer data provider, and generates reports by using the custom KPI report template.
  1. Open the Data subtab in the Simulations tab.
  2. Click the Add button to upload data.
  3. Enter Basic Provider as the name.
  4. Select loan validation with score and grade as the operation.
  5. Browse to <InstallDir>/teamserver/samples/simulationcustomdatasource/simulationdatasource/data/Loan Custom KPI (Basic).simx. The file uses queries to gather data for the simulation and the custom KPIs.
  6. Click Create.
  7. Open the Simulations subtab.
  8. Click the Add button "" to create a new simulation configuration.
  9. Enter Generated Simulation as the name.
  10. Set Server to Test and Simulation Execution, and Report format to Template with custom KPIs.
  11. Change the title of the simulation report to Generated Simulation.
  12. Set the input data to Basic Provider.
  13. Set the Query parameter:
    • Scenario count: 1,000
    • Minimum amount: 0
    • Maximum amount: 1,000,000
  14. Click Save, and then click Create New Version.
Running a simulation with a custom data provider and KPIs
You run a simulation that uses the custom KPIs in reports.
  1. In the Simulations subtab, hover over the Generated Simulation configuration, and click the Run button "" to start the simulation.
  2. Click OK in the Starting simulations window.

    The Report subtab opens to show the progress of the simulation. When the simulation finishes, its status changes to green.

  3. Click the report Generated Simulation. It shows that 1,000 scenarios were run.
Running other custom simulations
You can find additional sample files in <InstallDir>/teamserver/samples/simulationcustomdatasource/simulationdatasource/data. You can use them in the following custom simulations:
  • A simulation that uses custom data and custom KPIs running in multithread:
    • Using a new data source created with the file Loan Custom KPI (Multi Thread).simx
    • Using the report format Template with custom KPIs
    • With the query parameters:
      • Partition count 4
      • Thread Count 4
      • Scenario Count 1000
      • Minimum Amount 0
      • Maximum amount 1000000
  • A basic simulation that uses custom data and no custom KPIs:
    • Using a new data source created with the file Loan Data Provider (Basic).simx
    • Using the report format Loan Report Template
    • With the query parameter:
      • Scenario Count 100
  • A simulation that uses an enumeration and date parameters with custom data and no custom KPIs:
    • Using a new Data source created with the file Loan Data Provider (Min-Max-Year Params).simx
    • Using the report format Loan Report Template
    • With the query parameters:
      • Scenario Count 100000
      • Minimum Amount 10000
      • Maximum amount 1000000
      • Loan Submission Year 2018
      • Birth Date of the Borrower 2/5/1982
  • A simulation that uses custom data and no custom KPIs running in multithread with the number of threads not editable:
    • Using a new Data source created with the file Loan Data Provider (Multi Thread).simx
    • Using the report format Loan Report Template
    • With the query parameter:
      • Scenario Count 100000

Rebuilding the sample

If you want to redo this sample, follow these instructions to rebuild it:
  1. Open the Samples Commands tab in the Samples Console.
  2. Expand Samples Commands > Decision Center > Test and simulation customization.
  3. Double-click clean.
  4. Double-click build.

    Rule Designer builds the custom JAR files.

How the sample works

The Loan Validation Service decision service is ready to use in the sample server. It includes predefined metrics, KPIs, data, and a simulation.

The source code for the custom classes is built when you use the build command in the sample. You are deploying it as a new resource in the Loan Validation Service decision service to make it available for tests and simulations. This sample is in <InstallDir>/teamserver/samples/simulationcustomdatasource. You can view and modify the sample source files in Rule Designer. To import the sample into your workspace, click the Import projects link in the Samples and Tutorials view, and switch to the Java perspective.

Open the provider description in InstallDir/teamserver/samples/simulationcustomdatasource/simulationdatasource/data/Loan Data Provider (Basic).simx. You see that it requires <uri>resuri://simulationSample.zip/1.0</uri>.

You define the Csv test custom provider in <InstallDir>/teamserver/samples/simulationcustomdatasource/simulationdatasource/data/Loan Csv Test Data Provider.tstx. The file contains the following information:

  • The URI to access the provider classes: <uri>resuri://simulationSample.zip/1.0</uri>
  • A dataset.name parameter that uses Dataset as the display name, and testdataprovider as the default value.
  • The reader class set to simulation.TestSuiteCsvReader.
  • The predicate list that defines the tests. The tests were generated in the Business console. The reader code uses the ID to identify the tests.
Note: For simplicity, the same compressed file contains the CSV data file and the custom provider classes.

The following sections show the predefined artifacts in the simulation.

Metrics
The metrics are listed on the Metrics tab. They can be divided into three categories:
  1. Metrics that produce a value for all loan requests.
  2. Metrics that produce a value only when the loan score and its associated grade are computed, so only for loans that pass the validation phase.
  3. Metrics that produce a value only for approved loans, so only for loans with an acceptable grade.
KPIs

The KPIs are listed on the KPIs tab.

Data
This sample uses the following files, which you can find in the directory <InstallDir>/teamserver/samples/simulationcustomdatasource/data:
  • loanvalidation-simulation.xlsx: Excel scenario file
  • /bin/simulationSample.zip: The simulationSample.zip file is generated in the bin directory when you build the sample. This compressed file contains all the custom Java™ code that defines a new connector to create scenarios with random data and to collect data for custom KPIs.
  • Several .simx files that contain data for custom simulations:
    • The URI reference to the Java code of the connector <uri>resuri://simulationSample.zip/1.0</uri>
    • The parameter definitions as properties. For example: <property name="scenario.count" xsi:type="longProperty" displayName="Scenario Count"><value>10000</value></property>
    • The job descriptor
      • With a reader that specifies the fully qualified name of the reader class <jsl:reader ref="simulation.RandomReader">
      • With a partitioner that specifies the fully qualified name of the partition mapper class to run the simulation with multiple threads <jsl:mapper ref="simulation.SimplePartitionMapper">
      • With a partition Analyzer that specifies the fully qualified name of the partition analyzer class to run the simulation with multiple threads <jsl:mapper ref="simulation.SimplePartitionAnalyzer">
      • With a partition Collector that specifies the fully qualified name of the partition collector class to run the simulation with multiple threads <jsl:mapper ref="simulation.SimplePartitionCollector">
      • With a listener that collects data for the custom KPIs < jsl:listener ref="simulation.SimpleWriteListener" />

Source files

You can find this sample in the directory <InstallDir>/teamserver/samples/simulationcustomdatasource.

The simulation package contains the following files under /src/simulation:
  • RandomReader.java: The reader class that implements the open and readItem methods to initialize the simulation and get the scenarios.
  • RandomInputGenerator.java: The generator class that builds the loans and borrowers for a scenario.
  • SimplePartitionMapper.java: The partition mapper class that implements the mapPartitions method to run the simulation on multiple threads
  • TestSuiteCsvReader.java: The reader class that implements the open and readItem methods to initialize the test suite and get the scenarios that use a CSV-formatted data file.
  • TestSuiteRandomReader.java: The reader class that implements the open and readItem methods to initialize the test suite and get the scenarios that use a random generator.
  • simulationcustomdatasource/simulationdatasource/data: Contains the simulations and test suites description file and the CSV data file.
  • KpiData.java: The utility class that builds the data that is required by the custom KPIs.
  • SimplePartitionAnalyzer.java: The analyzer class that receives the partition collector results and aggregates them.
  • SimplePartitionCollector.java: The collector class that sends intermediate results to the analyzer.
  • SimpleWriteListener.java: The listener class that implements the afterWrite method to retrieve the KPI data.
Note:

The simulation runtime component is built on top of the JSR 352 Java batch specification.