Configuring a ML for IBM z/OS scoring service by using shell scripts

You can use the supplied shell scripts to create and manage a standalone scoring service that is configured to run remotely away from the MLz system or in an external WLP server. The scoring service will become available to other MLz services after you manually add it to the administration dashboard.

Before you begin

Review the Installation roadmap and make sure that you've completed all preceding tasks in the sequence, including the configuration of <mlz_setup_userid> and the customization of your z/OS system environment as described in Configuring user ID for setting up Machine Learning for IBM z/OS Enterprise Edition.

Procedure

Option 1: Configure a scoring service that runs remotely away from the MLz system.
1. Locate the $IML_INSTALL_DIR/bin directory on the system that you installed ML for IBM z/OS.
  
  The $IML_INSTALL_DIR/bin directory contains the server.sh script, which you can use to create, configure, or manage a scoring service server.
2. Create a new WLP server for the scoring service by issuing the following command:
```
./server.sh create <serverName>
```
3. When prompted, respond by entering requested information or accepting the default.
  All your input is stored in the scoring.cfg.<serverName> configuration file in the $IML_HOME/configuration directory.
  
  For authentication method, make sure that you use the same keystore that you selected during the initial MLz configuration. See Configuring MLz and Configuring secure network communications for MLz for more information.
  
  At the end of the configuration process, you have the option to start the scoring service right away in the current script session or later in a separate script session:
  - Consider starting the scoring service right away if you don't plan to change the log level setting of your scoring server or the configuration of your Spark. When prompted, enter "Y" to start the service and then skip to Step 7 to verify that the service is indeed started.
  - Consider starting the scoring service later if you plan to customize the log level setting of your scoring server and the configuration of your Spark. When prompted, enter "N" to skip the option and then continue onto Steps 4 - 5.
4. Optionally, configure your Spark by customizing the spark.properties.<serverName> file.
  The ML for IBM z/OS scoring service runs Spark in local mode. A Spark session uses the following default properties, which you can modify:
```
spark.master=local[*]
spark.scheduler.mode=FAIR
spark.local.dir=./spark-temp
spark.ui.enabled=false
 
```
  Complete the following steps to modify the Spark properties file:
  1. Make a copy of the spark.properties.<serverName> file from the $IML_INSTALL_DIR/configuration/defaults directory.
  2. Move the new copy into the $IML_HOME/configuration directory.
  3. Edit the new copy by resetting existing properties or adding new ones based on your need.
    Reset the properties depending on how you want to run Spark. For example, set spark.master to local[N] if you want to control the number of threads to be used. See Spark Configuration for more information.
  4. Save the updates.
  A new Spark session will use the updated spark.properties.<serverName> file in the $IML_HOME/configuration directory.
5. Optionally, configure the log level of the scoring server.
  By default, the ML for IBM z/OS scoring service sets the log level to INFO. At this level, the log records all informational messages for all requests that are issued to the scoring server, including those for model deployment. Depending on the volume of requests your scoring service receives, the size of the log file can grow really large and fast. If the volume is consistently high, consider reducing the size of the log file by setting the log level to WARN or ERROR.
  
  MLz evaluates the following items in the exact priority order shown when determining the scoring log configuration:
  1. A user-defined log4j2.xml.<serverName> file located in the $IML_HOME/configuration directory.
    You can use this file to customize the log level and output format for the scoring service.
    The following example shows a configuration file set to the INFO log level:
    <Configuration status="WARN"> <Appenders> <Console name="Console" target="SYSTEM_OUT"> <PatternLayout pattern="[%T]|%d{yy/MM/dd HH:mm:ss}|%-5p|%c:%L| %m%n"/> </Console> </Appenders> <Loggers> <Root level="ERROR"> <AppenderRef ref="Console"/> </Root> <logger name="com.ibm.ml.scoring.online" level="INFO" additivity="false"> <appender-ref ref="Console"/> </logger> </Loggers> </Configuration>
  2. A valid log level defined in the $IML_LOG_LEVEL environment variable. You can set this variable in the .profile file. The following example shows the log level set to INFO:
```
export IML_LOG_LEVEL="INFO"
```
  3. Manually update the scoring.cfg.<serverName> file by opening the $IML_HOME/configuration/scoring.cfg.<serverName> file.
  4. If a field named log_level already exists, change its value to WARN or ERROR. Otherwise, append new line log_level=WARN or log_level=ERROR at the bottom of the scoring.cfg.<serverName> file.
  5. Save the scoring.cfg.<serverName> file.
  6. By default, if none of the above options are configured, MLz sets the scoring log level to INFO.
6. Start the scoring service server <serverName> if you choose not to do so at the end of Step 3.
  - Issue the following command to start the Liberty Profile scoring server:
```
./server.sh start <serverName>
```
  - If flask_http_port or flask_https_port is specified, issue the following command to start the Gunicorn Python server:
```
./server.sh start-python <serverName>
```
7. Verify that the scoring service is successfully started by checking the server status messages.
  - For the Liberty Profile scoring server, check the status message in the console.log file in the $IML_HOME/output/<serverName>/logs directory. The server is successfully started if you see a message similar to the following example:
```
Liberty service <serverName> started.
Scoring instance <serverName> initialized. Check $IML_HOME/iml-logs/scoring/<serverName>/console.log for details.
```
  - For the Gunicorn Python server, check the status message in the gunicorn.log file in the $IML_HOME/output/<serverName>/logs directory. The server is successfully started if you see a message similar to the following example:
```
'10/16/2023 09:08:14 PM' - INFO - gunicorn.error:278 - Listening at: https://9.30.71.31:44607 (33620922)
'10/16/2023 09:08:14 PM' - INFO - gunicorn.error:278 - Using worker: tornado
'10/16/2023 09:08:14 PM' - INFO - gunicorn.error:278 - Booting worker with pid: 66529
'10/16/2023 09:08:14 PM' - INFO - gunicorn.error:278 - Booting worker with pid: 66530
'10/16/2023 09:08:14 PM' - DEBUG - gunicorn.error:281 - 2 workers
```
8. If necessary, update a scoring service by issuing the following command:
```
./server.sh update <serverName>
```
  When prompted, enter the information you need to update the configuration of the service.
9. If necessary, stop the scoring service server <serverName> by issuing the following command:
```
./server.sh stop <serverName>
```
  If a Gunicorn server is attached, issue the following command to stop the server:
```
./server.sh stop-python <serverName>
```
10. If necessary, discover all existing servers by issuing the following command:
```
./server.sh list
```
  If necessary, remove a server by issuing he following command:
```
./server.sh remove <serverName>
```
  Verify that the server is indeed removed by issuing the list command again to display the remaining servers.
Option 2: Configure a scoring service to run in an external WLP server.
1. Locate the $IML_INSTALL_DIR/bin directory on the system that you installed ML for IBM z/OS.
  
  The $IML_INSTALL_DIR/bin directory contains the server.sh script, which you can use to create, configure, and manage a scoring service server.
2. Create a new scoring service server by issuing the following command:
```
./server.sh create <serverName>
   --wlp.install.dir <wlp_install_dir> --wlp.usr.dir <wlp_usr_dir>
```
  While <wlp_install_dir> is the path to the root directory where your existing WLP server is installed, <wlp_usr_dir> is the path to the user directory within the WLP installation directory. The default path is <wlp_install_dir>/usr.
3. When prompted, respond by entering requested information or accepting the default.
  
  All your input is stored in the scoring.cfg.<serverName> configuration file in the $IML_HOME/configuration directory. The script will also create the <wlp_usr_dir>/servers/<serverName> directory for the new server.
  
  If necessary, review the configuration parameters and settings in the scoring.cfg.<serverName> file as described in Step 3 of Option 1.
  
  For authentication method, make sure that you use the same keystore that you selected during the initial MLz configuration. See Configuring MLz and Configuring secure network communications for MLz for more information.
  
  At the end of the configuration process, do not start the new scoring server if you want to customize Spark as described in Step 6.
4. Generate configuration files for <serverName> by issuing the following command:
```
./server.sh config <serverName>
   --wlp.install.dir <wlp_install_dir> --wlp.usr.dir <wlp_usr_dir>
```
5. Copy the generated <serverName> configuration files to the directory of the external WLP server by issuing the following commands:
```
cp $IML_HOME/generated/server.xml.<serverName> <wlp_usr_dir>/servers/<serverName>/server.xml
cp $IML_HOME/generated/server.env.<serverName> <wlp_usr_dir>/servers/<serverName>/server.env
cp $IML_HOME/generated/jvm.options.<serverName> <wlp_usr_dir>/servers/<serverName>/jvm.options
cp $IML_HOME/generated/bootstrap.properties.<serverName> <wlp_usr_dir>/servers/<serverName>/bootstrap.properties
```
6. Optionally, configure your Spark by customizing the spark.properties.<serverName> file.
  The ML for IBM z/OS scoring service runs Spark in local mode. A Spark session uses the following default properties, which you can modify:
```
spark.master=local[*]
spark.scheduler.mode=FAIR
spark.local.dir=./spark-temp
spark.ui.enabled=false
 
```
  Complete the following steps to modify the Spark properties file:
  1. Make a copy of the spark.properties.<serverName> file from the $IML_INSTALL_DIR/configuration/defaults directory.
  2. Move the new copy into the $IML_HOME/configuration directory.
  3. Edit the new copy by resetting existing properties or adding new ones based on your need.
    Reset the properties depending on how you want to run Spark. For example, set spark.master to local[N] if you want to control the number of threads to be used. See Spark Configuration for more information.
  4. Save the updates.
  A new Spark session will use the updated spark.properties.<serverName> file in the $IML_HOME/configuration directory.
7. Start the scoring service server <serverName> by issuing the following command from the <wlp_install_dir>/bin directory:
```
./server start <serverName>
```
  The scoring service server will start and run in the background. If flask_http_port or flask_https_port is specified, start the Python Gunicorn server by issuing the following command from the $IML_INSTALL_DIR/bin:
```
./server start-python <serverName>
```
8. Verify that the scoring service server is successfully started by checking the server status message in the console.log file.
  The console.log file is in the <wlp_usr_dir>/servers/<serverName>/logs directory. The server is successfully started if you see the following message:
```
com.ibm.ml.scoring.online.service.ServiceManager | Scoring server started ... 
```
  If a Gunicorn server is configured and started, check the server status message in the gunicorn.log file in the output/<serverName>/logs directory. The server is successfully started if you see a message similar to the following example:
```
'10/16/2023 09:08:14 PM' - INFO - gunicorn.error:278 - Listening at: https://9.30.71.31:44607 (33620922)
'10/16/2023 09:08:14 PM' - INFO - gunicorn.error:278 - Using worker: tornado
'10/16/2023 09:08:14 PM' - INFO - gunicorn.error:278 - Booting worker with pid: 66529
'10/16/2023 09:08:14 PM' - INFO - gunicorn.error:278 - Booting worker with pid: 66530
'10/16/2023 09:08:14 PM' - DEBUG - gunicorn.error:281 - 2 workers
```
9. If necessary, update a new scoring service by issuing the following command:
```
./server.sh update <serverName>
   --wlp.install.dir <wlp_install_dir> --wlp.usr.dir <wlp_usr_dir>
```
  When prompted, enter the information you need to update the configuration of the service.
10. If necessary, stop the scoring service server <serverName> by issuing the following command from the <wlp_install_dir>/bin directory:
```
./server.sh stop <serverName>
```
  If a Gunicorn server is attached, issue the following command to stop the server:
```
./server.sh stop-python <serverName>
```
11. If necessary, discover all existing servers by issuing one of the following commands:
```
./server.sh list
```
```
./server.sh list  --wlp.install.dir <wlp_install_dir> 
   --wlp.usr.dir <wlp_usr_dir>
```
  If necessary, remove a server by issuing one of the following commands:
```
./server.sh remove <serverName> 
   --wlp.install.dir <wlp_install_dir> 
   --wlp.usr.dir <wlp_usr_dir>
```
  The remove command will remove all the configuration files and the generated files of <serverName> in the $IML_HOME/configuration and $IML_HOME/configuration/generated directories.
  
  Verify that the server is indeed removed by issuing the list command again to display the remaining servers.

What to do next

Add the scoring service you just created to the Scoring Services page of the administration dashboard. See Adding a ML for IBM z/OS scoring service to the administration dashboard for instructions. You must complete this task; otherwise, the new scoring service will not be available in the ML for IBM z/OS user interface for model deployment.
Optionally, configure and run the ML for IBM z/OS scoring service in a CICS® region as described in Configuring ML for IBM z/OS scoring services in a CICS region.
Optionally, configure the ML for IBM z/OS scoring service for high availability as described in Configuring scoring services for high availability.