Configuration options

For more advanced users, there are additional configuration options to allow for a deeper understanding of Watson NLP log files and metrics.

Prometheus Metrics

Watson Runtime reports metrics in Prometheus format on port 8086 by default. These can be very helpful for checking service health, but make develop does not yet spin up a dashboard to visualize them.

Health and Readiness checks

Within the runtime image is the GRPC health probe which can be used to check the health of the runtime gRPC server running in the Pod.

In your Kubernetes Pod specification manifest, specify a livenessProbe and/or readinessProbe for the container:

readinessProbe:
  exec:
    command: ["/bin/grpc_health_probe", "-addr=:8085"]
  initialDelaySeconds: 5
livenessProbe:
  exec:
    command: ["/bin/grpc_health_probe", "-addr=:8085"]
  initialDelaySeconds: 10

Logging Channel Convention

In this repository, logging channel names are set to be 12 characters, where terms are separated by hypens. The logging channel should provide some kind of direction as to which module is producing the log. Further, when mocks are available, as is the case with servicers, the letters I and M should be used to indicate whether the module being run is an Implementation or a Mock. As an example, MR-SERVICR-M refers to the mock of the Model Runtime Servicer, while MR-SERVICR-I refers to the fully implemented Model Runtime Servicer. When possible, channels created in tests should also include TEST as part of the channel name.

Configuration Parameters

Watson-Runtime has many configuration values that can be overridden either through environment variables or .yaml configuration files.

Configure the paths to your custom config .yamls with the CONFIG_FILES environment variable, e.g.

export CONFIG_FILES=/path/to/config.yaml,/path/to/other/config.yaml

Any configuration key can be overridden with an environment variable by using all upper-case and substituting '.' with '_'. e.g. set the tls.server.key key with TLS_SERVER_CERT.

All available configuration keys and their default values can also be found in the config file (watson_runtime/config/config.yml).

Config Key	Description
`watson_library`	Name of the Watson Library whose data model and RPCs should be used.

Config Key	Description
`alog_filters`	List of channels and their corresponding levels to be set to as a string. This is useful for silencing packages that are verbose in their logging, such as `boto3`. Format is `<channel>:<level>`, where entries are delimited by commas.
`alog_channel_width`	Number of characters that alog will show for each header when displaying logs.
`alog_thread_id`	If `x.lower() is True`, alog will show thread IDs in its header when displaying logs.
`log_formatter`	The format that we want alog to print everything as (`pretty` or `json`). Generally, you want `pretty` for development, and `json` for production.
`log_level`	Default level for logs emitted by `alog`.

Config Key	Description
`capacity`	Model Mesh's capacity for holding models (specified in bytes).
`max_loading_concurrency`	Specifies the maximum number of concurrent loads to be allowed by Model Mesh.
`max_loading_timeout_ms`	Maximum load time (specified in msec) before a load is marked as failed by Model Mesh.
`latency_based_autoscaling_enabled`	Runs model-mesh with the experimental latency based autoscaling mode
`max_model_concurrency`	When latency based autoscaling is enabled, sets the default maximum number of concurrent requests per model per instance
`max_model_concurrency_per_type.<type>`	The maximum inference concurrency for an instance of this model type
`default_model_size`	Default model size for Model Mesh (specified in bytes).
`default_model_size_multiplier`	Used when estimating the size of models in the Model Runtime. This (integer) value is multiplied by the size of the model archive (in COS/Minio) to give the estimate of the model's size.
`model_size_multipliers.<type>`	Used when estimating the sizes of specific model types in the Model Runtime. Models of this type will be multiplied by this (float or integer) value to give the estimate of that model's size.
`numeric_runtime_version`	Versioning metadata for Model Mesh (used when initializing the server).
`runtime_version`	Versioning metadata for Model Mesh (used when initializing the server).
`load_local_models`	If set to `true`, models that are mounted in `LOCAL_MODELS_DIR` will be loaded at bootup.
`local_models_dir`	The directory that contains local models that should be loaded at bootup. Defaults to `/models`.

All TLS config items expect either a PEM encoded string value or a path to a PEM encoded file. These values can contain multiple concatenated certificates.

Config Key	Description
`tls.server.key`	The server's private TLS key
`tls.server.cert`	The server's public certificate. TLS is enabled if both this and `tls.server.key` are provided
`tls.client.cert`	The client's public certificate. Mutual TLS is enabled if this is given along with `tls.server.key` and `tls.server.cert`

Config Key	Description
`service_port`	Port on which the Watson Runtime gRPC server should be run.
`metrics_port`	Port on which Prometheus should run (metric aggregation).
`server_thread_pool_size`	Number of worker threads to be used by the gRPC.
`grpc_server_sleep_interval`	The gRPC server is non-blocking when it start; after it starts, we simply enter a sleep loop. This is the interval used in the sleep command in that loop.
`strict_rpc_mode`	Boolean value, whether or not to raise an error if the model's response does not match the expected response type name. Defaults to True.

Config Key	Description
`metering.enabled`	Boolean value, whether to meter or not
`metering.log_dir`	Directory to save metrics logs.
`metering.log_interval`	Time interval to repeatedly write metrics logs (in seconds).

Configuration options

Prometheus Metrics

Health and Readiness checks

Logging Channel Convention

Configuration Parameters

Related to Serving Other Watson_* Libraries

Related to Logging

Related to Model Mesh, Model Runtime & Predict

Related to TLS

Related to General Server Configuration

Related to Metering