What's new and changed in IBM Spectrum Conductor?

Key terminology changes

Spark instance group term

Spark instance groups are now referred to as instance groups. If you are familiar with previous versions of IBM Spectrum Conductor, note this terminology change.

Content within IBM Knowledge Center and on the cluster management console pages use the new terminology.

Notebook Manager, Version Management, and Anaconda Management menu names and flow

The menu names and cluster management console navigation flow for notebook, Spark version, and Anaconda management has changed for this version of IBM Spectrum Conductor. If you are familiar with previous versions of IBM Spectrum Conductor, note these cluster management console changes:

Previous flow	IBM Spectrum Conductor 2.4.0 flow
Workload > Spark > Notebook Management	Resources > Frameworks > Notebook Management
Workload > Spark > Version Management	Resources > Frameworks > Spark Management In addition to this flow change, note that the menu option is now called Spark Management instead of Version Management
Workload > Spark > Anaconda Management	Resources > Frameworks > Anaconda Management

Content within IBM Knowledge Center has been updated to reflect the new cluster management console flow.

System configurations and integrations

IBM Knowledge Center online

This version of IBM Spectrum Conductor offers documentation in one central location: online at https://www.ibm.com/support/knowledgecenter/SSZU2E. You can continue to access IBM Knowledge Center the same ways as in previous releases of IBM Spectrum Conductor (see Product documentation in IBM Knowledge Center for details), as long as you have internet access.

A copy of the documentation is no longer installed with IBM Spectrum Conductor. Instead, providing documentation online reduces installation time and files, saves on disk space for the installed host, and ensures that the content you are reading is always the most current. We regularly update documentation based on user feedback and comments, so checking the online IBM Knowledge Center ensures that you always have the most up-to-date information.

Installing, upgrading, and configuring

Rolling upgrade with IBM Spectrum Symphony

IBM Spectrum Conductor 2.4.0 supports IBM Spectrum Symphony 7.3.0; therefore, for an environment with both IBM Spectrum Conductor and IBM Spectrum Symphony, if you perform a rolling upgrade to IBM Spectrum Conductor 2.4.0, then also upgrade IBM Spectrum Symphony to 7.3.0.

Installing IBM Spectrum Conductor to a local environment and deploying instances to a shared file system

Now, with your local installation of IBM Spectrum Conductor, you can deploy instances (instance groups, Anaconda distribution instances, and application instances) to a shared file system instead of to the local environment. See Installing IBM Spectrum Conductor to a local environment and deploying instances to a shared file system for detailed steps.

Instance group deployments to root squashed shared file systems

You can enable support of root squash instance groups deployments when you install IBM Spectrum Conductor using a shared file system, by means of a new installation environment variable called ROOT_SQUASH_INSTALL, and then configuring the cluster administrator OS user to be a member of the administrator OS user group of every instance group administrator OS user. By default, the administrator OS user group of an instance group is the primary group of the instance group administrator OS user; or alternatively, the administrator OS user group can be provided as a configuration parameter when creating an instance group. For details on where this fits into the IBM Spectrum Conductor installation flow, see Installing IBM Spectrum Conductor to a shared environment.

Additionally, if you have already installed IBM Spectrum Conductor 2.4.0 without root squash deployment enabled, you can manually migrate your installation to support root squash. See Migrating IBM Spectrum Conductor to use root squashed shared file systems for these steps.

Proxy host name for WEBGUI server

When hosts in your cluster use different domains, you can now specify a proxy host name for the cluster management console web server by configuring the GUI_PROXY_HOSTNAME parameter in the $EGO_CONFDIR/wsm.conf file. For more information, see the ../reference_sym/wsm_conf_fileref.html#v482212.

Service Director as internal DNS server

Your IBM Spectrum Conductor cluster must be able to resolve IP addresses of host names. When host name resolution is not available through an external DNS server, especially in cloud environments, you can configure the Service Director in IBM Spectrum Conductor as your internal DNS server. For more information, see Running IBM Spectrum Conductor without external DNS.

Conda environments

You can now create or link a conda environment to an instance group. Conda environments do not have to be tied to a notebook.

You can now discover existing conda environments in your system.

Changed default rank for consumers

New consumers and existing consumers that do not have a rank defined in your cluster are now assigned a default rank (10000), specified by the new EGO_CONSUMER_PRIORITY_DEFAULT parameter in the ego.conf file, where previously new consumers were ranked 0 to receive the highest priority. Use EGO_CONSUMER_PRIORITY_DEFAULT to change the default consumer rank when many consumers in your cluster share the highest rank (0) and nullify the advantage of setting a consumer rank. For more information, see ego.conf reference.

Changed configuration of ESC_HOLD_SLOT_FOR_JOB_CONTROLLER to Y

During installation and with a rolling upgrade to IBM Spectrum Conductor 2.4.0, the ESC_HOLD_SLOT_FOR_JOB_CONTROLLER parameter in the egosc_conf.xml file is set to Y; in previous releases, this value was not set, and defaulted to N. ESC_HOLD_SLOT_FOR_JOB_CONTROLLER specifies that the EGOSC service should wait for the job controller to finish before releasing slot allocation (see egosc_conf.xml reference for details on this parameter).

New exportresplan command to export slot-based resource plan

To export the slot-based resource plan that is active in your cluster, you can now use the new egosh consumer exportresplan subcommand. This subcommand exports your resource plan as an XML file to a directory of your choice. You can work on the copy offline until you are ready to import the updates and apply the changes in your cluster. For more information, see the consumer reference.

Security

Options to enforce cluster-wide security settings

You can now enforce authentication and encryption settings at the cluster level to prevent users from changing those settings for instance groups. To enforce cluster-wide settings, configure one or more of the following new parameters in the ascd.conf file: CONDUCTOR_SPARK_ENFORCE_SPARK_EGO_AUTH_MODE, CONDUCTOR_SPARK_ENFORCE_ENCRYPTION, CONDUCTOR_SPARK_ENFORCE_NOTEBOOK_SSL, and CONDUCTOR_SPARK_ENFORCE_SECURITY_SPARK_AUTH. For more information, see the ascd.conf reference.

New egocontrol command

When you do not want the egosh command to run as the root user, you can now use a new egocontrol ego start command to start a remote host in your cluster as root. With this update, you can set the SUID on the egocontrol binary, then use the egocontrol ego start command to start the cluster on a remote host. For more information, see egocontrol and egosetsudoers.

Enhanced log retrieval

You can now restrict unauthorized users from downloading host and system logs, so that logon users can retrieve logs only as an OS execution user of the consumers for which they have log retrieval permissions. Previously, the logon user could download any log even without access permissions.

Log retrieval is controlled by the RestrictHostLogRetrieve parameter in the pmc_conf_ego.xml file and is always restricted, except when RestrictHostLogRetrieve is set to false. See Restricting log retrieval.

To further restrict log retrieval, define a list of directories from which users can retrieve logs and deny access through soft links by configuring parameters in a new whitelist.conf file:

• Use the WHITELIST parameter to configure directories from which users can retrieve logs.
• Use the EGO_RFA_ALLOW_SOFTLINK parameter to restrict file access through soft links.

With this security configuration, the logon user who is assigned log retrieval permissions for the associated consumers can retrieve logs as an execution user from the specified directories, both from the cluster management console and through RESTful APIs.

• To retrieve logs as an execution user from the cluster management console:
  • Go to Resources > Hosts to retrieve host logs (see Viewing log files from Host Logs).
  • Go to Reports & Logs > System Logs to retrieve system logs (see Viewing log files from the System Logs page).
• To retrieve logs as an execution user by using RESTful APIs, use the new executionuser parameter, which is now a required parameter with the following REST APIs:
  • get /ego/v1/logs/system/{hostname}/{filename}
  • get /ego/v1/logs/service/{servicename}/{serviceinstanceid}/{filename}
  • get /ego/v1/logs/service/{servicename}/{serviceinstanceid}
  • get /ego/v1/logs/system/{hostname}
  See the Resource orchestrator RESTful APIs.

If you do not want restrictions on log retrieval, you can change this configuration and allow all logon users to access logs in any directory (see Removing restrictions on log retrieval).

Extended functionality for GSS-Kerberos plug-in

When your cluster uses either Microsoft Active Directory (AD) or MIT Kerberos as the Key Distribution Center, you can now extend authentication through the GSS-Kerberos plug-in with the following configurable options in the sec_ego_gsskrb.conf file:

Enable PAM authentication

Enable PAM authentication with the ENABLE_PAM_AUTH parameter, so that users without a known password for the Kerberos user principal (for example, because they use a keytab with a random password) can log on to the cluster with their PAM user name and password.

Include or exclude user groups

Limit users added to the cluster by including user groups with the INCLUDED_USERGROUP parameter or excluding user groups with the EXCLUDED_USERGROUP, so that only users in included user groups are added or only users in excluded user groups are excluded. You can also configure the FOLLOW_GETENT_GROUP parameter to strictly follow the output of the getent group groupname command to identify members of a user group.

Keep Kerberos credential on user logoff

Retain the Kerberos credential cache on user logoff by enabling the KEEP_KRB5CC_ON_USER_LOGOFF parameter. When different applications share the Kerberos environment, use this configuration to reuse the credential cache even after the user log offs. Without this option, the Kerberos credential cache is always cleaned up when the user logs off.

For more information, see Configuring the sec_ego_gsskrb plug-in for Spark workload.

Dependency on Admin password removed

When GSS-Kerberos authentication is enabled in your cluster, daemon authentication (including authentication between EGOSC/EGO service/SSM and VEMKD; between EGO service and EGOSC; and between EGO services) is no longer dependent on the Admin user password. Previously, the EGO credential used for authentication was generated based on the Admin user and its password in the users.xml file. For enhanced security, the EGO credential is now generated following successful authentication with the Kerberos service principal (specified by the EGO_SEC_KRB_SERVICENAME parameter in the ego.conf file) and its keytab (specified by the KRB5_KTNAME parameter).

AES-256 encryption for your reporting database

You can now use an AES256 cipher to encrypt the user name and password configured for your cluster's reporting database. You can choose AES256 when you configure a new data source or modify your existing data source. For more information, see Enabling the Derby database or Setting up an external database for production.

Apache Struts no longer used or included

For security, the cluster management console no longer uses or includes Apache Struts, and instead, uses Spring framework and Spring Security. For information on the supported versions of Spring for IBM Spectrum Conductor 2.4.0, see the Spring entries in Software included with IBM Spectrum Conductor.

Instance groups and notebooks

Access control list (ACL) for permissions to instance group and notebooks no longer required

We have removed the access control list (ACL) dependency with this version of IBM Spectrum Conductor, so that you no longer require this for instance groups and notebooks. Instead, a user group based permission mechanism is used to enable access to instance group and notebook administrators and users. You can optionally specify a user group to be used by this mechanism for each instance group and notebook; or by default, this mechanism uses the primary user groups of the instance group execution user and the notebook execution user.

Before creating instance groups, assign the appropriate write permissions for your instance group and notebook deployment directories, so that the instance group and notebook administrators and workload execution users can write to these directories. For more details on instance group and notebook requirements, see Prerequisites for an instance group.

Enabling deployment permission for an administrator user group

By default, the system assigns the primary user group of the instance group or notebook execution OS user to the directories and files of the instance group or notebook deployment. You can optionally specify an administrator user group to be assigned to the directories and files of the instance group or notebook deployment, using a new optional Administrator user group field available in the instance group configuration page within the cluster management console. For details, see Defining basic settings for an instance group and Enabling notebooks for an instance group.

Additionally, if you create your own notebook packages (that is, with custom scripts for your notebook) and you specify a value for the Administrator user group field, you can add the new DEPLOY_NB_ADMIN_USER_GROUP environment variable to your deployment scripts to use that administrator user group for notebook deployment. See Creating notebook packages for details.

Enabling notebook user impersonation

You can enable Spark application user impersonation, so that Spark applications run as the submission user. This version of IBM Spectrum Conductor additionally offers user impersonation for notebooks, so that when you start a notebook, notebook services and their Spark workload run as the notebook owner OS user. The built-in Jupyter 5.7.8 notebook supports user impersonation.

To indicate user impersonation for a notebook, select the new Supports user impersonation option when adding the notebook to the IBM Spectrum Conductor cluster and enabling it for an instance group. For detailed steps on supporting user impersonation for a notebook, see Adding notebooks.

Additionally, for notebooks enabled with Kerberos user authentication, you can also specify your principal and the location of your keytab file for your notebook, using environment variables for the notebook. The system then uses this information when starting the Kerberos authenticated notebook using an impersonation user. Notebook user level customization through environment variables is new to this version of IBM Spectrum Conductor. For detailed steps, see Adding environment variables to notebooks.

Updating existing instance groups to use a new Spark or Jupyter version

You no longer require re-creating an instance group for new Spark or new Jupyter version.

Once an administrator has upgraded the Apache Spark version on the IBM Spectrum Conductor system, to a new Spark version, you can then update your existing instance groups to use that new Spark version. For prerequisites, best practices, and steps on using the new Spark version with your instance group, see Updating instance groups to use a new Spark version.

Once you have upgraded the Anaconda distribution on the IBM Spectrum Conductor system, the associated Jupyter notebook version is also upgraded. For prerequisites, best practices, and steps on updating existing instance groups to use the new Jupyter version, see Updating instance groups to use a new Jupyter version.

Preventing notebooks from starting with the instance group

You can now prevent notebooks associated with an instance group from starting automatically when the instance group is started. Setting the CONDUCTOR_SPARK_AUTOSTART_NOTEBOOKS_WITH_INSTANCE_GROUP parameter to OFF in the ascd.conf file helps manage your resources, so that notebooks do not take up slots when they are not in use. For more information, see the ascd.conf reference.

Enabling users to start or stop owned notebooks

You can now start and stop notebooks that they own as long as their user account is assigned the EGO_SERVICE_CONTROL permission. Previously, users who were assigned this permission were able to view their own notebooks, but could not start or stop them. For more information, see Starting notebooks in an instance group and Stopping notebooks in an instance group.

Enabling users to add environment variables to their own notebooks

You can now add environment variables to a notebook that you own, providing you with more customization and flexibility to manage notebooks. A notebook's service activity scripts takes the values provided in these environment variables to complete actions for the notebook, such as starting a Kerberos authenticated notebook using an impersonation user, or overriding an existing environment variable. For more information, see Adding environment variables to notebooks.

Enabling environment variables in data volume definitions for Dockerized Jupyter notebooks

You can now add environment variables in your data volume definitions for Dockerized Jupyter notebooks. With this update, you can use dollar signs ($), open curly brackets ({), and closed curly brackets (}) to define environment variables (such as /scratch/dev/${SPARK_EGO_USER}) in your host path and container path definitions for data volumes. For more information, see the ascd.conf reference.

Disabling logins for notebook users

When your environment controls user access outside of IBM Spectrum Conductor, you can disable the CONDUCTOR_NOTEBOOK_AUTH_ENABLED parameter in ascd.conf to turn off the login prompt when notebook users launch notebooks in a browser. With this parameter disabled, notebook users can access their notebook in a browser without having to log in. For more information, see the ascd.conf reference.

Simplified package creation and deployment for an instance group

We have enhanced the user experience for creating and deploying dependent packages for an instance group:

• In addition to uploading a dependent package from the package repository or from your local computer, as a convenience, you can now also create a package from a single-file. The system creates a package from the file, uploads it to the package repository, and then deploys it to all hosts in the instance group.
• Use the new package options Use $DEPLOY_HOME and Use $SPARK_HOME options to indicate that the system should use the $DEPLOY_HOME and $SPARK_HOME environment variables during package deployment as the value of the instance group's deployment and Spark home directories, respectively. If you write custom service package scripts, and you added your package using the Add packages from the repository or Upload Packages options, selecting these options allows you to also use the $DEPLOY_HOME and $SPARK_HOME environment variables in your scripts.

For details of both these enhancements, see Adding dependent packages.

Maximum of 4 GB for dependent packages

The size of dependent packages for an instance group, including notebooks, and an application instance is now increased to a maximum of 4 GB, where previously the limit was 2 GB. For more information, see Creating dependent packages or Creating notebook packages.

Specify notebook log patterns for the SparkCleanup service

By default, the SparkCleanup service cleans up logs created by the built-in Jupyter notebook: it cleans up all notebook deployment directories and notebook base data directories. If you use a custom notebook that logs to a location other than the default log under the NOTEBOOK_DEPLOY_DIR/service_logs directory, and you want the SparkCleanup service to cleanup this log, then simply use the new NOTEBOOK_LOG_PATTERNS environment variable to specify another pattern or even multiple patterns. For details on usage, see Configuring cleanup for instance groups.

Host factory framework for cloud bursting

Support for manual provisioning from the cluster management console

You can now manually submit requests to provision and return cloud resources from the cluster management console (see Provisioning cloud hosts manually), where previously you only used cURL commands or a browser-based plug-in. With this update, cloud provisioning requests are broken down for more detailed monitoring as those sent to the HostFactory service and those sent to the cloud provider (see Viewing cloud requests).

Closing cloud requests

If you submitted a request for cloud hosts to host factory, but no longer want the provisioned cloud hosts, you can now close that cloud request (see Closing requests for cloud hosts).

Microsoft Azure as cloud provider

In addition to IBM Cloud (previously SoftLayer®®) and Amazon Web Services as cloud providers, you can now provision compute hosts also from the Microsoft Azure cloud. For more information, see Configuring your cluster for Microsoft Azure.

Generic interface to connect to any cloud provider

IBM Spectrum Conductor, by default, can provision hosts from Amazon Web Services (AWS) and IBM Cloud. To use your own cloud provider, you can create a custom plug-in that implements your own logic to provision hosts through host factory. For more information, see Creating a custom cloud provider.

Host provisioning from multiple cloud providers

Hosts can now be provisioned from multiple cloud providers simultaneously for each requestor, where previously only one cloud provider could be enabled at a time. To enable multiple providers, configure one or more providers - Amazon Web Services (AWS), IBM Cloud, Azure, or your own custom cloud provider - in the providers parameter in the hostRequestors.json file. For more information, see the hostRequestors.json reference.

Bare metal server support for IBM Cloud provisioning

When your cluster is set up to provision hosts from IBM Cloud by using a base OS image, you can choose to provision bare metal servers by setting the isBareMetalFlag parameter in the softlayerprov_templates.json template file. For more information, see the softlayerprov_templates.json reference.

Easier identification of provisioned cloud hosts

You can now configure the hostPrefix parameter in the host provisioning template to more easily identify cloud hosts in a cluster. This parameter takes effect for hosts provisioned from IBM Cloud (see the softlayerprov_templates.json reference) and Microsoft Azure (see the azureprov_templates.json reference).

Support to request hosts based on a minimum resource requirement of cores and memory

When requesting cloud resources manually through RESTful APIs (with your requestor enabled in REST mode), you can now submit a request for hosts that meet a minimum requirement of cores and memory per computational unit. Use this definition to request cloud hosts that are best suited to meet the demands of your workload. For example, when workload is compute-intensive, you can specify that each provisioned core must have at least 2 GB and a total available memory of 256 GB. For more information, see Submit request for cloud hosts as a requestor.

Provisioning of AWS Spot instances

You can now configure host factory to provision Spot instances from the AWS cloud. A Spot instance is an unused EC2 instance in AWS that is available for less than the On-Demand instance price and can help lower your cloud hosts. You can request only Spot instances or a mix of both On-Demand and Spot instances (see the awsprov_templates.json reference).

Instance tags for AWS resources

Tag AWS EC2 instances, both On-Demand and Spot, by configuring the instanceTags parameter in the awsprov_templates.json template. Use this configuration to assign metadata to your EC2 instances to better organize and identify them, for example, by purpose, owner, or environment (see the awsprov_templates.json reference).

Temporary instance-profile credentials for AWS authentication

If the HostFactory service is running on an EC2 host within an IAM role that has permissions for provisioning and releasing EC2 On-Demand and Spot instances, you are no longer required to configure a credential file in the awsprov_config.json file. HostFactory authenticates with temporary instance profile credentials on the EC2 host if the AWS_CREDENTIAL_FILE parameter is not specified (see the awsprov_config.json reference).

Faster host returns

Hosts provisioned from the cloud are now returned faster. You can also control the maximum number of hosts to be returned per return request by configuring a new optional HF_RETURN_BATCH_LIMIT parameter in the hostfactoryconf.json file (see the hostfactoryconf.json reference).

cwsA requestor plug-in renamed

The cwsA requestor plug-in is now renamed as cws (see Configuration options for the cws requestor plug-in).

Sample configuration files

Host factory now includes sample configuration files to better manage configuration updates in your cluster. Keeping the samples as a handy reference, you can copy the desired configuration from the sample files to your live configuration files, then customize the configuration. Look for the following samples in the same directory as the live configuration files:

• Cloud provider templates:
  • softlayerprov_templates_sample3.8.json for IBM Cloud (see the softlayerprov_templates.json reference)
  • awsprov_templates_sample3.8.json for AWS (see the awsprov_templates.json reference)
  • azureprov_templates_sample3.8.json for Azure (see the azureprov_templates.json reference)
• HostFactory service (see the hostfactoryconf.json reference):
  • With SSL (default): hostfactoryconf.json.ssl
  • Without SSL: hostfactoryconf.json.nonssl

Resource planning

Calendar-based slot-based resource planning

New to this version of IBM Spectrum Conductor is calendar-based resource planning, which allows you to configure slot-based resource plans for specific calendar points to accommodate recurring application requirements. For example, you can have a plan for weekdays versus for weekends; or a plan for every month end, quarter, or year end. The cluster management console Resources > Resource Planning (Slot) > Resource Plan flow now includes a New button for creating and managing slot based resource plans, including calendar-based settings. See detailed steps in Creating or modifying a slot-based resource plan.

Additionally, you can now also order your plans numerically, so that IBM Spectrum Conductor uses those plans in a prioritized manner. For example, you may prioritize weekday resource plans over weekend plans. To switch the order of the plans, simply change the order in the cluster management console or use the -o option in the egosh resplancal modify subcommand.

Finally, the new egosh resplancal subcommand provides CLI for creating and managing resource plans. You can use this CLI if you are working with resource plans through a script instead of the cluster management console. See resplancal for details.

GPUs

Enable a service to use GPUs

You can define a IBM Spectrum Conductor service to use GPUs (instead of, or in addition to, CPUs) configured for your cluster. Complete this configuration by adding two new parameters to your service profile: UseGPU and ExclusiveGPU, and setting a value for an existing parameter: InstanceToSlotRatio. For detailed steps, see Enabling a service to use GPUs.

Once configured, you can then view the GPU settings for the service using the egosh service view command. For details on this command, see service.

Automatic GPU configuration

Set the new EGO_GPU_AUTOCONFIG parameter within the ego.conf configuration file to enable automatic GPU configuration; restart EGO on all management hosts in the cluster for the changes to take effect. By default, this is not enabled; however, setting this parameter to Y indicates that the LIM daemon within the in system should detect and collect information for GPU resources automatically. Previously this configuration was manual.

If you are upgrading from a previous version of IBM Spectrum Conductor and have set the EGO_GPU_ENABLED parameter, but want to use EGO_GPU_AUTOCONFIG after upgrading to IBM Spectrum Conductor 2.4.0, first run the gpuconfig.sh disable command (as described in Enabling GPUs), and then set EGO_GPU_AUTOCONFIG=Y.

Once GPUs are configured and enabled, you can then see the GPU resource information by running the egosh resource list -o ngpus command, or in the cluster management console.

Support for up to 32 GPU resources

Set the new EGO_HOST_MAX_GPU parameter within the ego.conf configuration file to indicate the maximum number of GPU resources on each host for all hosts in the IBM Spectrum Conductor cluster. You can set this value from 1 to 32, inclusive. Previously, this maximum was 4 GPUs.

If you are upgrading from a previous version of IBM Spectrum Conductor and have set the EGO_GPU_ENABLED parameter, but want to use EGO_HOST_MAX_GPU after upgrading to IBM Spectrum Conductor 2.4.0, first run the gpuconfig.sh disable command (as described in Enabling GPUs), and then set the EGO_HOST_MAX_GPU value to the number of GPU resources for your upgraded cluster.

Specify the number of GPUs for notebooks

If your cluster is enabled for GPUs, when you enable notebooks for an instance group, you can now also specify the number of exclusive GPU slots required by each notebook service. By default, the number of GPUs is set to zero. For details on this new field and where it fits into the notebook flow for an instance group, see Enabling notebooks for an instance group

Logging and troubleshooting

ArcSight CEF (Common Event Format) audit logs

When logging is enabled, you can now also save the audit logs in CEF standard, which provides a specific level of detail for the logs so that they can be analyzed with ArcSight tools.

To enable CEF audit logs for EGO functions, set the new EGO_AUDIT_LOG_CEF (and optionally EGO_CEF_NO_SYSLOG) parameters in the ego.conf configuration file.

To enable CEF audit logs for ascd functions, set the new ASC_AUDIT_LOG_CEF (and optionally ASC_AUDIT_LOG_CEF_HEADER) parameters in the ascd.conf configuration file.