Release notes - Guardium Insights Version 3.4.0

IBM Guardium Insights is a hybrid cloud data security hub that helps you improve visibility into user data activity and risk. Guardium Insights helps you protect data more efficiently, enhance information technology flexibility, and reduce operational costs as you embrace new business paradigms (such as moving data to the cloud). Guardium Insights helps reduce the cost and complexity related to collecting, managing, and retaining data security and compliance data. It provides new analytics to enhance threat investigations - and it provides quick reporting functionality (including prebuilt reports). Risk scoring and alerting in Guardium Insights help you prioritize your activities.

Version 3.4.x This content only applies to Guardium Insights Version 3.4.x.

Guardium Insights is a powerful tool that can help you secure your data. Simple to use, Guardium Insights allows you to set up connections to your data sources.

Guardium Insights provides tools to help you analyze data:

  • Outlier mining: Detecting anomalies in activities and exceptions.
  • Risk events: Identifying assets at risk using broad data points.
  • Reports: Dive into the raw data for deep investigation.

Contents

Download Guardium Insights v3.4.0

Guardium Insights V3.4.0 can be downloaded as an archive file (2.4.0.tar.gz) from: https://github.com/IBM/cloud-pak/tree/master/repo/case/ibm-guardium-insights

You can install only the products for which your site is entitled.

For further instructions, read the README.md file located after unzipping the latest tar file.

Install Guardium Insights v3.4.0

Before installing Guardium Insights, review the system requirements.

This offering is deployed as a new installation of Guardium Insights – or as an in-place upgrade. Please follow these instructions:

What's new in IBM Guardium Insights Version 3.4.0

Ease of use
Streaming analytics consolidation with risk events: Outliers are detected from streaming data. This surfaces potential emerging risks (for data that is not subject to policies). With this new feature, you can see and react to emerging threats more quickly.
Technical debt
Improvements to mini-snif have reduced the number of duplicate instance messages.
System health dashboard
A system health dashboard has been introduced in Guardium Insights with fifteen new dashboard cards. You can now monitor the health of your full-stack Guardium Insights deployments (including OpenShift®). This new dashboard can be added by selecting it from the dashboard templates view. For more information, see Dashboards.

Three of the health dashboard cards (PVC Usage, Pod Restart and Direct Streams Ingestion) require Prometheus to be enabled. For more information, see Configuring Prometheus.

Slack integration
The Slack integration is used to send outgoing messages from Guardium Insights to Slack, an instant messaging platform. Customers of all Guardium products can send notifications from within Guardium to Slack, increasing efficiencies and keeping up with modern ways of working. You can create an app in your Slack workspace and then configure Guardium Insights with its incoming webhook or bot user token (with the chat:write scope). After integration, you can use Slack to receive notifications and policy alerts. For more information, see Slack configuration.
Active queries monitor
The active queries monitor is a new dashboard card and report that shows key information about queries. The report shows the origin of the query, the user who ran it, its name, start time, elapsed time, and status. In addition, you can use the report to view the SQL of the query and, for SELECT statements, stop the query.

With this feature, you can view the status and progress of running jobs, and cancel them if needed, helping them to prevent system performance issues.

Asset inventory
  • View the connected entities of an asset and their relationships with an interactive topology map.
  • Six new cards are added to the Classic Guardium Insights overview dashboard to view high-level asset information.
  • Along with the default list of all assets, you can also view the list of assets that are grouped by hostname.
  • You can now create a category when you add auto-tagging rules for assigning tags to the assets.
For more information, see Asset inventory.
Risk event categories feedback

The risk event categorization has been enhanced and is now based on a machine learning model. With your help in providing feedback, this model can adjust itself to your organization’s needs. The machine learning model uses both positive and negative feedback to fine-tune the categorization. It uses an incremental learning algorithm that modifies the model gradually with each feedback provided so that a single case does not have a disproportionate impact. For more information on the risk event categories and feedback see, Risk event categories and Providing feedback respectively.

Variants
Variants allow you to use report data more easily by customizing reference data inside reports, as required. Customization of report data also allows for better joins to occur with custom data. For more information, see Working with variants.
Workflows
With Guardium Insights workflows, you can intuitively schedule jobs in simple or advanced manners, as suits your needs. When you create a workflow in Guardium Insights, you are setting up a process for scheduling jobs and then tracking the distribution, review, and completion of those jobs. You can schedule reports to run and set up workflows for those reports - or you can schedule import jobs (these require an integration for importing and exporting data, which you can set up when you create the workflow if you don't already have one).
Universal connector
  • Cloud databases: Support added for PostgreSQL, AWS PostgreSQL, and Aurora MySQL.
  • On-premises databases: Syslog support added for PostgreSQL, EnterpriseDB PostgreSQL, and Yugabyte.
For more information see Connecting to data sources by using the universal connector.
GDPR Compliance program
A new guided compliance program with enhanced user experience was added to help you secure your data and achieve General Data Protection Regulation (GDPR) compliance. For more information, see Compliance milestones.
Reports
You now have the ability to display report data for a customized time period. You can group timestamps by the hour, day, week, month, year, or a specific date.
Enhanced dashboard filtering
This release provides the ability to add global dashboard filters by clicking individual cells in report-type cards and selecting filter criteria. Once set, global filters are applied to all cards on the dashboard that support the criteria, allowing you to quickly identify and visualize stories in your data.
Improved group population
When customizing dashboards, you can now add a card for a group. This allows you to add members to your groups from your dashboard. In addition, you can now add members to groups from within a report and you can automate group population by creating a workflow for a report that adds report results to a group.
Notifications for risk events and policy alerts
Risk events and policy alerts can now notify users using SMTP, Syslog, SNMP trap, Slack, and ticketing Integrations.
Data ingested after a Guardium Insights update is placed in new reporting tables
Online reporting restricts time boundaries to be on either side of the update time. Times before the update will use the previous schema. Times after the update will use the new reporting tables. Offline reports allow report time to span the update time. Offline reports will merge the results across the old and new reporting tables.
Joining custom data sets to reports
If you create a custom data set, you can now join it with report data.
Support for installing Guardium Insights on Amazon Elastic Kubernetes Service (EKS)
See Installation on Amazon Elastic Kubernetes Service (EKS) to learn how to install Guardium Insights on Amazon Elastic Kubernetes Service (EKS).

Known limitations and workarounds for Guardium Insights v3.4.0

Table 1. Known limitations and workarounds for Guardium Insights v3.4.0
Issue key Description
INS-29331 In rare cases, there are Db2® errors for services such as the reports and risk services. These may prevent report execution or risk event generation. When this occurs, these errors are seen in the logs for the related service:
SQLCODE=-1803, SQLSTATE=57056, SQLERRMC=NULLID.SYSSN200 0X5359534C564C3031, DRIVER=4.26.14
SQLCODE=-901, SQLSTATE=58004, SQLERRMC=Plan/Environment mismatch!, DRIVER=4.26.14

Workaround: See Db2 errors for reports and risk services.
INS-37220 After upgrading Guardium Insights, the datamart-processor may not be able to write files to storage. As a result, data ingestion no longer takes place (the files are not ingested, but they are preserved).

Workaround: To re-upload the files that have been preserved - and to resume ingestion - restart ssh-service.
  • INS-37007
  • INS-42808
 After upgrading Guardium Insights from version 3.2.x to version 3.3.x and then to version 3.4.0, universal connector connections do not work due to a certificate error.

Workaround: See Existing universal connector certificate does not work in a restored environment.
INS-37352 When there are very large amounts of data, the Data mart ingestion page displays this error:
Data mart unavailable Cannot load data mart statistics. Refresh the page to try again

Workaround: If the Data mart ingestion page displays this error, you can access the data mart ingestion information by opening the Data mart ingestion status report. This report includes data marts collected from both collectors and aggregators. To open the reports page, select Reports in the main menu. Open this menu by clicking the main menu icon (main menu))
INS-37724 When working with compliance milestones, you can Refine alerts with the Configure alert recipients action. When you choose this action and refine alerts, you can elect to send emails for actions. When you click the Send email to action and then click Invite users, the resulting landing page includes an Add users button that does not work.

Workaround: Go to the user management screen and add the user. Then return to the Refine alerts page to add the user to the list.
INS-38008 Upgrading Guardium Insights fails with non-zero return code error when the length of the spec.guardiumInsightsGlobal.ingress.hostname value in your custom resource (CR) file is longer than 58 characters.

Workaround: Before upgrading, ensure that the length of the spec.guardiumInsightsGlobal.ingress.hostname value in your custom resource (CR) file is 58 characters or fewer.
INS-39694 After modifying data retention settings, the new settings do not take effect until you restart the data retention pod.

Workaround: Restart the data retention pod after changing the settings.
INS-41777 Guardium Insights upgrade becomes stuck because CSV does not update

Workaround: See Guardium Insights upgrade becomes stuck because CSV does not update.
INS-41829 The schedule for data marts v5 is missing after upgrading Guardium Insights.

Workaround: Before upgrading to Guardium Insights v3.4.0, you must stop streaming.

When you start data mart streaming after upgrade completion, enter a new date for v5 data marts. The initial start date can be set in the Schedule managed units export wizard in the page for the central manager. This date should be earlier than the date on which you stopped streaming data marts.
INS-42573 Guardium Insights displays a 500 Internal Server Error when opening a compliance milestone.

Workaround: Reset user configurations for the environment before opening compliance milestones.
INS-42575 After upgrading Guardium Insights, logging in to the application takes several seconds.

Workaround: Restart the Service Pod before logging in to the application.
INS-42580 During upgrade, the Guardium Insights custom resource (CR) appears to display an intermediary status for several minutes (for example, upgrading from version 3.3.3 to 3.4.0 displays an extended status that indicates upgrade to version 3.3.5).
Workaround: This is expected behavior. The upgrade will commence during the next operator reconciliation. If there was a reconciliation running prior to patching the CR, the status of the Guardium Insights CR may take some time to update. You can safely force the operator to upgrade immediately by running this command:
oc delete pod -lapp.kubernetes.io/name=ibm-guardium-insights-operator
INS-42659 AWS streams appear as unhealthy after upgrading Guardium Insights.

Workaround: Ensure that you have installed Tenant Minisnif and Tenant GUC Custom Resources Version 3.3.4 or later (see this step).
INS-42701 After connecting to DynamoDB with the universal connector, traffic for the connection is not captured in reports.

Workaround: Download the Dynamo Db plug-in and upload it to Guardium Insights (see Connecting to data sources by using the universal connector).
INS-42822 After upgrading Guardium Insights and restoring a backup, newly-created universal connections appear as Unhealthy and pods are stuck in a Pending state.

Workaround: Increase the storage volume count or contact IBM Cloud support.
INS-42885

Guardium Insights v3.4 changes the data model from v3.3 in order to improve report and ingestion performance. Shortly after upgrade, you may see scheduled reports running longer than usual or, in some circumstances, failing. The reason behind this may be that the data in the old data model and the new data model are being joined into a single result set in an attempt to minimize disruption. If you encounter longer than normal scheduled report execution times, this should be temporary and subsequent runs should be much faster.

Workaround: If scheduled reports are failing, inspect the reports-runner pod logs. If you encounter an SQLCODE=-4712, SQLSTATE=5U026 error, consider turning on the Table join optimization feature in the Tenant settings. If the problem persists or if you are encountering ERRORCODE=-1224, SQLSTATE=55032 instead, consider temporarily splitting the report into smaller time frames. These issues should only occur if the scheduled report includes a date range both prior to and after the v3.4 upgrade. Future reports should not encounter these issues.
INS-42890 Reports in the Active report categories (for example, Active Full SQL, Active Exception, and Active Policy Violation) can be used to debug issues with traffic capture, policy rule configuration, and for instantaneous confirmation of traffic capture for direct-streamed data sources in Guardium Insights.

These reports may contain duplicate data. The duplicates will not appear in the main reporting categories on the audit data (for example, DB Activity, Full SQL, and Policy Violation) as the processing of the active data to the permanent data consolidates and removes the duplication.

Workaround: None for the Active report categories. You can use the main report categories instead.
INS-42960 After restoring a backup to Guardium Insights, Guardium Data Protection connections can become red and data marts are not pulled from Guardium Data Protection. This occurs rarely.

Workaround: See Existing data mart pull configuration does not work in a restored environment.
INS-43003 After upgrading Guardium Insights from version 3.3.0 to 3.4.0, some of the universal connector connections became unhealthy.

Workaround: Reconfigure the unhealthy universal connector connections..
INS-43004 After upgrading Guardium Insights, existing Guardium Data Protection connections do not ingest v4 data marts.

Workaround: Re-register the Guardium Data Protection connections to update existing certificates.
INS-43127 Asset inventory page does not load after upgrading Guardium Insights.

Workaround: Manually delete the collections (assets_filter_template, assets_filter_template_mapping, asset_policy, asset_rule) from MongoDB and then launch the Asset inventory page again.
INS-43135 After running systest-preupgrade-validationGuardium Insights, the log contains this warning:
[WARNING] File MONGO_USER is not available.

Workaround: See MongoDB warning in pre-upgrade validation script.
INS-43179 After restoring a backup to Guardium Insights, data marts are not pulled from Guardium Data Protection and the guard_filetransfer_log file contains a Permission denied, please try again error.

Workaround: See Existing data mart pull configuration does not work in a restored environment.
INS-43190 After backing up Guardium Insights version 3.4.0 from cluster A - and then restoring onto cluster B with a different domain name (FQDN) - Filebeat/syslog universal connector connections do not work.

Workaround: Reconfigure the Filebeat/syslog universal connector connections.
INS-43243 Legacy GI to GDP API communication protocol port number cannot be modified if it is removed.

Workaround: Instead of removing the port number, replace it with a copy and paste of the correct port number.
INS-43625 When restoring a backup of Guardium Insights, this error occurs:
2024-07-15 00:53:12: Failed to open connection
2024-07-15 00:53:12: [IBM][CLI Driver] SQL30081N  A communication error has been detected. 
     Communication protocol being used: "TCP/IP".  Communication API being used: "SOCKETS".  
     Location where the error was detected: "172.21.145.22".  
     Communication function detecting the error: "connect".  
     Protocol specific error code(s): "111", "*", "*".  SQLSTATE=08001 SQLCODE=-30081

Workaround: If this is present in the Db2 restore section:

Mon Jul 15 03:51:36 UTC 2024 INFO: c-sysqagi-db2-db2u-0: decompress backup files in /mnt/blumeta0/backup

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
command terminated with exit code 123

Your backup was not downloaded correctly to your PVC and it has been corrupted.

If multiple .gz files exist in your backup directory/DB2/ folder, start with the smallest tgz folder (in terms of byte size) and try to untar with this command:

tar zxf <file.tar.gz>

If you see this:

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

That is the file that is corrupted.

If you still have the original backup files, you can do a md5sum of the backups and compare to the ones used in the restore PVC.

If you no longer have the original backup files, verify that the affected tgz file is does not have the DB2.full prefix in the name of the file.

If the DB2.full file is the affected file, then you will need to restore from a different backup directory. If the affect file has a prefix of DB2.delta and if the original backups are still available, then manually copy over the original gz file to the backup directory using the oc cp or kubectl cp command.

Otherwise, remove all delta files in the backup and restore only the gz file with the DB2.full prefix.

Resources

IBM Guardium Insights documentation: Guardium Insights overview

Guardium Insights v3.4.x system requirements and prerequisites

IBM Security Learning Academy: https://www.securitylearningacademy.com