Overview of IBM Cloud Pak for Watson AIOps
IBM Cloud Pak® for Watson AIOps is an AIOps platform that deploys advanced, explainable AI using the IT Operations (ITOps) toolchain data so that you can confidently assess, diagnose, and resolve incidents across mission-critical workloads.
IBM Cloud Pak for Watson AIOps eases the path to adopting advanced AI for ITOps to decrease your operational costs. With this Cloud Pak, you can increase your customer satisfaction by proactively avoiding incidents and accelerating your time to resolution.
The scale of IT systems and their complexity is continually increasing over the last few years because of digital transformation, containerization, and hybrid cloud adoption. IT teams are being inundated with routine maintenance activities and expanding cloud services, leaving them little or no time to contribute toward innovation. To accelerate business automation, reduce complexity, save costs, and automate regular tasks, companies must use the power of AI.
IBM Cloud Pak for Watson AIOps powers automation by using diverse data sets from an entire range of hybrid environments from cloud to on-premises, and bringing the information together across ITOps. With this Cloud Pak, you can tap into shared automation services to get insight into how your processes run. You can also visualize hotspots and bottlenecks, and pinpoint what to fix with event detection to prioritize which issues to address first.
IBM Cloud Pak for Watson AIOps helps you uncover hidden insights from multiple sources of data, such as logs, metrics, and events. The Cloud Pak delivers those insights directly into the tools that your teams already use, such as Slack or Microsoft Teams, in near real-time. Included AI management tools provides you with unprecedented visibility into your organization's infrastructure so that you can predict failures and facilitate problem resolution.
For information about the IBM Cloud Pak for Watson AIOps architecture, see Architecture.
IBM Cloud Pak for Watson AIOps represents the state of managed entities by using the following three concepts:
Events, which are a record containing structured data that summarizes key attributes of an occurrence on a managed entity, which might be a network resource, some part of that resource, or other key element associated with your network, services, or applications.
An event might or might not indicate something anomalous and is a point-in-time, immutable statement about the entity in question.
Alerts, which represent an ongoing anomalous condition against a single managed entity. Unlike events, alerts might evolve over time as the condition changes. Alerts have a start and an end time. The creation and evolution of alerts are informed by events.
Incidents, which represent the context around an issue that is currently severely impacting operations. An incident includes all alerts that are related to the issue and information about how the affected resources are related. The creation and evolution of incidents are informed by alerts.
Features and benefits
With IBM Cloud Pak for Watson AIOps, you can accelerate your journey toward AI powered operations. You can use the Cloud Pak to set up an integrated, AI-driven set of tools to pull, and analyze data from multiple IT platforms to help with ITOps management. IBM Cloud Pak for Watson AIOps combines a set of capabilities to provide a single solution that facilitates predicting, communicating, and resolving events before they become serious problems. Issues are inevitable in any IT landscape, it's how and when you respond that can make a world of difference to your organization.
IBM Cloud Pak for Watson AIOps includes the following key features:
- Guided tours
- Connectors and data ingestion
- AI modeling and training
- Resource management
- Log anomaly detection
- Metric anomaly
- Event grouping
- Similar ticket comparison
- Incidents and alerts
- Change risk
- Additional capabilities
These capabilities are supported by a network of connectors and capabilities that manage all facets of the AIOps lifecycle from model training to execution.
IBM Cloud Pak for Watson AIOps collects information from all of your IT assets, such as applications, the infrastructure that they run on, and the networking systems that support them. It then uses that data to uncover hidden insights and identify root causes of events. Using this data to train your models lead to better event discovery and a more accurate understanding of your topology. With this advanced understanding of your topology, you can pinpoint where events occur and how far-reaching their impact is. When IBM Cloud Pak for Watson AIOps detects potential incidents, it creates an incident about the incident. Then, it uses a ChatOps environment, such as Slack or Microsoft Teams, to notify, update, and provide your team with near real-time potential remedies, which empowers your teams to automate tasks with clear and open AI decision-making.
The user experience is simplified around AI lifecycle management. The modern experience breaks the complex task of AI model training into a simpler milestone-based experience. This makes it easier for any user to select, filter data, train the models, and view the progress and its output. Through this new experience, you can train the built-in AI models of change risk, event grouping, log anomaly detection, and similar tickets with your data with just a few clicks.
First-time users are presented with a guided tour upon login. This tour is customized for IBM Cloud Pak for Watson AIOps to introduce users to the various home page components, shows how to access tools and capabilities, and where to complete key tasks across the platform.
The home page content is customized based on the user’s role-based access. The initial tour shows the user the following components:
- Insights-related summary cards that provide a high-level glance at important metrics in the environment. These insights are related to only the components that the user has access to view or manage.
- The Quick navigation and Support links for easily accessing commonly used tools, product documentation, IBM Support, and more.
- The main navigation menu for the IBM Cloud Pak Automation console, which lists the tools and pages that are accessible by the user.
- The Tours menu for accessing more tours and guidance on how to use the Automation console and completing essential tasks. The tour menu includes links for the welcome tour, and for tours to help get started with data and tool connections and AI modeling.
- The getting started links for setting up connections, learning more about IBM Cloud Pak, and managing user access control.
More tours can be accessed at any time by the user to get a better understanding of how to get started with different tasks and components. Users can select a tour by clicking the Tours icon in the Automation console toolbar, and clicking their preferred tour. The following overview tours are available for IBM Cloud Pak 4.1.1:
Welcome to IBM Cloud Pak for Watson AIOps
A repeat of the initial Welcome guided tour.
Discover AIOps insights
Overview of the AIOps insights dashboard, which summarizes how IBM Cloud Pak for Watson AIOps helps organizations anticipate, troubleshoot, and resolve IT incidents. The tour loads sample data to walk the user through available toolbars and charts, including Mean time to restore, Noise reduction, Incident activity, Runbook usage, and the optional, nonproduction use Applications disruption costs chart.
Start with data and tool connections
Available from the Data and tool connections and the home page. This tour guides users through the Data and tool connections to show how to add connections between IBM Cloud Pak for Watson AIOps and other tools.
Introduction to resource management
Available from the Resource management tool. This tour identifies the resource management capabilities, which you can use to gain an overview of applications, resource groups, and resources and select them to view their details and topology. The tour introduces you to resource groups, which you can use to build application definitions. These resource groups are created automatically when you connect tools, however, you can add groups by using the template page.
Understand applications and resource groups
Available from the Resource management tool when a user selects an application or a resource group to view details. This tour shows a user how to view incident details, view affected resources, view and navigate the timeline, and how to get insights from changes to a resource over the time.
Learn about the timeline
Available from the Resource management tool. This tour is available when a user selects to view details for an application that has an associated incident. This tour walks the user through the details of the timeline widget, showing how to interact with the data and see changes that have occurred for alerts.
Explore AI modeling
Available from the AI Model Management and the home page. This tour helps you to understand the two types of AI algorithms that are available to enable AI modeling. This tour uses the Change risk model summary card to show you how to set up and monitor the training of model-generating algorithms.
Introduction to automations
Available from the Automations tool, which the user can use to proactively set up actions and policies to detect and remediate events, both current and future. With this tour, a user learns how to create policies, view policy details, create and view runbooks, and more.
Create a new policy
This step-by-step tutorial is available from the Automations tool to guide users in the creation of a new policy. This step-by-step tutorial uses the Promote alerts to an incident policy template to help you understand the basics of setting up a new automation policy.
Introduction to incidents and alerts
Available from the Incidents and alerts tool. This tour provides an overview of how incidents bring insights together to present the holistic view of an incident. The tour guides you through the different tabs, options, and views of the Incidents and alerts tool to show you how to view your existing incidents and alerts, including how to filter the incident list, drill down to view alert details, clear alerts, and more.
Investigate an incident
Available from the Incidents and alerts tool when a user views an incident in detail. This tour guides a user through the process to investigate an incident. A user is introduced to how to view probable causes and top alerts, review associated automations, change incident settings, investigate incident alerts, and how to visualize an incident. The tour also introduces details on incidents, trigger alerts, and the legend and timeline for visualizing an incident.
Connectors and data ingestion
Data is the life blood of AI but configuring data feeds can be complex. You can use the Automation console tools and connectors to set up connections to a wide variety of systems for data ingestion. IBM Cloud Pak for Watson AIOps supports connections to large system solutions including log management, event management, and service management tools within an IT environment, such as Splunk, Falcon LogScale, LogDNA, PagerDuty, and ServiceNow. You can also use the Netcool probes to ingest events into IBM Cloud Pak for Watson AIOps.
A gRPC framework is used for some connectors to support local or remote deployment of a bidirectional data flow. These connectors, such as the Instana, Slack, and GitHub connector, can be deployed locally in the same project (namespace) as IBM Cloud Pak for Watson AIOps, or deployed remotely.
For the full list of supported connections, see Configuring data and tool connections.
Connectors are available for connecting to IBM Tivoli Netcool/OMNIbus and IBM Tivoli Netcool/Impact.
A Netcool connection is used to define a connection to IBM Tivoli Netcool/OMNIbus. An IBM Tivoli Netcool/Impact connection is used to define a connection to IBM Tivoli Netcool/Impact. For more information about these connections and integrating with Netcool, see Netcool integrations.
An Instana connector is available for connecting to and integrating with Instana to enable connectivity to Instana events, metrics, and topology. Instana assets can now be combined with IBM Cloud Pak for Watson AIOps AI, such as runbooks, metric anomaly detection, and log anomaly detection to help you gain insights. For more information about setting up this connection, see Creating Instana connections.
IBM Cloud Pak for Watson AIOps also includes Ansible connectors for connecting to Ansible and an SSH connector to connect to another system where scripts and commands can run.
The Ansible connection provides you with the ability to set up a connection to an Ansible automation platform controller server to gain benefits from the configuration and orchestration capabilities of Ansible. For more information about setting up the connection, see Ansible automation provider.
The SSH connection is agentless and connects directly to the target machine. It authenticates by using SSH public key authentication. You can only create one SSH Connection. For more information, Creating SSH connections.
You can train AI models to hone its ability to derive insights from your incoming data connections. To get the most out of your AI models, you can manually map your log data to the JSON training format and train your models with it. For more information about training specific types of models, including suggested mappings for events and logs, see the following training topics:
For more information about what you can do to gain further insight into your IT infrastructure, see the following topics:
For guidance on how to plan data connection and AI model training, see the following topic: Planning data loading and training.
IBM Cloud Pak for Watson AIOps brings together the capability to group resources from different data types into applications (services). You can flexibly define an application (service) to meet your business needs. With applications (services), you can obtain an integrated view of your resources to understand inter-dependencies.
IBM Cloud Pak for Watson AIOps focuses on the user experience and health of your applications (services). This application-centric approach uses a built-in application template model to dynamically aggregate and group tagged resources to enable rapid visualization of different application components.
An integrated service and topology management and topology viewer provides complete up-to-date visibility of your applications and resources, and provides you with control over dynamic infrastructure and services. You can configure your topology visualizations to also show real-time or historical data.
In addition, to help you pinpoint incidents within applications and resource topologies, IBM Cloud Pak for Watson AIOps tools work together to correlate events.
Log anomaly detection
IBM Cloud Pak for Watson AIOps includes a pair of AI algorithms to gather log data from one or multiple components in the application architecture to identify a baseline of expected log message types. IBM Cloud Pak for Watson AIOps uses this baseline to discover abnormal behavior in your live log data. These AI algorithms can run independently of the other. If both algorithms are enabled, then any log anomalies discovered by both are reconciled, so that only one alert is generated. In this case, the severity of the combined alert is equal to the highest severity of the two alerts. Each algorithm is used to gather data differently:
Natural language log anomaly detection uses natural language techniques on a subset of your log data to discover abnormal behavior. This log anomaly detection takes large amounts of log data and trains on it to learn what is normal behavior for a particular component. This model goes beyond just looking at error states or frequency of metadata around log messages. Instead, it determines when something becomes an anomaly compared to what patterns it typically exhibits.
Statistical baseline log anomaly detection uses a statistical moving average on all of your log data to discover abnormal behavior. This algorithm automatically detects unusual patterns in logs and notifies you when they occur. Data that is used for analysis is updated every 30 minutes so this algorithm provides value quickly.
For more information, see Log anomaly training definition.
Metric anomaly detection provides the ability to analyze any time series metric automatically using AI without the need for configuration or tuning. This AI leverages the Instana Integration, automations, and policies. You can group metric anomalies with log anomalies and other Alerts.
Metric anomaly detection can help you measure key performance indicators, like memory availability or latency, to identify trends and deviations in these areas early.
For more information, see About metric anomaly detection.
AI algorithms are provided with IBM Cloud Pak for Watson AIOps to group events together and present the events within a single incident.
Temporal grouping is used to group events that are discovered to co-occur over time. When a problem arises, typically multiple parts of a system or environment are impacted. When events in different areas co-occur, it makes sense to look at them together and treat them as one problem to try to determine what might have occurred. Grouping co-occurring events together reduces the number of tickets and incidents that are opened and the number of people looking at the same problem, thereby significantly reducing noise in your monitoring systems. It helps you to understand the context of an issue so you can prioritize, triage, and resolve it more quickly.
Topological grouping is used to group events based on the resource groups in which events occur. For example, if you have a resource group that is made up of all the resources within a given Kubernetes namespace, then any events on pods, microservices, or other resources in that namespace are grouped together in a single topological group. Topological grouping helps you understand when events are connected based on their topology, providing valuable context for why related events might occur together.
Scope-based grouping automatically groups events relating to an incident if they have the same defined scope and occur during the same period. A scope can be used to identify where events originate based on a common attribute, for example, the location of a server room. By understanding when events are related based on both time and location, you can more quickly diagnose incidents.
For more information, see Event grouping training definition.
Similar ticket comparison
The similar ticket AI algorithm in IBM Cloud Pak for Watson AIOps can be used to define an AI model so that when incidents occur, the model can review details for similar tickets from the past to help determine a resolution to current problems. This model aggregates information from past similar tickets. It can also extract the steps that are used to fix previous incidents, if documented. This AI algorithm discovers details about similar messages, anomalies, and events within your tickets that occurred in the past and are impacting the current application. For more information, see Setting up training for similar tickets.
Incidents and alerts
IBM Cloud Pak for Watson AIOps enables you to identify health issues across your application, services, and network infrastructure on a single management console. It provides a Incidents and alerts tool that brings together incidents, alert details, alert severity, and alert filtering. This capability helps you to analyze the context of incidents across the full enterprise estate. This viewer acts as a single pane of glass incident dashboard, which displays key details like the probable cause, topology, and associated automations to resolve the incident.
Some alerts in the console are presented in alert groups, based on analytics. By using the alert list, you can explore why these alerts were grouped together, and this understanding can further help with alert resolution.
Alerts are displayed in the Alert Viewer. From a ChatOps integration (Slack or Microsoft Teams), you can launch into an Alert Viewer that shows the alerts that are associated to a specific incident.
For more information, see Managing incidents and alerts.
IBM Cloud Pak for Watson AIOps ChatOps support includes connections with existing social collaboration tools, Slack, and Microsoft Teams. These connections help to deliver AI-powered automation for ITOps and management within their preferred collaboration tool of choice.
From a ChatOps integration you can launch into the Automation console Incidents and alerts tool to view more information about an alert that is being viewed in ChatOps.
IBM Cloud Pak for Watson AIOps uses ChatOps to provide a message-based interface for reporting incidents with the ITOps that you are monitoring. This ChatOps interface displays the following types of operations information:
|Anomaly Detection||Provides details about detected anomalies in your real-time or offline data, such as the time when the incident was detected, the incident description, status, and more.|
|Relevant events||Groups-related events to aid incident diagnosis. Events can include alerts, PagerDuty alerts, or log anomalies.|
|Probable cause||Derives root fault component, and derives the full scope of components that are affected by an incident.|
|Ticket Similarity||For a particular ticket, finds the highest "n"-ranked similar tickets from the past.|
|Next Best Action||For a particular incident, suggests the highest "n" actions from similar tickets from the past.|
with IBM Cloud Pak for Watson AIOps an engine and UI for automation is included that you can use to create automation policies, runbooks, and actions to help you resolve incidents quickly by setting up and enabling an automatic response as situations arise. Starting up additional capacity or restarting a failed service can happen before your team knows there is an issue.
For setting up automated procedures, you can proactively set up policies, runbooks, and actions to detect and remediate events, both current and future.
Policies are rules that contain multiple condition and action sets. They can be triggered to automatically promote events to alerts, reduce noise by grouping alerts into an incident, and assign runbooks to remediate alerts. For more information, see Policies.
Runbooks automate procedures that do not require human interaction, thereby increasing the efficiency of IT operations processes. With runbooks, your operators can spend more time innovating and are freed from performing time-consuming manual tasks. For more information, see Runbook Automation.
Actions in runbooks are the collection of manual steps grouped into a single automated entity. An action improves runbook efficiency by automatically performing procedures and operations. For more information, see Actions
IBM Cloud Pak for Watson AIOps includes an AI algorithm for providing change risk assessment and includes the capability of surfacing change risk assessments through ChatOps.
The change risk AI algorithm provides an assessment of the risk of implementing a proposed change; for example, a code change or a new software version. Hundreds of changes can affect an application during its lifecycle. Change risk correlates historical change data with historical incident data and determines how likely it is that a specific code change would cause a problem. Using the assessment percentage provided, you can determine how safe it is to proceed with the change. Training this AI model can help you ensure that risky changes for an application are assessed before deployment. For more information, see Change risk training definition.
The change risk assessment in ChatOps provides you with a model confidence percentage that represents the confidence in the AI's prediction of the impact of a change. This percentage is relative to the risk associated with a new or updated ServiceNow change request ticket. The model confidence percentage is calculated based on the change risk AI model that is based on the change risk AI algorithm and the data that you feed into it. Change risk can be evaluated based on any type of change, whether it is changes to code, or to configurations, or to data. If you have historical change request tickets for that type of change, IBM Cloud Pak for Watson AIOps can provide a model confidence percentage for it. For more information, see Change risk in ChatOps.
Infrastructure automation is included as an optional feature with IBM Cloud Pak for Watson AIOps. This Infrastructure automation component consists of the following components, which were available with IBM Cloud Pak for Multicloud Management:
- Infrastructure management, previously called IBM Red Hat CloudForms.
- Managed services, previously called Terraform & Service Automation or IBM Cloud Automation Manager.
Infrastructure management delivers the insight, control, and automation enterprises need to address the challenges of managing virtual environments, which are far more complex than physical ones. This technology enables enterprises with existing virtual infrastructures to improve visibility and control, and those just starting virtualization deployments to build and operate a well-managed virtual infrastructure.
Managed services provide you with the capability to automate provisioning of infrastructure and virtual machine applications across multiple cloud environments with optional workflow orchestration.
Infrastructure automation enables IT Operations and Site Reliability Engineer (SRE) teams to use infrastructure as code practices to drive IT velocity and shift to the left of operations.
Infrastructure automation is installed within a separate cluster from IBM Cloud Pak for Watson AIOps. For more information, see Infrastructure automation.
Bringing the features together
By combining AI-driven insights with a modern user experience, you can achieve reduced Mean Time to Resolution (MTTR) when an incident occurs and avoid incidents in the first place through a change risk advisor. The user experience brings in different types of data together and builds out an application perspective, delivering unique insights from the AI models.
Application impact avoidance
The trained AI models establish a baseline and automatically detect anomalies across structured and unstructured data. The models reduce noise and bring related entities across different data sets together into a complete “incident” ahead of a potential incident or application impact. A much smaller set of high-impact “incidents” versus a flood of events and incidents, provides deep insights with explainability.
These incidents help you to visually understand the probable cause to help prevent future downstream impact. The Next Best Action recommendation from past incidents, and the built-in Runbook Automation capabilities accelerate the time to remediation of an incident. The probable cause capability allows users to isolate and find the root cause of the problem faster.
- Derive root fault component and probable cause of affected components to isolate problems faster.
- Detect anomalies by using automatic parsing to detect anomalies in logs and alerts earlier than rule-based alerts.
- Integrate with pre-trained AI models to gain new insights from existing tools. Keep teams focused, surfacing insights and recommendations into existing workflows.
- Avoid application impact through correlating data across all relevant data sources. Detect hidden anomalies, anticipate issues, and resolve faster. Correlate a vast amount of unstructured and structured data in real time with AIOps tools.
- Proactively avoid risks and automate runbooks for more efficient workflows.
- Proactively avoid incidents with explainable AI that advises based on inputs across ITOps.
- Create a visual representation of the data that is flowing through your business and pinpoint bottlenecks in your operations.
- Draw on advanced AI and analytics to create a baseline that you’ll use to track key performance indicators (KPIs) over time.