Incidents can cause a host of problems for organizations, from temporary downtime to data loss. When done well, incident management can provide an efficient and effective way to fix all kinds of incidents with little disruption and in a way that leaves organizations more prepared for the next incident.

With roots in the IT service desk, incident management has long served as the primary interface between IT Operations (ITOps) and the end user. As technology has advanced and become more complex, so has the way organizations see incident response. It has expanded far beyond helping users fix problems to become a process for maintaining constant app uptime and accelerating continuous improvement efforts.

What is incident management?

Incident management is a process used by IT Operations and DevOps teams to respond to and address unplanned events that can affect service quality or service operations. Incident management aims to identify and correct problems while maintaining normal service and minimizing impact to the business.

Incident management for IT

Incident management within a company’s IT operations, often referred to as ITIL incident management, addresses a wide range of issues that can impact service and business operations, from a laptop crashing or a printer error to Wi-Fi connectivity issues and network downtime.

Incident management, under the framework of ITSM (IT service management), functions as one aspect of the ITSM service model. Rather than focusing on creating systems and technology, incident management for IT is more user-focused, aiming to keep systems online and running—whether it be an app or an endpoint (e.g., a sensor or desktop computer).

Incidents vs. service requests

Within ITSM, the IT department has various roles, including addressing issues as they arise. The severity of these issues is what differentiates an incident from a service request.

A service request, simply put, is when a user is asking for something to be provided, such as advice or equipment. Services can include requesting assistance in resetting a password or getting additional memory for a desktop computer.

An incident, on the other hand, is more urgent and indicates an underlying error that needs addressing.

Incidents vs. problems

An incident is a single, unplanned event that causes a disruption in service, while a problem is the root cause of a disruption in service, which can be a single incident or a series of cascading incidents.

The difference plays out in remediation and how responders approach fixing the issue. Incident response is reactive. IT departments get an alarm and address the incident. When addressing a problem, however, IT teams identify the root cause and then fix it. Problem management takes a proactive approach, looking at various types of incidents and patterns that emerge to understand how future incidents can be prevented.

Incident management for DevOps

DevOps teams are focused on finding more efficient ways to build, test and deploy software, which in part, requires addressing incidents quickly. Like ITIL incident management, DevOps incident management aims to fix issues without disrupting operations. For example, DevOps teams might monitor for poor mean time between failures (MTBF) metrics, which can indicate that there’s an underlying issue that needs to be investigated.

Because DevOps is rooted in continuous improvement, there is a significant focus on post-mortem analysis and a blame-free culture of transparency. The goal is to improve the overall system performance, resolve future incidents more quickly, and prevent future incidents from happening.

Like today’s IT teams, DevOps may use automated provisioning, incident prioritization and artificial intelligence (AI)-enabled root-cause analysis tools to ensure uptime, address the most pressing incidents first and more quickly learn how to fix—and prevent—future problems.

Incident management process

Organizations typically create an incident management process that documents the sequence of events the response team should take. Everyone should know which staff are responsible for handling incidents, the time it should take to solve the issue, when to escalate the incident to the next level and how to document the incident and the way it was resolved.

Once the process is defined, the incident management workflow typically goes as follows:

• Identify the incident: Whether it’s an end user submitting a ticket to the help desk or an automated alert system notifying the team of an issue, the response team needs a way to receive reports of problems within the system.
• Log and classify the incident: This includes entering the report into an incident logging system and assigning prioritization, including which level of staff should handle it. For example, Level 1 incidents are usually handled by newer, less experienced staff while Level 2 and Level 3 incidents are increasingly more challenging to solve and require the most experienced responders.
• Contain the issue: If it is a security incident, response teams must act quickly to contain the issue, whether it’s a DDoS attack or a data breach. In all cases, teams must ensure the incident doesn’t spread and further impact the system.
• Diagnose the incident: This is where the troubleshooting comes in. Response teams may use a knowledge base or ChatOps tool to suggest possible causes and save time.
• Resolve the incident: Once the cause has been identified, teams get to work addressing the incident, whether it’s provisioning additional memory or addressing a network outage.
• Close and review the incident: Post-mortem reviews are an important aspect of improving reliability and availability in today’s digital environments. This data not only increases the organization’s institutional knowledge, but it can also be used in machine learning and AI-enabled tools to help identify incidents more quickly and even create notifications when incidents are likely to happen.

Why use incident management?

All organizations need to fix problems and resolve incidents. It’s how they keep the business running. But there are also clear benefits to having effective incident resolution tools—and teams—that can react quickly without major disruption to the business. Those benefits include the following:

• Faster problem resolution: Incident management tools, automation and AIOps help teams identify problems and fix them quickly. This, in turn, improves efficiency by allowing teams to focus on core business operations instead of constant firefighting.
• Better user experience: When incidents are getting fixed right the first time and are fixed faster, it improves service quality for the end user. This begins with a clear and easy-to-use system for reporting service disruptions and continues with good communication as incidents are addressed.
• More operational efficiency: Incident response creates a system where issues have a clear path to resolution and helps build institutional knowledge over time. This knowledge—either held by staff or integrated into an automated system driven by AI—helps document important performance metrics (e.g., mean time to resolution (MTTR)) can help ensure the organization is maintaining a high level of service.
• Deeper insights: With an effective incident management system in place, teams can address major incidents faster and extract insights for root cause analysis. When team members document how past incidents were resolved, they start to create a playbook for solving similar problems in the future.
• Meeting SLAs: A service-level agreement (SLA) defines the level of service a company is required to provide to a customer. Therefore, incident response and management play a key role in meeting the metrics and key performance indicators (KPIs) defined in the SLA.

Incident management tools and automation

The growing complexity of IT operations, driven in part by the many applications organizations rely upon in day-to-day business operations, has made incident response tools and automation more important than ever.

Here are some of the most common incident management tools:

• Monitoring tools: Help identify outages, trigger alerts, and diagnose incidents. Monitoring tools also reduce costs by freeing DevOps teams to better manage the software lifecycle.
• Service desks: A place for users to submit tickets, chat with the service desk team, monitor the progress of their tickets and perform some self-service tasks. Typically, the service desk is run through a management system that enables key incident management tasks, such as prioritization and categorization.
• AlOps platforms: Using logs and historic data, AIOps can provide context for better decision-making, smarter resource allocation and faster incident response. Companies that use AIOps for incident management have reported reducing IT costs and MTTR by 50%.
• VDocumentation: Scripts that automatically document changes to an environment, making it easy to record incidents for postmortem analysis. For example, teams can set up the PowerCLI scripts to run on a monthly schedule to record incidents for deeper analysis.

Incident management and IBM

IBM offers a proactive incident management software solution that enables your IT staff to correlate information across all relevant data sources, detect hidden anomalies, anticipate issues and resolve them faster to proactively get ahead of any negative end-user and business impacts.

Watch how a site reliability engineer (SRE) at a mock e-commerce business can make applications run smoothly: